Direct memory access architecture with multiple stages and steps

Abstract

A DMA architecture capable of performing multi-level, multi-step, and parallel determination of multiple memory addresses is described. In one aspect, the DMA system includes one or more hardware DMA threads. Each DMA thread includes a request generator configured to generate m memory addresses for a multidimensional tensor in parallel during each parallel memory address computation cycle, and to generate a corresponding request for performing a memory operation on the memory system for each address. The request generator includes m memory address units, each memory address unit including a step size tracker configured to generate a corresponding step size index value for each dimension of the tensor, and to generate a corresponding step offset value for that dimension based on the corresponding step size index value. Each memory address unit includes a memory address computation element configured to generate memory addresses for tensor elements and send requests to perform memory operations.

Description

Technical Field

[0001] This invention relates to a direct memory access architecture with multiple levels and steps. Background Technology

[0002] Direct memory access (DMA) is the ability of a device or subsystem to access memory independently of the processor. This frees the processor from data transfers, allowing it to perform other operations. DMA can be used to offload costly memory operations from the processor, such as large memory transfers and scatter-aggregate operations. Summary of the Invention

[0003] This specification describes techniques involving DMA architectures that can, for example, perform multiple levels and steps in parallel and determine multiple memory addresses during a single clock cycle.

[0004] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a direct memory access (DMA) system comprising one or more hardware DMA threads. Each DMA thread includes: a request generator configured to, during each parallel memory address computation cycle, (i) generate m memory addresses for a multidimensional tensor in parallel, and for each memory address, (ii) generate a corresponding request to the memory system to perform a memory operation on the multidimensional tensor. The request generator includes m memory address units. Each memory address unit includes a step size tracker configured to, for each dimension of the multidimensional tensor, generate (i) a corresponding step size index value for the dimension, and based on the corresponding step size index value, generate (ii) a corresponding step offset value for the dimension; and a memory address computation element configured to, during each parallel memory address computation cycle and based on each corresponding step offset value, generate memory addresses for tensor elements of the multidimensional tensor; and transmit a request to perform the memory operation using the memory address to the memory system, wherein m is greater than or equal to 1. Other implementations of this aspect include corresponding apparatus and methods.

[0005] These and other implementations may each optionally include one or more of the following features. In some aspects, the request generator is configured to generate the memory addresses in parallel during a single clock cycle, and to perform each parallel memory computation during the single clock cycle. During each clock cycle, the memory address computation element of each memory address unit generates a memory address for a tensor element that is the same as or different from the memory address computation element of each other memory address unit.

[0006] In some aspects, the request generator is configured to receive a descriptor for the multidimensional tensor, the descriptor defining a corresponding step size for a step value for each dimension. The request generator may include m channels, each channel including a corresponding step size tracker and a corresponding memory address computation element. The corresponding step size tracker and corresponding memory address computation element for each channel compute the corresponding memory address in parallel with each of the other channels. The step size tracker may be configured to generate memory addresses for the multidimensional tensor based on nested loops, the nesting of loops for each dimension of the multidimensional tensor including a corresponding loop for traversing the dimensions of the multidimensional tensor. A step size value per step for each dimension represents the loop boundary for the corresponding loop for that dimension, and a step size index value for each dimension represents the loop index for the corresponding loop for that dimension.

[0007] In some aspects, each step tracker is configured to update the step index value for each dimension in the dimension during each clock cycle. The combination of step index values ​​for each step tracker may differ from the combination of step index values ​​for each of the other step trackers. Each step tracker may include a chain of step incrementers comprising multiple step incrementers, each configured to determine a dimension memory address offset value for the corresponding dimension. The first step increment of the chain of step incrementers corresponding to the innermost loop of the nested loops may be configured to receive an advance. Updating the step index value for one or more dimensions in the dimension during each clock cycle may include updating the step index value for the one or more dimensions based on the advance by the first step increment.

[0008] In some aspects, each of one or more second step incrementers in the step incrementer chain corresponding to the loop nested within the innermost loop is configured to receive a wrap amount from a previous step tracker in the step incrementer chain. Updating the step index value for one or more of the dimensions during each clock cycle may include updating the step index value for the one or more dimensions by the second step incrementer based on the wrap amount.

[0009] Some aspects may include a progress tracker, which includes a response reordering unit and a synchronization unit. The response reordering unit may be configured to maintain a state for each tensor regarding whether a memory operation has been performed on the tensor element. The synchronization unit may be configured to provide a plurality of partial updates to the processor core, each of which specifies the overall state of memory operations performed on the tensor elements of the multidimensional tensor.

[0010] In some respects, each request includes a unique identifier. The response reordering unit can be configured to receive responses from the memory system in any order. Each response may include a unique identifier of the request for which the response was provided. The response reordering unit can be configured to release the set of unique identifiers for reuse by the request generator when at least a threshold number of consecutive unique identifiers are received in the responses.

[0011] Typically, another innovative aspect of the subject matter described in this specification can be embodied in a system comprising one or more processor cores, a memory system, and a DMA engine comprising one or more DMA threads. Each DMA thread may include a request generator configured to, during each parallel memory address computation cycle, (i) generate m memory addresses for a multidimensional tensor in parallel, and for each memory address, (ii) generate a corresponding request to the memory system to perform a memory operation on the multidimensional tensor, wherein the request generator comprises m memory address units, where m is greater than or equal to 1. Each memory address unit may include a step size tracker configured to, for each dimension of the multidimensional tensor, generate (i) a corresponding step size index value for the dimension, and, based on the corresponding step size index value, generate (ii) a corresponding step offset value for the dimension; and a memory address computation element configured to: generate memory addresses for tensor elements of the multidimensional tensor during each parallel memory address computation cycle and based on each corresponding step offset value; and transmit a request to perform the memory operation using the memory address to the memory system. Each DMA thread may include a progress tracker, which includes a response reordering unit and a synchronization update unit configured to provide partially synchronized updates to the one or more processor cores for memory operations managed by the DMA engine. Other implementations of this aspect include corresponding apparatus and methods.

[0012] These and other implementations may each optionally include one or more of the following features. In some aspects, the request generator is configured to generate the memory addresses in parallel during a single clock cycle and to perform each parallel memory computation during a single clock cycle.

[0013] During each clock cycle, the memory address computation element of each memory address unit can generate a memory address for a tensor element that is the same as or different from the memory address computation element of each other memory address unit. The request generator can be configured for the multidimensional tensor receiving descriptor, which defines a corresponding step size for the step value of each dimension. The request generator can include m channels, each channel including a corresponding step size tracker and a corresponding memory address computation element, wherein the corresponding step size tracker and the corresponding memory address computation element of each channel compute the corresponding memory address in parallel with each other channel.

[0014] Typically, another innovative aspect of the subject matter described in this specification can be embodied in a method performed by a DMA system. This method includes: (i) generating m memory addresses for a multidimensional tensor in parallel during each parallel memory address computation cycle by a request generator, and for each memory address, (ii) generating a corresponding request to the memory system to perform a memory operation on the multidimensional tensor, wherein the request generator comprises m memory address units, where m is greater than or equal to 1, and wherein each memory address unit comprises a step size tracker and a memory address computation unit; generating (i) a corresponding step size index value for each dimension of the multidimensional tensor by the step size tracker of each memory address unit, and generating (ii) a corresponding step offset value for the dimension based on the corresponding step size index value; generating memory addresses for tensor elements of the multidimensional tensor based on each corresponding step offset value by the memory address computation element of each memory address unit during each parallel memory address computation cycle; and transmitting a request to perform the memory operation using the memory addresses to the memory system.

[0015] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The DMA architecture described in this document enables the generation of multiple (m) memory addresses for a multidimensional tensor in parallel, for example, per clock cycle, providing faster memory address generation and higher memory throughput. The DMA architecture may include multiple channels, each with a step size tracker that performs multi-step techniques to compute memory addresses for tensor elements in parallel based on nested loops for the multidimensional tensor. The techniques used by the request generator of the DMA architecture enable multiple step size trackers to operate in parallel using different step size index values ​​for multiple dimensions of the tensor to generate addresses for different tensor elements in parallel during a clock cycle and independently of each other step size tracker. Each step size tracker may update its step size index value during each clock cycle to prepare for determining the memory address for its next tensor element.

[0016] The DMA architecture may also include a progress tracker that provides partially synchronized updates to the processor core, which will consume data stored in memory at a determined memory address. This allows the processor core to begin consuming data before the entire DMA memory transaction is complete, thereby reducing the latency imposed on the processor core by memory transfers and improving the overall efficiency of processor computation. The progress tracker may include a response reordering unit that can receive multiple responses from memory at once and in any order, allowing the memory to process and respond to generated requests out of order. Since the size of the response reordering unit is finite, it can release the identifier used for the request when responses to at least a threshold number of requests have been received. This allows the request generator to use the released identifier to continue issuing memory requests without waiting for responses to all maximum number of requests, thereby improving the speed and efficiency of memory transfers.

[0017] The various features and advantages of the foregoing subject matter are described below with reference to the accompanying drawings. Other features and advantages will be apparent from the subject matter and claims described herein. Attached Figure Description

[0018] Figure 1 This is a diagram of an example environment where the DMA thread generates and tracks the progress of memory operations.

[0019] Figure 2A This is a diagram of an example request generator.

[0020] Figure 2B Example pseudocode for determining memory addresses is described.

[0021] Figure 3 This is a diagram of an example step size tracker.

[0022] Figure 4 This is a diagram of an example step incrementer chain.

[0023] Figure 5 This is a diagram of an example progress tracker.

[0024] Figure 6 This is a diagram of an example response reordering unit.

[0025] Figure 7 This is a flowchart illustrating an example process for generating a request for memory operation.

[0026] Figure 8 This is a flowchart illustrating an example process for tracking the progress of memory operations.

[0027] In the various figures, the same reference numerals and names denote the same elements. Detailed Implementation

[0028] Generally, this document describes a DMA architecture capable of performing multiple levels and steps in parallel, for example, during a single clock cycle and within a single clock cycle, and determining multiple memory addresses. A chip may include one or more DMA engines that offload memory transfer operations from one or more processor cores of the chip. Each DMA engine may include one or more DMA threads. Each DMA thread is a hardware unit that manages the execution of DMA transactions on behalf of one or more cores. A clock cycle may be the time required for a DMA engine or core to perform an operation.

[0029] Figures 1-6 The example DMA architecture shown and described below provides a design that can support up to four tensor dimensions and up to four source and four destination memory addresses per clock cycle. However, the DMA architecture is not particularly suited to having only four dimensions or four addresses per cycle. Similar architectures can be used for other numbers of dimensions and addresses per cycle. Furthermore, the number of addresses per cycle can differ from the number of dimensions of the tensor for which the addresses will be determined. For example, a DMA thread could include five channels generating five addresses per cycle, while the hardware is configured to compute addresses of tensors with up to four dimensions, up to 10 dimensions, or another maximum number of dimensions. That is, the architecture is parameterizable, and the choice of settings depends on the area / frequency targets of the design.

[0030] Figure 1This is a diagram of an example environment 100 where DMA thread 120 generates and tracks the progress of memory operations. DMA thread 120 can generate and track the progress of memory operations as part of transactions representing requests from processor cores. DMA thread 120 is a hardware unit that can be part of a DMA engine, which includes DMA thread 120 and optionally includes one or more additional DMA threads. DMA thread 120 can manage DMA transactions, such as scatter-gather and other memory transfer operations, for one or more processor cores (including core 110). For example, DMA thread 120 can orchestrate the transfer of multidimensional tensors between different memories in a memory system on an on-chip including core 110 and DMA thread 120. DMA thread 120 orchestrates data movement by sending requests (commands) to the memory system and tracks the completion of these requests, allowing it to synchronize progress with the cores of the requesting transactions. Once a read / write request / command is in the memory system, each request can be served independently, regardless of request ordering. DMA thread 120 handles the ordering of requests / responses and synchronization with the cores. Offloading these memory operations to DMA thread 120 frees up computation cycles on kernel 110 for other tasks (e.g., performing machine learning computations, shaping tensors, etc.).

[0031] Kernel 110 can request a DMA transaction by sending descriptor 112 to DMA thread 120. Each DMA transaction may include one or more memory transfer operations. Descriptor 112 includes information about the DMA transaction. For example, descriptor 112 may include information specifying the source memory 152 (e.g., one or more memory addresses of source memory 152) from which data will be read, the destination memory 154 (e.g., one or more memory addresses of destination memory 154) to which data will be written, the size and shape (e.g., dimensions) of the source tensor for which tensor elements will be stored in source memory 152, and the size and shape of the destination tensor for which tensor elements will be stored in destination memory 154. A tensor element is a piece of data in a tensor that corresponds to a specific index position in the tensor.

[0032] The size and shape of the source tensor can be the same as or different from the size and shape of the destination tensor. For example, if the tensor is reshaped by kernel 110, the size and shape can be different. Descriptor 112 can define the size and shape of each tensor using a per-step step value for each dimension of the tensor. In a for loop, the step size is the increment of the loop for each iteration, while the per-step step is the total number of steps before the loop is reset, such as the loop bound.

[0033] For example, the step size for each dimension of a tensor can be equal to the number of tensor elements in that dimension. Specifically, an 8×6×4×2 four-dimensional tensor can have a step size of 8 for the first dimension, a step size of 6 for the second dimension, a step size of 4 for the third dimension, and a step size of 2 for the fourth dimension. As described in more detail below, the step size can be used to traverse each dimension of the tensor and compute the memory address for each tensor element.

[0034] Descriptor 112 may also include stride dimension offset values ​​for each dimension. These stride dimension offset values ​​(also referred to as dimension offset values) are used to determine the memory address for the tensor element, as described below. The dimension offset is the stride distance. At each step of the process along the tensor dimension, DMA thread 120 causes the memory address to "jump" by the stride dimension offset value. For the source tensor, descriptor 112 may include stride dimension offset values ​​for each dimension of the source tensor. For the destination tensor, descriptor 112 may also include stride dimension offset values ​​for each dimension of the destination tensor.

[0035] DMA thread 120 includes a descriptor queue 122 that stores descriptors 112. For example, DMA thread 120 can execute multiple DMA threads sequentially based on descriptors 112 received and stored in descriptor queue 122. In some implementations, descriptor queue 122 is a first-in, first-out (FIFO) queue, allowing DMA thread 120 to execute DMA transactions in the order in which descriptors 112 for DMA transactions are received. The execution of DMA transactions is fully pipelining and can be implemented in an out-of-order manner, but appears to the processor as if executed in program order.

[0036] DMA thread 120 also includes a descriptor splitter 124. The descriptor splitter 124 can extract information used by the source sub-thread 130 and the destination sub-thread 140 from the descriptor 112, and provide appropriate information to each sub-thread 130 and 140.

[0037] Generally, source thread 130 generates a read request to read data from source memory 152, sends the read request to memory system 150, tracks the progress of the read operation, and synchronizes kernel 110 with the progress of the read operation. Similarly, destination thread 140 generates a write request to write data to destination memory 154, sends the write request to memory system 150, tracks the progress of the write operation, and synchronizes kernel 110 with the progress of the write operation. Memory system 150 can be for kernel 110 or for the main memory of a chip including kernel 110, such as random access memory (RAM) for kernel 110 or the chip. The memory system implements a true memory interconnect, such that data read for each source memory read request is paired with a write request to the destination memory. Data never passes through DMA threads (request addresses are issued and responses are received, but those responses and requests do not carry memory data).

[0038] Source sub-thread 130 includes a request generator 132 that generates read requests based on descriptor 112. As described in more detail below, request generator 132 may generate multiple memory addresses in parallel, for example, per clock cycle, and generate a read request for each memory address. For example, when request generator 132 traverses a multidimensional tensor, it may generate a corresponding memory address for each of the multiple tensor elements during a single clock cycle of kernel 110. Each read request may include a request identifier “request ID”, the memory address from which data is to be read, and a memory opcode. The request ID may be a sequence number or tag used to associate the request with a response, since responses may be received out of order, as described below. The memory opcode indicates a memory operation, such as whether the request is for a read, write, a set of memory, or another operation targeting the requested memory address.

[0039] Source thread 130 also includes a progress tracker 134 that tracks the progress of a read operation specified by a read request. For example, memory system 150 may send a read response to progress tracker 134 indicating that a read operation has been completed. Each read response may include a request ID of the read request that is sending the response. In this way, progress tracker 134 can use the request ID to track the progress of a DMA transaction.

[0040] Destination sub-thread 140 includes a request generator 142 that generates write requests based on descriptor 112. As described in more detail below, request generator 142 may, for example, generate multiple memory addresses in parallel per clock cycle and generate a write request for each memory address. For example, when request generator 142 traverses a multidimensional tensor, it may generate a corresponding memory address for each of the multiple tensor elements during a single clock cycle of kernel 110. Each write request may include a request ID and specify the memory address where data is to be written.

[0041] Destination sub-thread 140 also includes a progress tracker 134 that tracks the progress of a write operation specified by a write request. For example, memory system 150 may send a write response to progress tracker 144 signaling that a write operation has been completed. Each write response may include a request ID for the write request for which it is sending the response. In this way, progress tracker 144 can use the request ID to track the progress of a DMA transaction.

[0042] Progress trackers 134 and 144 can send synchronization messages 115 and 116 to kernel 110, respectively, to update kernel 110 during the DMA transaction corresponding to descriptor 112. Synchronization messages 115 and 116 can specify the completion level (e.g., the percentage or total number of memory operations completed) and / or the request ID of a response that has been received.

[0043] As described below, progress trackers 134 and 144 can send synchronization messages 115 and 116, which provide partial or incomplete updates on the progress of DMA transactions. For example, each progress tracker 134 and 144 can be configured to send synchronization messages 115 and 116 each time a specified number (e.g., a threshold number) of responses have been received for a DMA transaction. In a particular example, each progress tracker 134 and 144 can be configured to send synchronization messages 115 and 116 each time a response has been received for a consecutive sequence of at least a threshold number of request IDs. Because kernel 110 knows the order in which memory operations are being performed (and therefore the order in which tensor elements are being moved), kernel 110 can begin processing the data that has already been transferred based on these partial updates without waiting for the entire set of DMA transactions to complete.

[0044] Using separate child threads for read and write operations can achieve higher throughput. For example, if each child thread 130 and 140 can generate a certain number of requests in parallel each clock cycle, say four requests per clock cycle, then the total number of requests generated by the two child threads 130 and 140 is twice that specific number, say eight requests.

[0045] In some cases, multiple DMA threads can be used to perform DMA transactions. For example, if the memory bandwidth is sufficient to handle more requests per clock cycle than a single DMA thread could generate, multiple DMA threads can be used to generate requests. If multiple DMA threads are used to transfer data from a multidimensional tensor, each DMA thread can receive a descriptor for a portion of the multidimensional tensor (e.g., a slice of the tensor). The descriptor can specify the size and shape of the slice of the tensor, as well as the memory address, similar to a descriptor for the entire tensor.

[0046] Figure 2A It can be used to implement Figure 1 A diagram of an example request generator 200 for each of the request generators 132 and 142. In this example, request generator 200 is configured to have an implementation with up to four tensor dimensions and capable of generating up to four memory addresses per clock cycle.

[0047] Typically, the request generator 200 determines the memory addresses of tensor elements within a multidimensional tensor or other multidimensional data structure (referred to as a tensor herein for brevity). The request generator 200 determines memory addresses such that tensor data can be read from and / or written to memory. The request generator 200 can compute the memory address of a tensor element based on the step index value of the tensor element that defines the position of the tensor element within the tensor. An example request generator 200 is implemented using a five-stage design with pipelined registers 220, 230, 240, 260, and 270 between adjacent stages.

[0048] To determine memory addresses, the request generator 200 can iterate through each dimension by progressively traversing each step index value for each dimension. For example, if the dimension comprises ten elements, the request generator 200 can progressively traverse the step index values ​​in order from one to ten. Conceptually, this can be performed using nested loops that include loops for each dimension of the tensor. In such an example, the tensor's dimensions can be traversed using its loops by incrementing the step index value for each iteration of the loop until a loop boundary equal to the number of elements in the loop is reached. When the loop boundary is reached, the next outer loop is incremented, and the current loop is reset to the first step index value corresponding to the first element in the dimension. The innermost loop may include memory address calculations to determine the memory address of the tensor element at the position corresponding to the step index values ​​of the four loops in the nested loops within the tensor. Figure 2B Example pseudocode 280 is shown for using four loops to determine the memory address.

[0049] See Figure 2BPseudocode 280 includes four loops 281-284 for traversing the four dimensions of the tensor. The illustrated pseudocode 280 describes half of a transaction (source-side read or destination-side write). For the entire transaction, the same or similar pseudocode can be instantiated twice independently. In pseudocode 280, the loop boundary (steps_per_stride) for each dimension is the same for both the source and destination sides of the transfer, but the stride offset value (stride_dimension_offset_value_i) can be different. That is, steps_per_stride_0 is the same in the pseudocode for both the source and destination sides, but stride_dimension_offset_value_i in the source pseudocode can be different from stride_dimension_offset_value_i in the destination pseudocode.

[0050] The outermost loop 281 corresponds to one of the dimensions and includes a step index value i0 and a loop boundary steps_per_stride_0. The loop boundary steps_per_stride_0 can be equal to the number of elements in the dimension corresponding to the outermost loop 281. Similarly, loop 282 corresponds to one of the dimensions and includes a loop boundary with a step index value i1 and steps_per_stride_1 (which can be equal to the number of elements in the dimension corresponding to loop 282), and loop 283 corresponds to one of the dimensions and includes a loop boundary with a step index value i2 and steps_per_stride_2 (which can be equal to the number of elements in the dimension corresponding to loop 283).

[0051] The innermost loop 284 also corresponds to one of the dimensions and includes a stride index value i3 and a loop boundary steps_per_stride_3 (which can be equal to the number of elements in the dimension corresponding to the innermost loop 284). For each iteration of the innermost loop, the dimension memory address offset value is calculated for each dimension of the tensor using function 285, and these dimension memory address offset values ​​are used to determine the memory addresses of the tensor elements corresponding to stride index values ​​i0-i3 using function 286. The dimension memory address offset value (destination_memory_address_offset_0) for the dimension corresponding to the outermost loop 281 is equal to the product of the stride index value i0 for that loop and the stride dimension offset value (stride_dimension_offset_value_0) for that dimension. The dimension memory address offset value is determined for each of the other dimensions in a similar manner, as follows: Figure 2B As shown above, the stride dimension offset value for a dimension can be included in the descriptor.

[0052] The memory address for a tensor element can then be calculated based on the base memory address and the dimension memory address offset for each dimension of the tensor. For example, the memory address for a tensor element can be based on (e.g., equal to) the sum of the base memory address and the dimension memory address offset for each dimension, such as... Figure 2B As shown.

[0053] Return to Figure 2A The request generator 200 can perform similar memory address calculations in parallel, for example, without requiring an actual iterative loop. In this example, the request generator 200 includes four channels 201-204 for calculating, for example, four memory addresses in parallel within one clock cycle. In other examples, two or more channels can be used to calculate two or more memory addresses in parallel; for example, three channels for three memory addresses, five channels for five memory addresses, and so on. That is, the request generator 200 can include m channels to calculate m memory addresses in parallel, where m is greater than or equal to 1. The request generator 200 can calculate m memory addresses during a parallel memory address calculation cycle, which can have a duration less than or equal to a single clock cycle.

[0054] The number of channels can be the same as or different from the number of dimensions of the tensor. For example, request generator 200 can be used to compute memory addresses for tensors with different numbers of dimensions based on information included in descriptor 112. For example, request generator 200 with four channels can use up to all four channels to compute up to four memory addresses per cycle for a three-dimensional tensor. The same request generator 200 can also use up to all four channels to compute up to four addresses per cycle for one-dimensional, two-dimensional, or four-dimensional tensors.

[0055] Performing such computations in parallel across multiple levels and steps (four levels in this example) can be challenging because each channel 201-204 must compute the memory address of a tensor element that differs from each of the other channels, and each channel operates independently of the others. When each channel 201-204 computes memory addresses in parallel (e.g., simultaneously), one channel cannot wait for another channel to complete and then iterate through one or more loops to determine the memory address for the next tensor element. Instead, each channel must be able to determine its next tensor element (e.g., the step size index value for that tensor element) and the memory address for that tensor element without waiting for another channel.

[0056] For each channel 201-204 (and therefore for each parallel memory address computation), request generator 200 includes memory address units 242-248. Each memory address unit 242-248 includes a corresponding step size tracker 222-228 and a corresponding memory address computation element 252-258. Typically, step size trackers 222-228 are configured to step through the tensor elements of the tensor and determine the dimension memory address offset value for the tensor element. Memory address computation elements 252-258 are configured to use the dimension memory address offset value received from step size trackers 222-228 to determine the memory address for the tensor element.

[0057] Request generator 200 includes computational element 210 that pre-computes values ​​for step size trackers 222-228. For example, computational element 210 may pre-compute various step size comparison values ​​that can be used by step size trackers 222-228 to determine the next step long index value for the next tensor element for which a memory address will be determined. As described below, the current step size index value can be compared with the step size comparison values ​​along with other criteria to determine the next step size index value. Computational element 210 may pre-compute step size comparison values ​​for each dimension of the tensor. These step size comparison values ​​may be, for example, a step size minus 1 for that dimension, a step size minus 2 for that dimension, a step size minus 3 for that dimension, etc., depending on the number of dimensions of the tensor for which request generator 200 is generating a memory address and sending a request for the current descriptor 112. Computational element 210 is optional, and so are the pre-computed values. Pre-compiling these values ​​can help improve critical path timing on the next clock cycle.

[0058] Computation element 210 may include a set of hardware adders that pre-compute step size comparison values ​​and store them in register 220 (or other suitable data storage element). Computation element 210 may compute comparison offset values ​​based on the per-step step size value received in the descriptor. The descriptor may include per-step step size values ​​for one or more dimensions. In this example, the descriptor may include per-step step size values ​​for dimensions 1-3 (sps_1 to sps_3) instead of per-step step size values ​​for dimension 0 (e.g., the dimension corresponding to the outermost loop). For example, if the per-step step size variable is represented using a 32-bit signed integer, it can be implied that the per-step step size value for dimension 0 is a maximum integer value, for example, a maximum integer value that can be stored in a signed 32-bit integer. In another example, the per-step step size value may be included in the descriptor, but... Figure 2A Not shown in the image.

[0059] Since the step size value can vary based on the size and shape of the tensor, computing element 210 can pre-calculate the step size comparison value for each descriptor and store the step size comparison value in register 220. The descriptor can also be stored in register 220.

[0060] The request generator 200 also includes a finite state machine (FSM) 232. The FSM 232 can initialize and control the step trackers 222-228 based on information from descriptor 112. For example, the FSM 232 can obtain descriptor information from register 230 and determine, based on the descriptor information, the number of requests to be sent for the DMA transaction defined by the descriptor. This number can be the number of tensor elements in a tensor. The FSM 232 can track the number of remaining requests to be sent and send an advance based on that number of remaining requests to each step tracker 222-224. The advance defines the number of memory addresses to be calculated during the next cycle of memory address calculation performed by memory address calculation elements 252-258.

[0061] For example, during the execution of a DMA transaction using all four channels 201-204, the advance can be equal to 4. However, if the total number of memory addresses to be calculated for the DMA transaction is less than 4, the advance in the last cycle will be less than 4. For example, if the total number of memory addresses is 18, the FSM 232 will provide an advance of 4 to each step tracker 222-228 for the first 4 cycles, and then provide an advance of 2 in the last cycle.

[0062] FSM 232 can also stop step tracker 232. For example, as described below, progress trackers 134 and 144 can track the progress of only a specific number of requests at a time. When request generator 200 runs out of allocated request IDs, request generator 200 can stop itself, for example, by stopping step tracker 232. When request IDs are released and can be reassigned, for example, when responses are received for at least a threshold number of sequential request IDs as described below, progress trackers 134 and 144 can return request ID credits.

[0063] Request generators 132 and 142 may also stop due to external interconnect backpressure (i.e., the memory system cannot yet accept new requests). In some implementations, a software-configurable hardware FSM can be used to independently throttle each DMA thread 120. The software can set a target request generation bandwidth for each DMA thread 120 on a configurable sampling window, and once the allocated bandwidth is reached, the DMA thread 120 will automatically stop its pipeline. Therefore, the DMA thread 120 can stop under three different conditions: memory system network backpressure, request bandwidth throttling, and exhausted request ID allocation (waiting for the progress tracker to return credits).

[0064] Each step tracker 222-228 uses the advance received from FSM 232, the current step index value for each dimension of the tensor, and the per-step step value for each dimension to determine the next step index value for each dimension. Each step tracker 222-228 also determines the dimension memory address offset value for each dimension based on the next step index value for the dimension and the step dimension offset value for the dimension. Each step tracker 222-228 outputs the determined dimension memory address offset value to its corresponding memory address calculation element 252-258 via register 240. As described below, the memory address calculation element 252-258 determines the memory address for the tensor element based on the received dimension memory address offset value.

[0065] Step trackers 222-228 determine the dimension memory address offsets for tensor elements that are distinct from each other. For example, consider a 2×2×2×2 tensor (or other shape tensor) comprising 16 total tensor elements. Since the four channels 201-204 generate four requests per cycle, each step tracker 222-228 will determine the dimension memory address offsets for a total of four tensor elements out of the 16 tensor elements. For example, step tracker 222 might determine the dimension memory address offsets for the first, fifth, ninth, and thirteenth tensor elements, while step tracker 224 determines the dimension memory address offsets for the second, sixth, tenth, and fourteenth tensor elements, and so on.

[0066] Step trackers 222-228 can determine their respective dimension memory address offsets in parallel and independently of each other. That is, in some implementations, step trackers 222-228 do not transmit any data to any other step trackers 222-228. Instead, each step tracker 222-228 can be configured to determine its next quantitative element (e.g., the step index value for its next quantitative element) based on its initialization and the advance received from the FSM 232, as described in more detail below. In this way, none of the step trackers 222-228 need to wait for another step tracker 222-228, and all step trackers 222-228 complete parallel computations in a single clock cycle. An example architecture of step trackers and techniques for determining dimension memory address offsets is provided in [the following section / documentation]. Figure 3 , 4 As shown in 7 and described below.

[0067] Each memory address computation element 252-258 includes first summing elements 262A-268A and second summing elements 262B-268B. Each first summing element 262A-268A can determine the sum of the dimensional memory address offset values ​​received from its step size trackers 222-228 for each parallel memory address computation cycle. For example, summing element 262A can determine the sum of the four-dimensional memory address offset values ​​generated by step size tracker 222 for a given tensor element. The first summing elements 262A-268A can be implemented as hardware adders.

[0068] The second summing elements 262B-268B, which can also be implemented as hardware adders, can determine the memory address for a tensor element based on the base address and the sum of the dimension memory address offsets calculated by their corresponding first summing elements 262A-268A. For example, summing element 262B can determine the memory address for a given tensor element by adding the base address to the sum of the four dimension memory address offsets generated by step tracker 222 for a given tensor element.

[0069] The second summing elements 262B-268B can output their memory addresses to register 270. The request sender 290 can generate a request for each memory address and send the request to the memory system, for example... Figure 1The memory system 150. A request may include a request ID and a memory address. Request IDs may be assigned sequentially to requests. For example, if a DMA thread is configured to have 500 pending requests at a time, the request IDs can start from 0 or 1 and go up to 499 or 500 respectively. If 0-499 are used, the first request may include request ID 0, the second request may include request ID 1, and so on. The request sender 299 may include a counter for determining the request ID for each request.

[0070] Four channels 201-204 can each generate memory addresses for tensor elements in parallel during a single clock cycle. The FSM 232 can control the step size trackers 222-228 of channels 201-204 to iterate through each tensor element of the tensor until a memory address is calculated for each tensor element in the tensor. When a request for a descriptor is complete, the FSM 232 can move to the next descriptor. However, the FSM 232 does not have to wait to receive responses to all requests. If at least a threshold number of sequential request IDs are available (e.g., responses have already been received for them), the progress tracker 132 or 134 can notify the request generator 200, allowing the request generator 200 to issue a request for the next descriptor using those available request IDs. This further improves the throughput and efficiency of the DMA thread.

[0071] As described above, the request generators 132 and 134 for DMA child threads 132 and 134 can be implemented using request generator 200. In this example, each child thread 132 and 134 will be able to send four requests per clock cycle.

[0072] Figure 3 It can be used for implementation Figure 2A A diagram of an example step size tracker 300 for each of the step size trackers 222-228. In this example, step size tracker 300 includes two incrementer chains 322 and 324, which can perform the same or similar functions to generate step size index values ​​and dimension memory address offset values ​​for tensor elements. This allows one of the step size incrementer chains to actively determine the dimension memory address offset value of the current descriptor to be processed by the DMA thread, while the other step size incrementer chain is initialized for the next descriptor to be processed by that DMA thread.

[0073] For example, step incrementer chain 324 can proactively determine the dimension memory address offset value for the current descriptor. Step incrementer chain 324 can determine the dimension memory address offset value for the current descriptor using the advance received from the FSM (e.g., FSM 232 of Figure 2) and the step size parameter (as defined by the descriptor). When step incrementer chain 324 is active, the FSM can initialize step incrementer chain 322, as referenced below. Figure 4 As stated above.

[0074] When the FSM is in the last cycle requesting memory addresses for the current descriptor, it can switch to the initialized step incrementer chain 322 and send an initialization amount to it. Step incrementer chain 322 can generate the first set of dimension memory address offsets on the next clock cycle after step incrementer chain 324 determines its last set of dimension memory address offsets. Using two step incrementer chains in this way can significantly improve the throughput and efficiency of the DMA thread, especially for small tensors. For example, if the request generator only spends three clock cycles determining all memory addresses for that tensor, using clock cycles to reinitialize a single step incrementer chain between tensors results in a 25% reduction in throughput (e.g., the number of memory operations performed per unit time).

[0075] When switching between step incrementer chains 322 and 324, the FSM can control a set of multiplexers 332-338 to select which step incrementer chain's output is sent to the memory address calculation unit via register 342. For example, the FSM can select the top channel of each multiplexer 332-338 when incrementer chain 322 is active, and select the bottom channel of each multiplexer 332-338 when incrementer chain 324 is active.

[0076] As per the above reference Figure 2B Each channel 201-204 includes a step tracker that can be implemented as a step tracker 300. In this example, the step tracker 300 is for channel 0 and outputs a four-dimensional memory address offset value for channel 0.

[0077] Although not shown, each step tracker 300 can also output a next long index value for determining the dimension memory address offset value. These next long index values ​​are fed back to the step tracker 300 for use in determining subsequent step index values ​​and dimension memory address offset values. That is, the step incrementer chain 324 can determine the step index value and the dimension memory address offset value for each dimension. These values ​​can be fed back to the step incrementer chain 324 as the current values ​​to be used to determine the next value.

[0078] The step size tracker 300 may also include multiplexers for step size index values, which receive step size index values ​​for each dimension from both step size incrementer chains 322 and 324, similar to how multiplexers 322-338 receive dimension memory address offset values ​​from both step size incrementer chains 322 and 324. The outputs of these multiplexers can be fed into step size incrementer chain 324 for use in determining subsequent step size index values.

[0079] When step incrementer chain 324 calculates the dimension memory address offset for the current descriptor, step incrementer chain 322 can use the initialization state to determine the dimension memory address offset for the first set of memory addresses for the next descriptor. However, the FSM can control multiplexers 332-338 to pass the dimension memory address offset received from step incrementer chain 324. Upon completion of the current descriptor, the FSM can control multiplexers 332-338 to pass the dimension memory address offset calculated by step incrementer chain 322 within one cycle, which will include the values ​​of the first four tensor elements for the next tensor. The FSM can also control a multiplexer for the step index value to pass the step index value from step incrementer 322 to step incrementer chain 324 within that one cycle. Afterward, step incrementer chain 324 has the current state of the step index value and can determine the dimension memory address offset for the remaining cycles of the descriptor. After the first cycle of the descriptor is completed, the FSM can control the multiplexer to pass the output of the step incrementer chain 324 again.

[0080] Figure 4 This is a diagram of an example step incrementer chain 400. Step incrementer chain 400 can include step incrementers for each dimension of the largest tensor that the DMA thread is configured to process. In this example, step incrementer chain 400 includes four step incrementers 410-440 for tensors with up to four dimensions. Figure 4 The example step incrementer chain shown is implemented as a combinational function, resembling a carry ripple adder circuit in style.

[0081] Each step incrementer 410-440 can receive a parameter set. The parameter set for step incrementers 410-440 may include the step size per step corresponding to the dimension of step incrementers 410-440 and a step size comparison value for each dimension pre-calculated by computing element 210. Step incrementers 410-440 can be initialized for each descriptor when these values ​​can be changed based on the size and shape of the tensor performing the DMA transaction.

[0082] Each step incrementer 410-440 can also receive a step index value for a dimension and a dimension offset value for that dimension. For the first cycle, the step index value for that dimension can be initialized to zero (as shown by the input values ​​of step incrementer chain 322). After the first cycle, the step index value input to step incrementers 410-440 is the next step index value output by step incrementers 410-440. As mentioned above, the dimension offset value for a dimension is determined by multiplying the step index value. (This is related to the use of...) Figure 2B Compared to the four loops in pseudocode 280, the step incrementer 410 functions similarly to the innermost loop of nested loops. However, instead of incrementing the step index by 1 on each iteration of that loop, the step incrementer 410 increments its step index value based on the advance received from the FSM. For example, if the advance is 4, the step incrementer 410 increments the step index value for that dimension by 4. If this increment exceeds the step size per stride for that dimension, the step incrementer can reinitialize the step index value to zero and continue incrementing until it has been incremented four times, which may include more than one reinitialization. For example, if the step size per stride is 3 and the advance is 4, the step incrementer 410 will increment from 0 to 3, reinitialize to 0, and then increment from 0 to 1 after four increments.

[0083] Instead of using state iteration, the step incrementer 410 can use a combination function that behaves similarly to a pair of optimized adders. Like an adder, part of the step incrementer takes two operands ("step 3 index" and "advance_amount") and produces a sum ("step 3 next index") and a carry ("wrap amount"). The functionality is similar for dimension offsets, except that the function that calculates the next dimension offset does not produce a wrap amount output.

[0084] Step incrementer 410 can output the wrap-around amount to step incrementer 420. The wrap-around amount can be equal to the number of times the step index value of step incrementer 410 will be reinitialized in the current loop based on the received lead. That is, the wrap-around amount reflects the number of times the four loops will wrap around based on the lead.

[0085] For each cycle in which step incrementers 410-440 calculate their dimension memory address offset values, for example, during a single clock cycle, step incrementer 410 may calculate the next long index value for its dimension, the wraparound amount for step incrementer 420, and the dimension memory address offset value (e.g., the product of the next long index value and the dimension offset value for that dimension).

[0086] Step incrementer 420 can use the wraparound amount received from step incrementer 410, similar to how step incrementer 420 uses the advance amount received from FSM. That is, the wraparound amount represents the number of times the step index value of the dimension corresponding to step incrementer 420 will be incremented in that loop. Step incrementer 420 can use the wraparound amount received from step incrementer 410 to increment its step index value to determine the next long index value. Step incrementer 420 can also use the next long index value (e.g., the product of the next long index value and the step dimension offset value for that dimension) to determine its dimension memory address offset value.

[0087] Similar to step incrementer 410, step incrementer 420 can also calculate the wrap-around amount and output it to step incrementer 430. The wrap-around amount can be equal to the number of times the step index value of step incrementer 420 is reinitialized in the current loop based on the wrap-around amount received from step incrementer 410. In other words, the wrap-around amount reflects the number of times the four loops will wrap around based on the received wrap-around amount.

[0088] Step incrementer 430 can use the wrap-around amount received from step incrementer 420 in a similar manner. That is, the wrap-around amount represents the number of times the step index value of the dimension corresponding to step incrementer 430 will be incremented in that loop. Step incrementer 430 can use the wrap-around amount received from step incrementer 420 to increment its step index value to determine the next long index value. Step incrementer 430 can also use the next long index value (e.g., the product of the next long index value and the step dimension offset value for that dimension) to determine its dimension memory address offset value.

[0089] Similar to step incrementer 420, step incrementer 430 can also calculate the wrap-around amount and output it to step incrementer 440. The wrap-around amount can be equal to the number of times the step index value of step incrementer 430 is reinitialized in the current loop based on the wrap-around amount received from step incrementer 420. That is, the wrap-around amount reflects the number of times the four loops are wrapped based on the received wrap-around amount.

[0090] Step incrementer 440 can use the wrap-around amount received from step incrementer 430 in a similar manner. That is, the wrap-around amount represents the number of times the step index value of the dimension corresponding to step incrementer 440 will be incremented in that loop. Step incrementer 440 can use the wrap-around amount received from step incrementer 430 to increment its step index value to determine the next long index value. Step incrementer 440 can also use the next long index value (e.g., the product of the next long index value and the step dimension offset value for that dimension) to determine its dimension memory address offset value.

[0091] For each cycle in which step incrementers 410-440 calculate their dimension memory address offset values, for example, during a single clock cycle, step incrementers 410-440 may each calculate the next long index value for their dimension, the wraparound amount for the next long incrementer (if appropriate), and the dimension memory address offset value (e.g., the product of the next long index value for that dimension and the step dimension offset value).

[0092] In some implementations, each incrementer 410-440 may use a set of criteria to determine the next long index value for its dimension and / or the amount of wraparound for its dimension. The criteria may include an increment (e.g., an advance for incrementer 410 or a wraparound for incrementers 420-440). The criteria may also include the step size per stride for that dimension, and a comparison of the current step size index value with the step size comparison value.

[0093] For example, a table, such as a lookup table, can be generated that specifies, for each specific combination of the increment, the step size per step, and which step size comparison value the current step size index value matches, what the next long index value will be and what the wraparound amount will be. The specific combination can vary based on the number of dimensions of the tensor for which the request generator can generate memory addresses. In this way, each step incrementer 410-440 can simply compare the increment and the current step size index value with the table to determine what the next long index value and wraparound amount will be.

[0094] Figure 5 It can be used to implement Figure 2A A diagram of example progress tracker 500 for each of progress trackers 134 and 144. Progress tracker 500 includes progress tracker queue 510, response reordering unit 520, and synchronization unit 530.

[0095] The progress tracker queue 510 can receive descriptors (or relevant portions of descriptors that require response and synchronization processing) and store the descriptors. The descriptors enable the synchronization unit 530 to determine the progress of the DMA transaction defined by the descriptor, as described below.

[0096] The response reordering unit 520 can receive data from the memory system (e.g., Figure 1 The memory system 150 receives responses. Each response can specify a request ID corresponding to the request. That is, the memory system can send a response to the progress tracker 500 for each completed request received from the request generator corresponding to the progress tracker 500.

[0097] The response reordering unit 520 can receive responses in any order and reorder them based on their request IDs. The memory system can process requests in an order different from the order in which they are received. For example, the memory system can use bandwidth optimization techniques to prioritize some requests over others. Therefore, the response reordering unit 520 can be configured to receive out-of-order responses and reorder them to track the progress of memory operations performed by the memory system. An example response reordering unit is shown in... Figure 6 It is shown in the figure and described in more detail below.

[0098] Synchronization unit 530 can receive progress data from response reordering unit and send synchronization messages to kernel, for example... Figure 1 The kernel 110. For example, synchronization unit 530 can receive data from response reordering unit 520 specifying the number of ordered request IDs that have been received. Synchronization unit 530 can be configured to send a synchronization message each time at least a threshold amount (or a threshold percentage) of memory operations defined by the descriptor has been completed. For example, synchronization unit 530 can determine the total number of memory operations to be performed on the current descriptor (e.g., depending on whether a child thread is reading or writing). Synchronization update unit 530 can be configured to send a synchronization message to the kernel each time at least 10% of the memory operations have been completed. As described above, the kernel can use these partial updates to begin consuming the transferred data without waiting for all memory operations defined by the descriptor to complete.

[0099] The response reordering unit 520 and / or synchronization unit 530 can be configured to notify the request generator that a set of request IDs can be reused. For example, each time at least a threshold number of ordered request IDs are received in a response from the memory system, these request IDs can be released to the response generator for reuse. This allows the request generator to continue generating requests after it has sent the maximum number of requests that can be processed by the progress tracker 500, but before all requests have been completed.

[0100] For example, suppose progress tracker 500 includes a response reordering buffer that can only track 500 memory operations at a time, and the request IDs are 0-499. If all 500 memory operations have been used for a request and no request has been responded to, the request generator must stop until it receives a notification from progress tracker 500 specifying an available request ID. If progress tracker 500 receives a response for request IDs 0-15 (but not all identifiers), and the threshold is less than 15, progress tracker 500 can send a notification (e.g., a request ID credit return message) specifying that the request generator can continue sending requests using request IDs 0-15 without waiting for all 500 memory operations to complete.

[0101] Figure 6 It can be used to implement Figure 5 A diagram of an example response reordering unit 600 is provided for response reordering unit 520. Response reordering unit 600 includes a response vector 610 and a reorder vector 630, each of which can be implemented using a bit vector register. Response vector 610 and reorder vector 630 may each include bits for each request ID that can be issued by the request generator. This bit can indicate the status of the request ID. For example, if the value of the bit is zero, no response to a memory operation has been received. If the value of the bit is 1, a response to a memory operation has been received. Response vector 610, reorder vector 630, and pop vector (described below) can all be of the same size, for example, including the same number of bits.

[0102] Response vector 610 can be configured to receive multiple responses at once, for example, up to four responses at once in this example. For instance, response vector 610 can be configured to receive multiple simultaneous responses matching the number of channels of the corresponding request generator. In other examples, response vector 610 can be configured to receive multiple simultaneous responses that are different from (e.g., greater than) the number of channels of the corresponding request generator.

[0103] The bits in the reorder vector 630 can be arranged in the order of the request IDs. At the input of the reorder vector 630 is a logic OR gate 624. The OR gate 624 can be a bit vector OR gate that includes OR gates for each bit of the reorder vector 630. For each request ID, the bits of the response vector of the request ID and the bits of the request ID output by the AND gate 622 (e.g., a bit vector AND gate) can be the inputs of the OR gate to determine the value of the bits of the request ID in the reorder vector 630.

[0104] The AND gate 622 has a pair of inputs for each request ID, and therefore a pair of inputs for each bit of the reorder vector 630. For a given request ID, the output of the AND gate is 1 if both the bit in the reorder vector 630 and the pop bit of the request ID in the pop vector maintained by the pop vector logic 640 have a value of 1. As described below, for example, when the request ID is released for use by the request generator, the pop bit of the memory address can be set to 1 to clear that bit back to a value of 0. That is, if a response to a request ID is received and the request ID has not yet been released, the output of the AND gate 622 for the bit corresponding to the request ID will be 1. If the request ID has been released, the output of the AND gate 622 for that bit will be zero because the input from the pop vector will be 1.

[0105] The response reordering unit 600 also includes head pointer logic 650, pop vector logic 640, and internal pop counter logic 660. The head pointer logic 650 can maintain a pointer to the next bit after the bit in the reordering vector 630 for the highest ordered request ID that has received a response. The ordered request IDs can start at the first request ID and extend to each ordered request ID that has received a response, until a request ID for which a response has not yet been received is reached. For example, if request IDs include 0-499 and responses for 0-8, 11, 56, and 61-78 have been received, the ordered request IDs would be 0-8. In this example, the head pointer would point to the bit for request ID 9. Once responses for request IDs 9 and 10 are received, the ordered request IDs would be 0-11, assuming a response for request ID 12 has not yet been received.

[0106] The head pointer logic 650 can also pre-compute additional head pointers, such as incrementing the head pointer by 1 (e.g., the next bit after the bit the head pointer points to, incrementing the head pointer by 2, etc.). In this way, the pop counter logic 660 can pop more than one bit from the reorder vector 630 during a single clock cycle. This is an optional feature that can be used to satisfy timing for a given clock frequency. For large reorder vectors with many responses per cycle, the logic complexity increases significantly. This pre-computation can be used when the target frequency is relatively fast or when there are many responses per cycle (e.g., many channels).

[0107] Internal pop counter logic 660 can monitor the bits in reorder vector 630 to determine how many bits of the reorder vector can be popped (e.g., cleared) as the head pointer moves. For example, internal pop counter logic 660 can predict any bit string with a value of 1, indicating that a memory operation for the corresponding bit has been responded to. When head pointer logic 650 moves to another bit, head pointer logic 630 can provide the position (e.g., bit) that the head pointer has moved to (e.g., head_plus_i_next). Based on the new position of the head pointer and the monitored bits, internal pop counter logic 660 can determine how many bits can be popped, e.g., up to a maximum number of pops per clock cycle. For example, if the head pointer can move up ten bits and the maximum number of pops per clock cycle is 4 bits, internal pop counter logic 660 can instruct pop vector logic 640 to pop 4 bits in the first cycle, 4 bits in the second cycle, and 2 bits in the third cycle. The head pointer will increment by the same number of entries popped in that cycle, so in this example, it can advance up to 4 bits per cycle.

[0108] Pop vector logic 640 can maintain a pop vector for the bits to be popped and provide this pop vector as an input to AND gate 622. Pop vector logic 640 can determine the bit to be popped based on the head pointer, an additional head pointer, and the number of bits to be popped received from internal pop counter logic 660. For example, if the number of bits to be popped is 4, pop vector logic 640 can pop the bit from the current head pointer to the head pointer plus 4. Since the value of the head pointer plus 4 has already been calculated, pop vector logic 640 does not need to consume clock cycles to determine the position of the bit to be popped.

[0109] The head pointer logic 650 can also receive the number of bits to be popped from the pop counter logic. The head pointer logic 650 can update the head pointer and pre-calculate additional head pointers based on the number of bits to be popped.

[0110] The response reordering unit 600 also includes an ordered item register 672 and calculation elements 670 and 674. The ordered item register 672 can maintain a count of the number of ordered items that have been popped but not yet released to the request generator. To this end, the calculation unit 670 aggregates the number of popped bits based on the output of the internal pop count logic 660.

[0111] The number of ordered items in register 672 is also sent to synchronization unit 530. Synchronization unit 530 can determine when to release request IDs to the request generator based on the number of ordered items that have been popped. For example, the synchronization unit can send data specifying the amount of request IDs that can be used by the request generator, such as request ID credits returned. Calculation unit 674 can subtract this number from the current number of ordered items in register 672 and update the register with the result (plus any newly popped items from internal pop counting logic 660). For example, if register 672 indicates that 15 ordered items have been popped, and synchronization unit 530 releases 10 to the request generator, calculation unit 674 can subtract the 10 released request IDs from the 15 request IDs and store and update the values ​​of the remaining 5 request IDs. In this way, register 272 stores a running count of the number of request IDs that can be released to the request generator.

[0112] Figure 7 This is a flowchart illustrating an example process 700 for generating a memory operation request. Process 700 can be generated by a request generator (e.g., Figure 1 The request generator 132 or 142 or the request generator 200 in Figure 2 is executed.

[0113] The request generator receives one or more descriptors (702). Each descriptor includes information about a DMA transaction (e.g., a set of memory transfer operations). For example, a descriptor may include information specifying a source memory from which data will be read, a destination memory to which data will be written, the size and shape (e.g., dimension) of the source tensor to which tensor elements are stored in the source memory, the size and shape of the destination tensor to which tensor elements are stored in the destination memory, and a stride dimension offset value for each dimension.

[0114] The request generator initializes the step size tracker (704). As described above, the request generator may include multiple channels, each of which computes a memory address in parallel, for example, per clock cycle. Each channel may include a step size tracker and a memory address computation unit. Each step size tracker may include a chain of step size incrementers. Initializing the step size tracker may include providing a step size parameter to each step size incrementer and initializing a step size index value for each step size incrementer.

[0115] The request generator generates memory addresses (706). For example, the request generator can use multiple channels to compute multiple memory addresses in parallel, e.g., during a single clock cycle. Specifically, during a clock cycle, each step tracker can compute the next long index value for each dimension of the tensor (which corresponds to a specific tensor element in the tensor), and use the next long index value for that dimension and the step dimension offset value for that dimension to compute the dimension memory address offset value for each dimension. The memory address computation unit for each channel can then compute the memory address based on the dimension memory address offset value for each dimension output by the step tracker on the channel and the base address. For example, the memory address for a channel (and therefore a tensor element) can be the sum of the base address and the dimension memory address offset value.

[0116] The request generator generates a request and sends it to the memory system (708). The request can be a read request or a write request. Each request can specify a request ID and a memory address calculated during that cycle. That is, the request generator can generate and send a corresponding request for each computer memory address. The request generator can then send the request to the memory system that performs a read or write operation using the memory address in the request.

[0117] The request generator determines if there are more tensor elements (710) for which memory addresses are computed. For example, as described above, the FSM can keep track of the number of remaining requests to be generated for the descriptor. If more tensor elements exist, process 700 returns to step 706 to generate more memory addresses.

[0118] If no additional tensor element exists, the request generator determines whether an additional descriptor exists for which a DMA transaction is to be performed (714). For example, the request generator may check the descriptor queue to determine if any additional descriptors exist in the queue. If not, the process ends. If yes, the process returns to step 704, where a step size tracker is initialized for the next descriptor. As mentioned above, the step size tracker can be initialized before the DMA transaction for the previous descriptor is completed.

[0119] Figure 8 This is a flowchart illustrating an example process 800 for tracking the progress of memory operations. Process 800 can be provided by a progress tracker (e.g., Figure 1 Progress tracker 134 or 144 or Figure 5 The progress tracker (500) is executed.

[0120] The progress tracker receives one or more responses (802). For example, a memory system may send a response to the tracked progress in response to the completion of a memory operation. The response may specify a request ID for the completed memory operation.

[0121] The progress tracker updates the reorder vector (804). The progress tracker can update the reorder vector to indicate that a memory operation corresponding to a request ID has been completed. For example, the progress tracker can update the bit for the request ID from value 0 to value 1 to indicate that a memory operation corresponding to the request ID has been completed.

[0122] The progress tracker determines whether the number of consecutive elements (e.g., bits for the request ID) is greater than or equal to a threshold (806). If so, the progress tracker can release the request ID for reuse by the request generator (808). If not, process 800 continues to step 810 without releasing any request IDs.

[0123] In step 810, the progress tracker determines whether the number of received responses is greater than or equal to a threshold. This number could be the number of responses received since a previous synchronization message was sent to the kernel performing memory operations. In another example, the progress tracker could determine whether at least a threshold percentage of the total number of responses to be received has been received.

[0124] In either example, if a threshold has been reached or exceeded, the progress tracker can synchronize with the kernel (812). For example, the progress tracker can send a synchronization message to the kernel indicating the total number or percentage of responses received. In another example, the progress tracker can send a synchronization message to the kernel indicating the number of responses received since a previous synchronization message was sent to the kernel.

[0125] If the threshold has not yet been reached, process 800 continues to step 814. In step 814, the progress tracker determines whether all responses for the descriptor have been received. If not, process 800 returns to step 802, where more responses are received. If so, the progress tracker can synchronize with kernel 816, for example, by sending a synchronization message indicating that all memory operations for the descriptor have been completed.

[0126] While this specification contains numerous details of specific implementations, these should not be construed as limiting any invention or potentially claimed scope, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may in some cases be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0127] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0128] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A direct memory access (DMA) system, comprising: One or more hardware DMA threads, wherein each DMA thread includes: A request generator is configured to, during each parallel memory address computation cycle, (i) generate m memory addresses for a multidimensional tensor in parallel, and for each memory address, (ii) generate a corresponding request to perform a memory operation on the memory system for the multidimensional tensor, wherein the request generator comprises m memory address units, and wherein each memory address unit comprises: A step size tracker is configured to generate memory address offset values ​​for corresponding tensor elements of the multidimensional tensor during each parallel memory address computation cycle. The generation includes determining (i) a corresponding step size index value for each dimension of the multidimensional tensor, and (ii) determining a corresponding memory address offset value for that dimension based on the corresponding step size index value, wherein the corresponding step size index value for a dimension of the multidimensional tensor corresponds to the position of the corresponding tensor element within the multidimensional tensor; and Memory address calculation element, the memory address calculation element being configured to: During each parallel memory address computation cycle, and based on the corresponding memory address offset for each dimension, the memory address of the corresponding tensor element of the multidimensional tensor is generated; and A request to perform the memory operation using the memory address is transmitted to the memory system; Each step tracker generates a step index value and a memory address offset value for a tensor element that is different from each of the other step trackers during each parallel memory address calculation cycle. Where m is greater than or equal to 2.

2. The DMA system as described in claim 1, wherein, The request generator is configured to generate the memory address in parallel during a single clock cycle and to perform each parallel memory computation during a single clock cycle.

3. The DMA system as described in claim 1, wherein, The request generator is configured to receive descriptors for the multidimensional tensor, the descriptors defining a corresponding step size for the step value of the dimension for each dimension.

4. The DMA system as described in claim 1, wherein, The request generator includes m channels, each channel including a corresponding step size tracker and a corresponding memory address calculation element, wherein the corresponding step size tracker and the corresponding memory address calculation element of each channel calculate the corresponding memory address in parallel with each other channel.

5. The DMA system as described in claim 4, wherein: The step size tracker is configured to generate memory addresses for the multidimensional tensor based on nested loops, the nested loops for each dimension of the multidimensional tensor including corresponding loops for traversing the dimensions of the multidimensional tensor; and The step size value for each dimension represents the loop boundary of the corresponding loop for that dimension, and the step size index value for each dimension represents the loop index of the corresponding loop for that dimension.

6. The DMA system as described in claim 5, wherein, Each step tracker is configured to update the step index value for each of the dimensions during each clock cycle.

7. The DMA system as described in claim 5, wherein, The combination of step index values ​​for each step tracker is different from the combination of step index values ​​for each other step tracker.

8. The DMA system as claimed in claim 7, wherein: Each step tracker includes a chain of step incrementers, which includes multiple step incrementers, each configured to determine a dimension memory address offset value for the corresponding dimension. The first step incrementer of the step incrementer chain corresponding to the innermost loop of the nested loop is configured to receive advance. as well as Updating the step size index value for one or more of the dimensions during each clock cycle includes updating the step size index value for the one or more dimensions by the first step size incrementer based on the lead amount.

9. The DMA system as claimed in claim 8, wherein: Each of one or more second step incrementers in the step incrementer chain corresponding to the loop in which the innermost loop is nested is configured to receive a wraparound amount from a previous step tracker in the step incrementer chain. as well as Updating the step index value for one or more of the dimensions during each clock cycle includes updating the step index value for the one or more dimensions by the second step incrementer based on the wrap-around amount.

10. The DMA system of claim 1, further comprising a progress tracker, the progress tracker including a response reordering unit and a synchronization unit.

11. The DMA system of claim 10, wherein, The response reordering unit is configured to maintain a state for each tensor indicating whether a memory operation has been performed on the tensor element.

12. The DMA system of claim 10, wherein, The synchronization unit is configured to provide a plurality of partial updates to the processor core, each of which specifies the overall state of memory operations performed on the tensor elements of the multidimensional tensor.

13. The DMA system of claim 10, wherein: Each request includes a unique identifier; The response reordering unit is configured as follows: Responses may be received from the memory system in any order, each response including a unique identifier of the request for which the response was provided; as well as When at least a threshold number of consecutive unique identifiers are received in the response, the set of unique identifiers is released for reuse by the request generator.

14. A system for a direct memory access (DMA) architecture, comprising: One or more processor cores; Memory system; as well as A DMA engine, comprising one or more DMA threads, wherein each DMA thread includes: A request generator is configured to, during each parallel memory address computation cycle, (i) generate m memory addresses for a multidimensional tensor in parallel, and for each memory address, (ii) generate a corresponding request to perform a memory operation on the memory system for the multidimensional tensor, wherein the request generator comprises m memory address units, where m is greater than or equal to 2, and wherein each memory address unit comprises: A step size tracker is configured to generate memory address offset values ​​for corresponding tensor elements of the multidimensional tensor during each parallel memory address computation cycle. The generation includes determining (i) a corresponding step size index value for each dimension of the multidimensional tensor, and (ii) determining a corresponding memory address offset value for that dimension based on the corresponding step size index value, wherein the corresponding step size index value for a dimension of the multidimensional tensor corresponds to the position of the corresponding tensor element within the multidimensional tensor; and Memory address calculation element, the memory address calculation element being configured to: During each parallel memory address computation cycle, and based on the corresponding memory address offset for each dimension, the memory address of the corresponding tensor element of the multidimensional tensor is generated; and A request to perform the memory operation using the memory address is transmitted to the memory system. Each step tracker generates a step index value and a memory address offset value for tensor elements that are different from those of each other step tracker during each parallel memory address computation cycle; and A progress tracker, comprising a response reordering unit and a synchronization update unit, the synchronization update unit being configured to provide partially synchronized updates to the one or more processor cores for memory operations managed by the DMA engine.

15. The system of claim 14, wherein, The request generator is configured to generate the memory address in parallel during a single clock cycle and to perform each parallel memory computation during a single clock cycle.

16. The system of claim 14, wherein, The request generator is configured to receive descriptors for the multidimensional tensor, which define a corresponding step size for the step value of each dimension.

17. The system of claim 14, wherein, The request generator includes m channels, each channel including a corresponding step size tracker and a corresponding memory address calculation element, wherein the corresponding step size tracker and the corresponding memory address calculation element of each channel calculate the corresponding memory address in parallel with each other channel.

18. A method performed by a DMA system, the method comprising: During each parallel memory address calculation cycle, the request generator (i) generates m memory addresses for a multidimensional tensor in parallel, and for each memory address, (ii) generates a corresponding request to perform memory operations on the memory system for the multidimensional tensor, wherein the request generator comprises m memory address units, wherein m is greater than or equal to 2, and wherein each memory address unit comprises a step size tracker and a memory address calculation unit. During each parallel memory address computation cycle, and with each memory address unit's step size tracker generating a memory address offset value for the corresponding tensor element of the multidimensional tensor, the generation includes determining (i) a corresponding step size index value for each dimension of the multidimensional tensor, and determining (ii) a corresponding memory address offset value for that dimension based on the corresponding step size index value, wherein the corresponding step size index value for a dimension of the multidimensional tensor corresponds to the position of the corresponding tensor element within the multidimensional tensor; and The memory address of the tensor element of the multidimensional tensor is generated by the memory address calculation element of each memory address unit and during each parallel memory address calculation cycle, based on the corresponding memory address offset value for the tensor element of the multidimensional tensor; and A request to perform the memory operation using the memory address is transmitted to the memory system. Each step tracker generates a step index value and a memory address offset value for tensor elements that are different from each of the other step trackers during each parallel memory address calculation cycle.

Images