Multi-level multi-stride direct memory access architecture

Abstract

To provide a DMA (direct memory access) architecture capable of executing multi-level multi-stride and determining a plurality of memory addresses in parallel.SOLUTION: Environment 100 includes one or more hardware DMA threads 120. Each DMA thread includes a request generator that generates M memory addresses in parallel for a multi-dimensional tensor for each parallel memory address calculation cycle and generates a request to a memory system to perform memory operation for each memory address. The request generator includes M memory address units. Each memory address unit includes: a step tracker that generates a step index value of a dimension for each tensor dimension and generates a stride offset value of the dimension based on the step index value; and a memory address calculation element that generates a memory address of a tensor element and sends a request to perform memory operation.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] background Direct Memory Access (DMA) is a feature that allows a device or subsystem to access memory independently of the processing unit, thus freeing the processing unit from data transfers and making it available to perform other operations. DMA can be used to offload expensive memory operations from the processing unit, such as large memory transfer operations and scatter / gather operations. Summary of the Invention [Means for solving the problem]

[0002] overview This specification describes techniques for a DMA architecture capable of multi-level multi-stride and determining multiple memory addresses in parallel, for example, during one clock period.

[0003] In general, one innovative aspect of the subject matter described herein can be included in a DMA (Direct Memory Access) system with one or more hardware DMA threads. Each DMA thread includes a request generator configured to (1) generate M memory addresses in parallel for a multidimensional tensor for each parallel memory address calculation cycle, and generate for each memory address a request to the memory system to perform a memory operation on the multidimensional tensor (2), where the request generator includes M memory address units. Each memory address unit includes a step tracker configured to generate (1) a step index value for each dimension of the multidimensional tensor and generate a stride offset value for the dimension (2) based on the step index value, and a memory address calculation element, where the memory address calculation element is configured to generate, for each parallel memory address calculation cycle, a memory address for a tensor element of the multidimensional tensor based on the stride offset value, and send a request to the memory system to perform a memory operation using the memory address, where M is 1 or more. Other embodiments of this aspect include corresponding apparatuses and methods.

[0004] These and other embodiments may each optionally include one or more of the following features: In some aspects, the request generator is configured to generate memory addresses in parallel during one clock cycle, each parallel memory calculation being performed during one clock cycle, and at each clock cycle, a memory address calculation element of each memory address unit generates a memory address of a tensor element that is the same or different than a memory address calculation element of another memory address unit.

[0005] In some aspects, the request generator is configured to receive a descriptor of the multidimensional tensor, the descriptor specifying, for each dimension, a number of steps of the stride value of the dimension. The request generator may include M lanes, each of the M lanes including a step tracker and a memory address calculation element. The step tracker and memory address calculation element of each lane calculates a corresponding memory address in parallel with the other lanes. The step tracker may be configured to generate memory addresses of the multidimensional tensor based on a loop nest, the loop nest including, for each dimension of the multidimensional tensor, a loop for traversing the dimensions of the multidimensional tensor. The number of steps per stride value of each dimension represents a loop range of the loop of the dimension, and the step index value of each dimension represents a loop index of the loop of the dimension.

[0006] In some aspects, each step tracker is configured to update a step index value for each of the dimensions at every clock cycle. The combination of step index values ​​for each step tracker may differ from the combination of step index values ​​for the other step trackers. Each step tracker may include a step increment chain including a plurality of step incrementers, each of the plurality of step incrementers configured to determine a dimension memory address offset value for the dimension. A first step incrementer of the step increment chain corresponding to an innermost loop of the loop nest may be configured to receive a forewarning amount. Updating the step index value of one or more of the dimensions at every clock cycle may include the first step incrementer updating the step index value of the one or more dimensions based on the forewarning amount.

[0007] In some aspects, one or more second step increment units in a step increment chain corresponding to a loop in which the innermost loop is nested are configured to receive the wrap amount from a previous step tracker in the step increment chain. Updating the step index values ​​of one or more of the dimensions every clock cycle may include the second step increment units updating the step index values ​​of the one or more dimensions based on the wrap amount.

[0008] Some aspects may include a progress tracker including a response sorter and a synchronizer. The response sorter may be configured to maintain, for each tensor, a status of whether a memory operation of the tensor element has been performed. The synchronizer may be configured to provide a plurality of update information to the processor core, each of the plurality of update information specifying an overall status of memory operations performed on tensor elements of the multidimensional tensor.

[0009] In some aspects, each request includes a unique identifier. The response reorderer may be configured to receive responses from the memory system in any order. Each response may include a unique identifier of the request for which the response is provided. The response reorderer may be configured to release a set of unique identifiers for reuse by the request generator when at least a threshold number of consecutive unique identifiers are received in a response.

[0010] In general, another innovative aspect of the subject matter described herein can be included in a system that includes one or more processor cores, a memory system, and a DMA engine that includes one or more DMA threads. Each DMA thread can include a request generator configured to (1) generate M memory addresses in parallel for a multidimensional tensor for each parallel memory address calculation cycle, and generate for each memory address a request to the memory system to perform a memory operation on the multidimensional tensor, the request generator including M memory address units, where M is 1 or greater. Each memory address unit can include a step tracker configured to generate (1) a step index value for each dimension of the multidimensional tensor and generate a stride offset value for the second dimension based on the step index value, and a memory address calculation element configured to generate, for each parallel memory address calculation cycle, a memory address for a tensor element of the multidimensional tensor based on the stride offset value, and send a request to the memory system to perform a memory operation using the memory address. Each DMA thread may include a progress tracking unit that includes a response reorderer and a synchronization update unit configured to provide one or more processor cores with updated information regarding synchronization of memory operations managed by the DMA engine. Other implementations of this aspect include corresponding apparatus and methods.

[0011] These and other embodiments may each optionally include one or more of the following features. In some aspects, the request generator parallelizes the memory addresses during one clock cycle. The parallel memory calculations are arranged to be generated in a sequence, with each parallel memory calculation taking place during one clock cycle.

[0012] At each clock cycle, the memory address calculation element of each memory address unit may generate a memory address of a tensor element that is the same or different from the memory address calculation elements of the other memory address units. The request generator may be configured to receive a descriptor of the multidimensional tensor, the descriptor specifying, for each dimension, a number of steps of the stride value of the dimension. The request generator may include M lanes, each of the M lanes including a step tracker and a memory address calculation element, and the step tracker and memory address calculation element of each lane calculates a corresponding memory address in parallel with the other lanes.

[0013] In general, another innovative aspect of the subject matter described herein may be included in a method performed by a DMA system, the method including a request generator generating (1) M memory addresses in parallel for a multidimensional tensor for each parallel memory address calculation cycle, and generating for each memory address (2) a request to the memory system to perform a memory operation on the multidimensional tensor, the request generator including M memory address units, M being 1 or greater, each memory address unit including a step tracker and a memory address calculator, the method further including the step tracker generating (1) a step index value for each dimension of the multidimensional tensor and generating (2) a stride offset value for the dimension based on the step index value, a memory address calculator element of each memory address unit generating, for each parallel memory address calculation cycle, a memory address of a tensor element of the multidimensional tensor based on the stride offset value, and sending the request to the memory system to perform the memory operation using the memory address. Effect of the Invention

[0014] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages: The DMA architecture described herein allows for the generation of multiple (M) memory addresses of a multi-dimensional tensor in parallel, e.g., per clock cycle, which allows for faster generation of memory addresses and improved memory throughput. The DMA architecture may include multiple lanes, each including a step tracker, which performs a multi-stride technique to calculate multiple memory addresses of multiple tensor elements in parallel based on a loop nest of the multi-dimensional tensor. This technique, employed by a request generator of the DMA architecture, allows multiple step trackers to operate in parallel, each with a different step index value for multiple dimensions of the tensor, and independently generate different tensor element addresses in parallel during a clock cycle. Each step tracker may update its step index value every clock cycle in preparation for determining the memory address of the next tensor element.

[0015] The DMA architecture may also include a progress tracker that provides some synchronization updates to the processor core, which consumes the data stored in the memory at the determined memory address. This allows the processor core to start consuming data before all of the DMA memory transactions are completed, reducing the waiting time the processor core would otherwise experience due to memory transfers and improving the overall computational efficiency of the processing device. The progress tracker may include a response reorderer that can receive multiple responses at once from the memory in any order, and the memory can address and respond to the generated requests out of order. The response reorderer has a limited size so that it can release a request identifier when responses to at least a threshold number of requests have been accepted. This allows the request generator to release a request identifier without waiting for responses to all of the maximum number of requests. The freed identifiers can then be utilized to continue issuing memory requests, increasing the speed and efficiency of memory transfers.

[0016] Various features and advantages of the above-mentioned subject matter are described below with reference to the drawings. Other features and advantages will be apparent from the subject matter described herein, and from the appended claims. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates an example environment in which a DMA thread generates memory operations and tracks their progress. [Figure 2A] FIG. 2 is a diagram of an example request generator. [Figure 2B] FIG. 13 illustrates an example pseudocode for determining a memory address. [Diagram 3] FIG. 1 is an illustration of an exemplary step tracker. [Figure 4] FIG. 2 is a diagram of an exemplary step increment chain. [Diagram 5] FIG. 2 is an example progress tracker. [Figure 6] FIG. 2 is a diagram of an example response reorderer. [Figure 7] FIG. 4 is a flow diagram illustrating an example process for generating a request for a memory operation. [Figure 8] FIG. 4 is a flow diagram illustrating an example process for tracking the progress of memory operations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description Generally, this specification describes a DMA architecture capable of multi-level multi-stride and determining multiple memory addresses in parallel, e.g., during one clock period. A chip may include one or more DMA engines that offload memory transfer operations from the chip's processor core(s). Each DMA engine may include one or more DMA threads. Each DMA thread is a hardware unit that manages the execution of DMA transactions on behalf of the core(s). A clock period may be the time required for execution of one operation by a DMA engine or core.

[0019] The exemplary DMA architecture shown in FIGS. 1-6 and described below provides a design that can support up to four tensor dimensions and up to four source and destination memory addresses per clock cycle. However, the DMA architecture does not strictly fit four dimensions or addresses per cycle. Similar architectures can be used for other numbers of dimensions and addresses per cycle. In addition, the number of addresses per cycle can differ from the number of dimensions of the tensor for which the addresses are determined. For example, a DMA thread may include five lanes to generate five addresses per cycle, but the hardware is configured to calculate addresses for tensors with up to four dimensions, up to ten dimensions, or another maximum number of dimensions. That is, the architecture is parameterizable, with the choice of settings varying depending on the area / frequency goals for the design.

[0020] 1 is a diagram of an example environment 100 in which a DMA thread 120 generates and tracks the progress of memory operations. DMA thread 120 may generate and track the progress of memory operations as part of transactions requested on behalf of a processor core. DMA thread 120 is a hardware unit. This hardware unit may be part of a DMA engine that includes DMA thread 120 and one or more other DMA threads (although this is not required). DMA thread 120 performs scatter / gather operations and DMA thread 120 may manage DMA transactions for one or more processor cores, including core 110, such as read / write requests and other memory transfer operations. For example, DMA thread 120 may direct the transfer of a multi-dimensional tensor between different memories of an on-chip memory system that includes core 110 and DMA thread 120. DMA thread 120 directs the movement of data by sending requests (commands) to the memory system and tracks the completion of these requests so that progress can be synchronized to the core that requested the transaction. The memory system fulfills each read / write request / command independently as it enters the memory system, regardless of the order of the requests. DMA thread 120 handles the ordering of requests / responses and synchronization with the cores. Offloading these memory operations to DMA thread 120 frees up computation cycles on core 110 for other tasks, such as performing machine learning calculations or reshaping tensors.

[0021] The core 110 may request a DMA transaction by sending a descriptor 112 to the DMA thread 120. Each DMA transaction may include one or more memory transfer operations. The descriptor 112 includes information about the DMA transaction. For example, the descriptor 112 may include information specifying a source memory 152 (e.g., memory address(es) of the source memory 152) of the memory system 150 from which data is read, information specifying a destination memory 154 (e.g., memory address(es) of the destination memory 154) to which data is written, information specifying the size and shape (e.g., dimensions) of a source tensor whose tensor elements are stored in the source memory 152, and information specifying the size and shape of a destination tensor whose tensor elements are stored in the destination memory 154. A tensor element is data that a tensor contains that corresponds to a particular indexed position in the tensor.

[0022] The size and shape of the source tensor may be the same as or different from the size and shape of the destination tensor. For example, the size and shape may be different if the shape of the tensor is being modified by the core 110. The descriptor 112 may specify the size and shape of the tensor using a value for the number of steps per stride for each dimension of the tensor. In a for loop, the step size is the size of the increment for each iteration of the loop, and the number of steps per stride is the number of steps before the loop is reset, e.g., the loop bound of the loop.

[0023] For example, the number of steps per stride for one dimension of a tensor may be equal to the number of tensor elements across that dimension. In particular, for a 4-dimensional tensor of size 8x6x4x2, the first dimension may have 8 steps per stride, the second dimension may have 6 steps per stride, the third dimension may have 4 steps per stride, and the fourth dimension may have 2 steps per stride. As will be described in more detail below, the number of steps per stride can be used to traverse each dimension of the tensor and calculate the memory addresses of the tensor elements.

[0024] Descriptor 112 may also include a stride dimension offset value for each dimension. As described below, these stride dimension offset values ​​(also referred to as dimension offset values) are used to determine the memory addresses of tensor elements. The dimension offset is the stride distance. At each step of processing along a tensor dimension, DMA thread 120 "hops" in memory addresses by the stride dimension offset value. Descriptor 112 may include, for a source tensor, a stride dimension offset value for each dimension of the source tensor. Descriptor 112 may also include, for a destination tensor, a stride dimension offset value for each dimension of the destination tensor.

[0025] The DMA thread 120 includes a descriptor queue 122 that stores the descriptors 112. For example, For example, DMA thread 120 may execute multiple DMA threads sequentially based on the descriptors 112 it receives and stores in descriptor queue 122. In some implementations, descriptor queue 122 is a first-in, first-out (FIFO) queue, allowing DMA thread 120 to execute DMA transactions in the order in which the descriptors 112 for the DMA transactions are received. Execution of DMA transactions is fully pipelined and may be implemented to perform out-of-order operations, but appears to the processing unit to execute in program order.

[0026] DMA thread 120 also includes a descriptor splitter 124. Descriptor splitter 124 may retrieve information utilized by source sub-thread 130 and information utilized by destination sub-thread 140 from descriptor 112 and provide the appropriate information to each sub-thread 130 and 140.

[0027] In general, the source sub-thread 130 generates and sends read requests to the memory system 150 to read data from the source memory 152, tracks the progress of the read operation, and synchronizes the core 110 to the progress of the read operation. Similarly, the destination thread 140 generates and sends write requests to the memory system 150 to write data to the destination memory 154, tracks the progress of the write operation, and synchronizes the core 110 to the progress of the write operation. The memory system 150 may be the main memory of the core 110 or the chip that includes the core 110, for example, the RAM (random access memory) of the core 110 or the chip. The memory system implements the wiring of the actual memory so that the data read for each source memory read request is paired with a write request to the destination memory. Data does not pass through the DMA thread (request addresses are sent out and responses are received, but these responses and requests do not hold memory data).

[0028] The source sub-thread 130 includes a request generator 132 that generates read requests based on the descriptor 112. As described in more detail below, the request generator 132 may, for example, generate multiple memory addresses in parallel per clock cycle and generate a read request for each memory address. For example, the request generator 132 may generate the memory addresses of each of multiple tensor elements while traversing a multi-dimensional tensor during one clock cycle of the core 110. Each read request may include a request identifier ("request ID"), a memory address from which data is to be read, and a memory opcode. As described below, responses may be received out of order, so the request ID may be a sequence number or tag used to associate requests with responses. The memory opcode indicates a memory operation, for example, whether the request is a read operation, a write operation, a memset operation, or another operation targeted at the memory address of the request.

[0029] Source sub-thread 130 also includes a progress tracker 134 that tracks the progress of the read operation specified by the read request. For example, memory system 150 may send a read response to progress tracker 134 indicating that the read operation is complete. Each read response may include a request ID for the read request for which the response was sent. In this way, progress tracker 134 can use the request ID to track the progress of the DMA transaction.

[0030] Destination subthread 140 includes a request generator 142 that generates write requests based on descriptor 112. As described in more detail below, request generator 142 may, for example, generate multiple memory addresses in parallel per clock cycle and generate a write request for each memory address. For example, request generator 142 may generate memory addresses for each of multiple tensor elements while traversing a multidimensional tensor during one clock cycle of core 110. Each write request may include a request ID and may specify the memory address to which the data is to be written.

[0031] Destination sub-thread 140 also includes a progress tracker 134 that tracks the progress of the write operation specified by the write request. For example, memory system 150 may send a write response to progress tracker 144 indicating that the write operation is complete. Each write response may include the request ID of the write request for which the response was sent. In this way, progress tracker 144 can use the request ID to track the progress of the DMA transaction.

[0032] Progress trackers 134 and 144 may send synchronization messages 115 and 116, respectively, to core 110 to provide core 110 with updates on the progress of the DMA transactions corresponding to descriptors 112. Synchronization messages 115 and 116 may specify the degree of completion (e.g., percentage or number of memory operations completed) and / or the request ID for which a response was received.

[0033] As described below, the progress trackers 134 and 144 may send synchronization messages 115 and 116 that provide partial or incomplete updates on the progress of the DMA transactions. For example, each progress tracker 134 and 144 may be configured to send synchronization messages 115 and 116 each time a specified number, e.g., a threshold number, of responses to a DMA transaction are received. In a particular example, each progress tracker 134 and 144 may be configured to send synchronization messages 115 and 116 each time a response to at least a threshold number of consecutive request IDs is received. Because the core 110 may know the order in which memory operations are performed (and thus the order in which tensor elements are being moved), the core 110 may begin processing the transferred data based on these partial updates without waiting for the entire series of DMA transactions to complete.

[0034] Using separate sub-threads for read and write operations can improve throughput. For example, if each sub-thread 130 and 140 can generate a certain number of requests per clock cycle in parallel, e.g., four requests per clock cycle, the number of requests generated by the two sub-threads 130 and 140 will be twice the certain number, e.g., eight requests.

[0035] In some cases, DMA transactions may be performed using multiple DMA threads. For example, if memory bandwidth is sufficient to handle more requests per clock cycle than one DMA thread can generate, then multiple DMA threads may be used to generate the requests. When multiple DMA threads are used to transfer data for a multidimensional tensor, each DMA thread may receive a descriptor for a portion of the multidimensional tensor, for example, a slice of the tensor. This descriptor may specify the size and shape of the slice of the tensor and the memory address, similar to the descriptor for the full tensor.

[0036] Figure 2A is a diagram of an example request generator 200. The request generator 200 may be used to implement each of the request generators 132 and 142 of Figure 1. In this example, the request generator 200 is configured for an implementation in which there are up to four tensor dimensions and up to four memory addresses may be generated per clock cycle.

[0037] In general, the request regenerator 200 may determine memory addresses of tensor elements in a multidimensional tensor or other multidimensional data structure (referred to herein as a tensor for simplicity's sake). The request generator 200 may determine memory addresses such that data for the tensor can be read from and / or written to memory. The request generator 200 may calculate the memory addresses of the tensor elements based on a step index value for the tensor element that defines the position of the tensor element within the tensor. The exemplary request generator 200 is implemented using a five-stage design with pipeline registers 220, 230, 240, 260, and 270 located between adjacent stages.

[0038] To determine the memory addresses, the request generator 200 may traverse each dimension by stepping through each step index value for each dimension. For example, if a dimension contains 10 elements, the request generator 200 may step through the step index values ​​from 1 to 10 one by one. Conceptually, this may be done using a loop nest that includes one loop for each dimension of the tensor. In such an example, a dimension of a tensor may be traversed with loops by incrementing the step index value of the loop with each iteration of the loop until the loop range is equal to the number of elements contained in that loop. When the loop range is reached, the next outer loop is incremented and the current loop is reset to the first step index value corresponding to the first element that the dimension contains. The innermost loop may include memory address calculations to determine the memory addresses of tensor elements at positions in the tensor that correspond to the step index values ​​of the four loops contained in the loop nest. Exemplary pseudocode 280 for determining memory addresses using four loops is shown in FIG. 2B.

[0039] Referring to FIG. 2B, pseudocode 280 includes four loops 281-284 that are used to traverse the four dimensions of a tensor. The pseudocode 280 shown describes half of a transaction (either the source side reads or the destination side writes). The same or similar pseudocode may be instantiated twice separately to complete a complete transaction. In pseudocode 280, the loop range for each dimension (steps_per_stride) is the same on the source and destination sides of the transfer, but the stride offset values ​​(stride_dimension_offset_value_i) may be different. That is, steps_per_stride_0 in the pseudocode is the same on the source and destination sides, but stride_dimension_offset_value_0 in the source side pseudocode may be different from stride_dimension_offset_value_0 in the destination pseudocode.

[0040] The outermost loop 281 corresponds to one of the multiple dimensions and includes a step index value i0 and a loop range steps_per_stride_0. The loop range steps_per_stride_0 may be equal to the number of elements included in the dimension 281 corresponding to the outermost loop. Similarly, the loop 282 corresponds to one of the multiple dimensions and includes a step index value i1 and a loop range steps_per_stride_1 (which may be equal to the number of elements included in the dimension corresponding to the loop 282), and the loop 283 corresponds to one of the multiple dimensions and includes a step index value i2 and a loop range steps_per_stride_2 (which may be equal to the number of elements included in the dimension corresponding to the loop 283).

[0041] The innermost loop 284 also corresponds to one of the multiple dimensions and includes a step index value i3 and a loop range steps_per_stride_3 (which may be equal to the number of elements included in the dimension corresponding to the innermost loop 284). For each iteration of the innermost loop, a dimension memory address offset value for each dimension of the tensor is calculated using a function 285, and these dimension memory address offset values ​​are used to determine the memory addresses of the tensor elements corresponding to the step index values ​​i0 to i3 using a function 286. The dimension memory address offset value (destination_memory_address_offset_0) of the dimension corresponding to the outermost loop 281 is equal to the product of the loop step index value i0 and the stride dimension offset value (stride_dimension_offset_value_0) of the dimension. The dimension memory address offset values ​​for the other dimensions are determined in a similar manner, as shown in FIG. 2B. As described above, the stride dimension offset value of the dimension may be included in the descriptor.

[0042] The memory address of the tensor element may then be calculated based on the base memory address and the dimension memory address offset value of each dimension of the tensor. For example, the memory address of the tensor element may be based on the sum of the base memory address and the dimension memory address offset value of the respective dimension, e.g., equal to the sum of the base memory address and the dimension memory address offset value of the respective dimension, as shown in FIG. 2B.

[0043] Returning to FIG. 2A, the request generator 200 may perform similar memory address calculations in parallel without actually repeating the loop. In this example, the request generator 200 includes four lanes 201-204 for calculating four memory addresses in parallel, for example, within one clock cycle. In other examples, two or more memory addresses may be calculated in parallel using two or more lanes, for example, three lanes for three memory addresses, five lanes for five memory addresses, etc. That is, the request generator 200 may include M lanes for calculating M memory addresses in parallel, where M is one or more. The request generator 200 may calculate M memory addresses during a parallel memory address calculation period. The length of the parallel memory address calculation period is one clock period or less.

[0044] The number of lanes may be the same as or different from the number of dimensions of the tensor. For example, the request generator 200 may be used to calculate memory addresses of tensors with different numbers of dimensions based on the information contained in the descriptor 112. For example, a request generator 200 with four lanes may use up to all four lanes to calculate up to four memory addresses of three-dimensional tensors per cycle. The same request generator 200 may also use up to all four lanes to calculate up to four addresses of one-dimensional tensors, two-dimensional tensors, or four-dimensional tensors per cycle.

[0045] Since each lane 201-204 must calculate a memory address of a different tensor element, and each lane operates independently of the other, it would be difficult to perform such calculations in parallel based on multi-level multi-stride (in this example, four-level multi-stride). Since each lane 201-204 calculates memory addresses in parallel, e.g., simultaneously, one lane cannot wait for the other lane to complete before iterating one or more loops to determine the memory address of the next tensor element. Instead, each lane must be able to determine the next tensor element (e.g., the step index value of the next tensor element) and determine the memory address of that tensor element without waiting for another lane.

[0046] The request generator 200 includes a memory address unit 242-248 for each lane 201-204 (and thus for each parallel memory address calculation). Each memory address unit 242-248 includes a step tracker 222-228, respectively, and a memory address calculation element 252-258, respectively. In general, the step trackers 222-228 are configured to step through the tensor elements of a tensor and determine a dimensional memory address offset value for the tensor element. The memory address calculation elements 252-258 are configured to use the dimensional memory address offset values ​​received from the step trackers 222-228 to determine the memory address of the tensor element.

[0047] The request generator 200 includes a computation element 210 that pre-computes values ​​for the step trackers 222-228. For example, the computation element 210 may pre-compute various step comparison values ​​that the step trackers 222-228 can use to determine a next step index value for the next tensor element whose memory address is to be determined. As described below, a comparison of the current step index value with the step comparison value may be used along with other conditions to determine the next step index value. The computation element 210 may pre-compute step comparison values ​​for each dimension of the tensor. When the request generator 200 generates memory addresses and sends requests, Depending on the number of dimensions of the current descriptor 112 tensor being computed, these step comparison values ​​may be, for example, the number of steps per stride of the dimension minus 1, the number of steps per stride of the dimension minus 2, the number of steps per stride of the dimension minus 3, etc. The computation element 210 is not required, nor are the pre-computed values ​​required. Pre-computing values ​​may help improve critical path timing on the next clock cycle.

[0048] The computation element 210 may include a set of hardware adders that pre-compute and store step comparison values ​​in a register 220 (or other suitable data storage element). The computation element 210 may calculate the comparison offset value based on a step-per-stride value received in a descriptor. The descriptor may include a step-per-stride value for one or more of the dimensions. In this example, the descriptor includes a step-per-stride value for dimensions 1-3 (sps_1-sps_3), but does not include a step-per-stride value for dimension 0 (e.g., the dimension corresponding to the outermost loop). For example, if the step-per-stride variable is represented as a 32-bit signed integer, the step-per-stride value for dimension 0 may be implied to be the maximum integer value, e.g., the maximum integer value that can be stored in a signed 32-bit integer. In another example, a step-per-stride value may be included in the descriptor, but is not shown in FIG. 2A.

[0049] Because the value of the number of steps per stride may vary based on the size and shape of the tensor, the computation element 210 may pre-compute and store a step comparison value for each descriptor in the register 220. The descriptors may also be stored in the register 220.

[0050] The request generator 200 also includes a finite state machine (FSM) 232. The FSM 232 may initialize and control the step trackers 222-228 based on information from the descriptor 112. For example, the FSM 232 may retrieve the descriptor information from the register 230 and determine, based on the descriptor information, the number of requests to send for the DMA transaction defined by the descriptor. This number may be the number of tensor elements contained in the tensor. The FSM 232 may track the number of remaining requests to send and send an advance amount based on the number of remaining requests to each of the step trackers 222-224. The advance amount defines the number of memory addresses to be calculated during the next cycle of memory address calculation performed by the memory address calculation elements 252-258.

[0051] For example, in the course of performing a DMA transaction using all four lanes 201-204, the advance notice amount may be equal to 4. However, if the number of memory addresses calculated for this DMA transaction is less than four, the advance notice amount for the last cycle will be less than 4. For example, if the number of memory addresses is 18, FSM 232 provides an advance notice amount of 4 to each step tracker 222-228 for the first four cycles, and then provides an advance notice amount of 2 for the last cycle.

[0052] Also, FSM 232 may stall step tracker 232. For example, as described below, progress trackers 134 and 144 only need to track the progress of a certain number of requests at a time. When request generator 200 uses up its assigned request IDs, it may stall itself, e.g., step tracker 232. When request IDs can be released and reallocated, for example, as described below, progress trackers 134 and 144 may return request ID credits once responses for at least a threshold number of consecutive request IDs have been received.

[0053] Also, request generators 132 and 142 may stall due to external interconnect backpressure (i.e., the memory system may not be able to read new data). (Newer requests cannot yet be accepted). In some embodiments, each DMA thread 120 can be throttled independently using a hardware FSM that is configurable by software. Software can set a target request generation bandwidth over a configurable sampling period for each DMA thread 120, and the DMA thread 120 automatically stalls its pipeline when it reaches its allocated bandwidth. Thus, a DMA thread 120 can stall in three different situations: (1) memory system network backpressure, (2) request bandwidth is throttled, or (3) request ID allocation is exhausted (waiting for the progress tracker to return credits).

[0054] Each step tracking unit 222-228 determines the next step index value of each dimension using the advance notice amount received from the FSM 232, the current step index value of each dimension of the tensor, and the value of the number of steps per stride of each dimension. Also, each step tracking unit 222-228 determines the dimension memory address offset value of each dimension based on the next step index value of the dimension and the stride dimension offset value of the dimension. Each step tracking unit 222-228 outputs the determined dimension memory address offset value to the corresponding memory address calculation element 252-258 via the register 240. As will be described later, the memory address calculation element 252-258 determines the memory address of the tensor element based on the received dimension memory address offset value.

[0055] The step trackers 222-228 determine dimensional memory address offset values ​​of different tensor elements. For example, consider a 2x2x2x2 tensor (or other shape of tensor) that contains a total of 16 tensor elements. Since the four lanes 201-204 generate four requests per period, each step tracker 222-228 determines dimensional memory address offset values ​​of a total of four tensor elements of the 16 tensor elements. For example, the step tracker 222 may determine dimensional memory address offset values ​​of the first tensor element, the fifth tensor element, the eighth tensor element, and the thirteenth tensor element, the step tracker 224 determines dimensional memory address offset values ​​of the second tensor element, the sixth tensor element, the tenth tensor element, and the fourteenth tensor element, and so on.

[0056] The step trackers 222-228 may determine their respective dimensional memory address offset values ​​in parallel, independent of one another. That is, in some implementations, the step trackers 222-228 do not communicate data with the other step trackers 222-228. Instead, as described in more detail below, each step tracker 222-228 may be configured to determine the next tensor element (e.g., the step index value of the next tensor element) based on the initialization of the step tracker 222-228 and the advance notice received from the FSM 232. In this manner, none of the step trackers 222-228 must wait for the other step trackers 222-228, and the parallel computation may be completed by all of the step trackers 222-228 in one clock cycle. Exemplary architectures of the step trackers and techniques for determining dimensional memory address offset values ​​are shown in FIGS. 3, 4, and 7 and described below.

[0057] The memory address calculation elements 252-258 include first addition elements 262A-268A and second addition elements 262B-268B, respectively. The first addition elements 262A-268A may calculate the sum of the dimensional memory address offset values ​​received from the respective step tracking units 222-228 for each parallel memory address calculation cycle. For example, the addition element 262A may calculate the sum of the four dimensional memory address offset values ​​for a particular tensor element generated by the step tracking unit 222. The first addition elements 262A-268A may be realized as hardware adders.

[0058] The second addition elements 262B-268B (which may also be realized as hardware adders) may determine the memory address of a tensor element based on the sum of the dimension memory address offset values ​​calculated by the corresponding first addition elements 262A-268A and the base address. For example, the addition element 262B may determine the memory address of a particular tensor element by adding the base address to the sum of the four dimension memory address offset values ​​for the particular tensor element generated by the step tracking unit 222.

[0059] Second summing elements 262B-268B may output the respective memory addresses to register 270. Request sender 290 may generate and send requests for each memory address to a memory system, such as memory system 150 of FIG. 1. The requests may include a request ID and a memory address. Request IDs may be assigned to requests in sequence. For example, if a DMA thread is configured to have 500 outstanding requests at a time, the request IDs may start at 0 or 1 up to a maximum of 499 or 500, respectively. If 0-499 is used, the request ID of the first request may be 0, the request ID of the second request may be 1, and so on. Request sender 299 may include a counter that determines the request ID of each request.

[0060] Each of the four lanes 201-204 may generate memory addresses of tensor elements in parallel during one clock cycle. The FSM 232 controls the step trackers 222-228 of the lanes 201-204 to iterate through each tensor element of the tensor until the memory address of each tensor element contained in the tensor is calculated. After issuing a request for a descriptor, the FSM 232 may proceed to the next descriptor. However, the FSM 232 does not need to wait for responses to all requests to be received. If at least a threshold number of consecutive request IDs (e.g., responses received) are available, the progress tracker 132 or 134 may notify the request generator 200 so that the request generator 200 can issue a request for the next descriptor using these available request IDs. This further improves the throughput and efficiency of the DMA thread.

[0061] As described above, the request generators 132 and 134 of the DMA sub-threads 132 and 134 may be implemented using a request generator 200. In this example, each sub-thread 132 and 134 is allowed to send four requests per clock cycle.

[0062] FIG. 3 is a diagram of an example step tracker 300. Step tracker 300 may be used to implement each of step trackers 222-228 of FIG. 2A. In this example, step tracker 300 includes two increment chains 322 and 324. Increment chains 322 and 324 may perform the same or similar functions to generate step index values ​​and dimension memory address offset values ​​for tensor elements. This allows one of the step increment chains to proactively determine the dimension memory address offset value for the current descriptor that the DMA thread will begin processing, while the other step increment chain is initialized for the next descriptor that the DMA thread will process.

[0063] For example, the step increment chain 324 may proactively determine the dimension memory address offset value of the current descriptor. The step increment chain 324 may determine the dimension memory address offset value of the current descriptor using an advance notice received from an FSM, e.g., FSM 232 of FIG. 2, and a (descriptor-specified) stride parameter. As will be described below with reference to FIG. 4, while the step increment chain 324 is active, the FSM may initialize the step increment chain 322.

[0064] While the memory address of the last cycle of the current descriptor is requested, the FSM is initialized The request generator may then switch to the previously initialized step increment chain 322 and send the initialization amount to the step increment chain 322. The step increment chain 322 may generate the first set of dimension memory address offset values ​​in the clock cycle immediately following the clock cycle in which the step increment chain 324 determined the last set of dimension memory address offset values. Using two step increment chains in this manner may significantly improve the throughput and efficiency of the DMA thread, especially when the tensors are small. For example, if the request generator requires only three clock cycles to determine all of the memory addresses for this tensor, using one clock cycle to reinitialize one step increment chain between tensors would result in a 25% decrease in throughput (e.g., number of memory operations performed per unit time).

[0065] When switching between step increment chains 322 and 324, the FSM may control a set of multiplexers 332-338 to select which step increment chain's output to send to the memory address calculation unit via register 342. For example, the FSM may select the top lane of each multiplexer 332-338 when increment chain 322 is active, and may select the bottom lane of each multiplexer 332-338 when increment chain 324 is active.

[0066] 2B, each lane 201-204 includes a step tracker, which may be implemented as step tracker 300. In this example, step tracker 300 is the step tracker for lane 0 and outputs four dimensional memory address offset values ​​for lane 0.

[0067] Also, although not shown, each step tracker 300 may output a next step index value used to determine a dimension memory address offset value. The next step index value is returned to the step tracker 300 for use in determining subsequent step index values ​​and dimension memory address offset values. That is, the step increment chain 324 may determine a step index value for each dimension and may determine a dimension memory address offset value for each dimension. These values ​​may be returned to the step increment chain 324 as current values ​​to be used in determining the next value.

[0068] The step tracker 300 may also include multiplexers for the step index values ​​that receive the step index values ​​for each dimension from both the step increment chains 322 and 324, just as multiplexers 322-338 receive the dimensional memory address offset values ​​from both the step increment chains 322 and 324. The outputs of these multiplexers are provided to the step increment chain 324 for use in determining subsequent step index values.

[0069] While the step increment chain 324 calculates the dimensional memory address offset values ​​of the current descriptor, the step increment chain 322 may use the initialized state to determine the dimensional memory address offset values ​​of the first set of memory addresses of the next descriptor. However, the FSM may control the multiplexers 332-338 to pass the dimensional memory address offset values ​​received from the step increment chain 324. Once the current descriptor is completed, the FSM may control the multiplexers 332-338 to pass the dimensional memory address offset values ​​calculated by the step increment chain 322 for one cycle. The dimensional memory address offset values ​​include the values ​​of the first four tensor elements of the next tensor. The FSM may also control the multiplexer for the step index value to pass the step index value for this one cycle from the step increment unit 322 to the step increment chain 324. Then, the step increment chain 322 may control the multiplexer for the step index value to pass the step index value for this one cycle to the step increment chain 324. The increment chain 324 will have the current state of the step index value and may determine the dimensional memory address offset values ​​for the remaining cycles associated with this descriptor. After the first cycle of this descriptor is completed, the FSM may control the multiplexer to pass the output of the step increment chain 324 again.

[0070] FIG. 4 is a diagram of an exemplary step increment chain 400. Step increment chain 400 may include a step increment unit for each dimension of the largest tensor that the DMA thread is configured to process. In this example, step increment chain 400 includes four step increment units 410-440 for up to four dimensional tensors. This exemplary step increment chain shown in FIG. 4 is implemented as a combinatorial function in a format similar to a carry ripple adder circuit.

[0071] Each step increment unit 410-440 may receive a set of parameters that may include the number of steps per stride of the dimension corresponding to the step increment unit 410-440 and each step comparison value of the dimension pre-computed by the computation element 210. Since these values ​​may vary based on the possible size and shape of the tensor for which the DMA transaction is being performed, the step increment units 410-440 may be initialized on a per-descriptor basis.

[0072] Each step increment unit 410-440 may also receive a step index value for its dimension and a dimension offset value for that dimension. The step index value for that dimension may be initialized to 0 for the first cycle (as also illustrated by the input value to the step increment chain 322). After the first cycle, the step index value input to the step increment units 410-440 becomes the next step index value output by the step increment units 410-440. As discussed above, the dimension offset value for a dimension is the value multiplied by the step index value to determine the dimension memory address offset value. Compared to using four loops in the pseudocode 280 of FIG. 2B, the step increment unit 410 functions similarly to the innermost loop of the loop nest. However, instead of incrementing the step index by one for each iteration of the loop, the step increment unit 410 increments its step index value based on a pre-announced amount received from the FSM. For example, if the preview amount was 4, then the step increment unit 410 would increment the step index value for that dimension by 4. If this increment number exceeds the number of steps per stride for the dimension, then the step increment unit may reinitialize the step index value to 0 and continue incrementing until it has been incremented four times. This may include reinitializing more than once. For example, if the number of steps per stride is 3 and the preview amount is 4, then the step increment unit 410 would increment from 0 to 3, reinitialize to 0, increment four times, then increment from 0 to 1.

[0073] Rather than using stateful iteration, the step increment unit 410 may use a combinatorial function that behaves similarly to a pair of optimized adders. Like an adder, part of the step increment unit takes two operands ("step 3 index" and "advance_amount") and outputs a sum ("step 3 next index") and a carry ("wrap amount"). This function is similar to the dimension offsets, except that the function that calculates the next dimension offset does not output a wrap amount.

[0074] The step increment unit 410 may output a wrap amount to the step increment unit 420. The wrap amount is determined by the step increment unit 410 based on the received advance notice amount. The step index value may be equal to the number of times that the step index value has been reinitialized in the current period, i.e., the wrap amount reflects the number of times that the four loops are wrapped around based on the advance amount.

[0075] Each time the step increment units 410 to 440 calculate their dimension memory address offset values, for example during one clock cycle, the step increment unit 410 may calculate the next step index value for that dimension, the wrap amount of the step increment unit 420, and the dimension memory address offset value (for example, the product of the next step index value and the dimension offset value for the dimension).

[0076] The step increment unit 420 utilizes the wrap amount received from the step increment unit 410 in a manner similar to the manner in which the step increment unit 420 utilizes the advance notice amount received from the FSM. That is, the wrap amount represents the number of times the step index value of the dimension corresponding to the step increment unit 420 is incremented in this period. The step increment unit 420 increments the step index value using the wrap amount received from the step increment unit 410 to determine the next step index value. The step increment unit 420 may also determine the dimension memory address offset value using the next step index value (e.g., the product of the next step index value of the dimension and the stride dimension offset value).

[0077] Similar to step increment unit 410, step increment unit 420 may also calculate and output a wrap amount to step increment unit 430. The wrap amount may be equal to the number of times the step index value of step increment unit 420 is reinitialized in the current period based on the wrap amount received from step increment unit 410. That is, the wrap amount reflects the number of times the four loops are wrapped around based on the received wrap amount.

[0078] The step increment unit 430 may similarly utilize the wrap amount received from the step increment unit 420. That is, the wrap amount represents the number of times the step index value of the dimension corresponding to the step increment unit 430 is incremented in this period. The step increment unit 430 may increment the step index value using the wrap amount received from the step increment unit 420 to determine the next step index value. The step increment unit 430 may also determine the dimension memory address offset value using the next step index value (e.g., the product of the next step index value of the dimension and the stride dimension offset value).

[0079] Similar to step increment unit 420, step increment unit 430 may also calculate and output a wrap amount to step increment unit 440. The wrap amount may be equal to the number of times the step index value of step increment unit 430 is reinitialized in the current period based on the wrap amount received from step increment unit 420. That is, the wrap amount reflects the number of times the four loops are wrapped around based on the received wrap amount.

[0080] The step increment unit 440 may similarly utilize the wrap amount received from the step increment unit 430. That is, the wrap amount represents the number of times the step index value of the dimension corresponding to the step increment unit 440 is incremented in this period. The step increment unit 440 may increment its step index value using the wrap amount received from the step increment unit 430 to determine the next step index value. The step increment unit 440 may also determine its dimension memory address offset value using the next step index value (e.g., the next The product of the original next step index value and the stride dimension offset value.

[0081] Each time that the step increment units 410-440 calculate their dimension memory address offset values, e.g., during one clock period, the step increment units 410-440 may each calculate a next step index value for that dimension, a wrap amount for the next step increment unit (if applicable), and a dimension memory address offset value (e.g., the product of the dimension's next step index value and the stride dimension offset value).

[0082] In some implementations, each incrementer 410-440 may use a set of conditions in determining the next step index value for that dimension and / or the wrap amount for that dimension. The conditions may include the increment amount (e.g., a preview amount for incrementer 410 or a wrap amount for incrementer 420-440). The conditions may also include the number of steps per stride for the dimension and a comparison of the current step index value to a step comparison value.

[0083] For example, a lookup table may be generated that specifies, for example, for each particular combination of increment amount, number of steps per stride, and which step comparison value matches the current step index value, what the next step index value will be and what the wrap amount will be. The particular combination may vary based on the number of dimensions of the tensor for which the request generator can generate memory addresses. In this manner, each step increment unit 410-440 may simply compare the increment amount and the current step index value to the table to determine what the next step index value and wrap amount will be.

[0084] 5 is a diagram of an example progress tracker 500. The progress tracker 500 may be used to implement each of the progress trackers 134 and 144 of FIG. 2A. The progress tracker 500 includes a progress tracker queue 510, a response reordering unit 520, and a synchronization unit 530.

[0085] The progress tracker queue 510 may receive and store one or more descriptors (or relevant portions of a descriptor as necessary to handle acknowledgments and synchronization) that allow the synchronizer 530 to determine the progress of the DMA transaction that the descriptor specifies, as described below.

[0086] The response sorter 520 may receive responses from a memory system, such as the memory system 150 of FIG. 1. Each response may specify the request ID of the request to which the response corresponds. That is, the memory system may send to the progress tracker 500 a response for each completed request received from the request generator corresponding to the progress tracker 500.

[0087] The response reorderer 520 may receive responses in any order and reorder the responses based on their request ID. The memory system may process requests in an order other than the order in which the requests were received. For example, the memory system may use bandwidth optimization techniques to prioritize some requests over others. In light of this, the response reorderer 520 may be configured to receive responses out of order and reorder them to track the progress of memory operations completed by the memory system. An exemplary response reorderer is shown in FIG. 6 and described in more detail below.

[0088] The synchronizer 530 may receive progress data from the response reorderer and send synchronization messages to a core, such as the core 110 of FIG. 20, data specifying the number of sequence request IDs received. The synchronization unit 530 may be configured to send a synchronization message whenever at least a threshold amount (or a threshold percentage) of memory operations defined by the descriptor are completed. For example, the synchronization unit 530 may determine the number of memory operations (e.g., read operations or write operations, depending on the sub-thread) that are performed in association with the current descriptor. The synchronization update unit 530 may be configured to send a synchronization message to the core whenever at least 10% of the memory operations are completed. As described above, the core may start consuming the transferred data using these updated information, without waiting until all of the memory operations defined by the descriptor are completed.

[0089] The response reorderer 520 and / or the synchronizer 530 may be configured to inform the request generator of a set of request IDs that the request generator can reuse. For example, each time at least a threshold number of ordered request IDs are received in a response from the memory system, these request IDs may be released to the response generator for reuse. This allows the request generator to continue generating requests after it has sent a maximum number of requests that the progress tracker 500 can handle, but before all requests are completed.

[0090] For example, assume that progress tracker 500 includes a response reordering buffer that can only track 500 memory operations at a time, with request IDs ranging from 0 to 499. If all 500 memory operations have been used in requests and there have been no responses to any of the requests, the request generator must stall until it receives a notification from progress tracker 500 specifying available request IDs. If progress tracker 500 receives responses for request IDs 0 to 15 (but not all responses for that identifier) ​​and the threshold is less than 15, progress tracker 500 may send a notification (e.g., a request ID credit return message) specifying that the request generator can resume sending requests with request IDs 0 to 15 without waiting until all 500 memory operations are completed.

[0091] FIG. 6 is a diagram of an exemplary response reorderer 600. The response reorderer 600 may be used to implement the response reorderer 520 of FIG. 5. The response reorderer 600 includes a response vector 610 and a reordering vector 630, each of which may be implemented using a bit vector register. The response vector 610 and the reordering vector 630 may each include one bit for each request ID that the request generator can issue. The bit may indicate the status of the request ID. For example, if the bit has a value of 0, a response to the memory operation has not yet been received. If the bit has a value of 1, a response to the memory operation has already been received. The response vector 610, the reordering vector 630, and the pop vector (described below) may all be the same size, e.g., include the same number of bits.

[0092] The response vector 610 may be configured to receive multiple responses at once, such as up to four responses at once in this example. For example, the response vector 610 may be configured to simultaneously receive a number of responses that matches the number of lanes of the corresponding request generator. In other examples, the response vector 610 may be configured to simultaneously receive a different number of responses than the number of lanes of the corresponding request generator, such as a number greater than the number of lanes.

[0093] The bits included in the reordering vector 630 may be arranged in the order of the request ID. At the input side of the reordering vector 630 is an OR gate 624. The OR gate 624 may be a bit vector OR gate with an OR gate for each bit of the reordering vector 630. For each request ID, the bits of the response vector for the request ID and the bits of the request ID output by the AND gate 622 (e.g., a bit vector AND gate) may be input to the OR gate to determine the value of the bit of the request ID included in the reordering vector 630.

[0094] The logical AND gate 622 receives, for each request ID, a bit 620 has one pair of inputs for each request ID. For a particular request ID, if the value of the bit in the reorder vector 630 and the value of the pop bit for the request ID in the pop vector maintained by pop vector logic 640 are both 1, then the output of the AND gate for that request ID is 1. As will be described below, the pop bit of a memory address may be set to 1 to clear the value of the bit to 0 when the request ID is released, for example, for use by the request generator. That is, if a response has been received for a request ID and the request ID has not yet been released, then the output of AND gate 622 for the bit corresponding to the request ID is 1. If the request ID is released, then the output of AND gate 622 for this bit is 0 since the input from the pop vector is 1.

[0095] The response reordering unit 600 also includes head pointer logic 650, pop vector logic 640, and internal pop count logic 660. The head pointer logic 650 may maintain a pointer to the next bit in the reordering vector 630 after the bit of the highest ordered request ID for which a response has been received. The ordered request IDs are the consecutive request IDs for which a response has been received, from the first request ID to the request ID for which a response has not yet been received. For example, if the request IDs have values ​​0-499 and responses to 0-8, 11, 56, and 61-78 have already been received, then the ordered request IDs have values ​​0-8. In this example, the head pointer points to the bit of the request ID with a value of 9. Once responses to the request IDs with values ​​9 and 10 have been received, the ordered request IDs have values ​​0-11, and it is assumed that a response to the request ID with a value of 12 has not yet been received.

[0096] The head pointer logic 650 may also pre-calculate another head pointer, such as the head pointer plus one (e.g., the bit next to the bit the head pointer is pointing to), the head pointer plus two, etc. In this way, the pop count logic 660 may pop more than one bit in the reordering vector 630 during one clock period. This functionality is not required and can be used to align with timing for a particular clock frequency. The logic becomes quite complex for large reordering vectors with many responses per period. This pre-calculation may be used when the target frequency is relatively fast or when there are many responses per period (e.g., many lanes).

[0097] The internal pop count logic 660 may monitor bits included in the reordering vector 630 to determine how many bits of the reordering vector may be popped (e.g., cleared) when the head pointer moves. For example, the internal pop count logic 660 may look ahead for a string of bits with a value of 1. The value of 1 indicates that a response to the memory operation corresponding to the bit has been received. When the head pointer logic 650 moves to another bit, the head pointer logic 630 may provide a destination (e.g., head_plus_i_next) (e.g., bits) for the head pointer to move to. Based on the new position of the head pointer and the bits being monitored, the internal pop count logic 660 may determine the number of bits that may be popped per clock period, e.g., up to a maximum number of pops. For example, if the head pointer can move up by 10 bits, the maximum number of pops is four bits per clock cycle, and the internal pop count logic 660 can instruct the pop vector logic 640 to pop four bits in the first cycle, pop four bits in the second cycle, and pop two bits in the third cycle. The head pointer is incremented by a number equal to the number of entries to pop during that cycle, so in this example it can advance a maximum of four bits per cycle.

[0098] The pop vector logic 640 may maintain a pop vector of bits to pop and provide this pop vector as an input to the AND gate 622. Based on the head pointer and the alternative head pointer and the number of bits to pop received from the internal pop count logic 660, the pop vector logic 640 determines which bits to pop. For example, if there are four bits to pop, the pop vector logic 640 may pop bits from the current head pointer to the head pointer plus four. Because the head pointer plus four has already been calculated, the pop vector logic 640 does not need to spend a clock cycle to determine the location of the bits to pop.

[0099] The head pointer logic 650 may also receive the number of bits to pop from the pop count logic. The head pointer logic 650 may update the head pointer and pre-calculate another head pointer based on the number of bits to pop.

[0100] The response reorderer 600 also comprises an order item register 672 and computation elements 670 and 674. The order item register 672 may keep a count of the number of popped order items that have not yet been released to the request generator. To do so, the computation unit 670 aggregates the number of popped bits based on the output of the internal pop count logic 660.

[0101] The number of ordered items in register 672 is also sent to synchronizer 530. Synchronizer 530 may determine when to release request IDs to the request generator based on the number of ordered items popped. For example, synchronizer may send data specifying the number of request IDs available to the request generator (e.g., request ID credit return). Calculator 674 may subtract this number from the number of ordered items currently in register 672 and update the register with the result (and newly popped items from internal pop count logic 660). For example, if register 672 indicates 15 ordered items have been popped and synchronizer 530 has released 10 to the request generator, calculation element 674 may subtract the 10 released request IDs from 15 request IDs and store and update the value of 5 request IDs. In this way, register 272 stores a running total of the number of request IDs available to the request generator.

[0102] 7 is a flow diagram illustrating an example process 700 for generating a request for a memory operation. Process 700 may be performed by a request generator, such as request generator 132 or 142 of FIG. 1 or request generator 200 of FIG. 2.

[0103] The request generator receives one or more descriptors (702). Each descriptor includes information about a DMA transaction, e.g., a series of memory transfer operations. For example, a descriptor may include information specifying a source memory from which data is read, information specifying a destination memory to which data is written, information specifying the size and shape (e.g., dimensions) of a source tensor whose tensor elements are stored in the source memory, information specifying the size and shape of a destination tensor whose tensor elements are stored in the destination memory, and information specifying a stride dimension offset value for each dimension.

[0104] The request generator initializes (704) a step tracker. As described above, the request generator may comprise multiple lanes, each calculating, for example, memory addresses per clock cycle in parallel. Each lane may include a step tracker and a memory address calculator. Each step tracker may include a step increment chain. Initializing the step tracker includes providing step parameters to each step incrementer and initializing a step index value for each step incrementer.

[0105] The request generator generates memory addresses (706). For example, the request generator may use multiple lanes to compute multiple memory addresses in parallel, e.g., during one clock cycle. In particular, during a clock cycle, each step tracker computes a next step index value for each dimension of the tensor (corresponding to a particular tensor element contained in the tensor). , a dimension memory address offset value for each dimension may be calculated using the stride dimension offset value for that dimension and the next step index value. Then, based on the dimension memory address offset value for each dimension and the base address output by the step tracking unit on the lane, the memory address calculation unit of each lane may calculate a memory address. For example, the memory address (and thus the tensor element) of the lane may be the sum of the base address and the dimension memory address offset value.

[0106] The request generator generates and sends requests to the memory system (708). The requests may be read or write requests. Each request may specify a request ID and a memory address calculated during this cycle. That is, the request generator may generate and send each request for each computer memory address. The request generator may send the requests to the memory system, which performs the read or write operation using the memory address included in the request.

[0107] The request generator determines 710 whether there are more tensor elements for which to compute memory addresses. For example, as described above, the FSM may track the number of remaining requests to generate for the descriptor. If there are more tensor elements, the process 700 returns to step 706 to generate more memory addresses.

[0108] If there is not another tensor element, the request generator determines whether there is another descriptor for which to perform a DMA transaction (714). For example, the request generator may check a descriptor queue to determine whether there is another descriptor in the queue. If there is not another descriptor, the process ends. If there is another descriptor, the process returns to step 704. In step 704, a step tracker is initialized for the next descriptor. As described above, the step tracker may be initialized before the DMA transaction of the previous descriptor is completed.

[0109] 8 is a flow diagram illustrating an example process 800 for tracking progress of a memory operation. The process 800 may be performed by a progress tracker, such as the progress tracker 134 or 144 of FIG. 1 or the progress tracker 500 of FIG. 5.

[0110] The progress tracker receives one or more responses 802. For example, the memory system may send a response to track progress in response to a memory operation being completed. The response may specify the request ID of the completed memory operation.

[0111] The progress tracker updates the reordering vector (804). The progress tracker may update the reordering vector to indicate that the memory operation corresponding to the request ID is complete. For example, the progress tracker may update the value of a bit of the request ID from 0 to 1 to indicate that the memory operation corresponding to the request ID is complete.

[0112] The progress tracker determines whether the number of consecutive elements (e.g., bits of the request ID) is greater than or equal to a threshold (806). If so, the progress tracker releases the request ID for reuse by the request generator (808). If not, the process 800 continues to step 810 without releasing the request ID.

[0113] In step 810, the progress tracker determines whether the number of responses received is greater than or equal to a threshold value. This number may be the number of responses received since the last sync message was sent to the core on which the memory operation is occurring. In another example, the progress tracker may determine whether at least a threshold percentage of the number of responses received have been received.

[0114] In either example, if the threshold is exceeded, the progress tracker synchronizes with the core. For example, the progress tracker may send a synchronization message to the core indicating the number or percentage of responses received. In another example, the progress tracker may send a synchronization message to the core indicating the number of responses received since the last synchronization message was sent to the core.

[0115] If the threshold has not been reached, process 800 continues at step 814. At step 814, the progress tracker determines whether all of the responses for the descriptor have been received. If all of the responses for the descriptor have not been received, process 800 returns to step 802, where more responses are received. If all of the responses for the descriptor have been received, the progress tracker may synchronize with the core (816), for example, by sending a synchronization message indicating that all of the memory operations for the descriptor are complete.

[0116] Although the present specification contains many specific implementation details, these should not be construed as limitations of the invention or the claims, but rather as descriptions of features specific to certain embodiments of a particular invention. Certain features described herein as separate embodiments may also be combined and realized in a single embodiment. Conversely, various features described as a single embodiment may also be realized in separate embodiments or in any suitable subcombination. Also, although features may be described above as operating in certain combinations and may initially be claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0117] Similarly, although the figures depict operations in a particular order, it should not be understood that such operations need to be performed in the particular order or sequence depicted, or that all of the operations depicted need to be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Also, the separation of various systems and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0118] Thus, specific embodiments of the inventive subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A DMA (Direct Memory Access) system, a DMA hardware unit, the DMA hardware unit comprising: a request generator configured to: (1) generate, in parallel, M memory addresses for a multidimensional tensor for each parallel memory address computation cycle; and (2) generate, for each memory address, a request to a memory system to perform a memory operation on the multidimensional tensor; the request generator includes M memory address units, each memory address unit comprising: a step tracking unit configured to generate memory address offset values ​​for tensor elements of the multidimensional tensor during each parallel memory address calculation cycle; a memory address calculation element, the memory address calculation element comprising: generating memory addresses of the tensor elements of the multidimensional tensor based on the memory address offset values ​​during each parallel memory address calculation cycle; and configured to send the request to the memory system to perform the memory operation using the memory address, the DMA hardware unit further comprising: A progress status tracking unit, the progress status tracking unit comprising: A response sorter configured to hold, for each tensor element, a status of whether or not a memory operation of the tensor element has been performed; and a synchronization update unit configured to provide a plurality of subset updates to a processor core, each subset update specifying an overall status of memory operations performed on the tensor elements of the multidimensional tensor; The tensor element for which an arbitrary step tracking unit among the step tracking units of the M memory address units generates a memory address offset value is different from the tensor element for which an arbitrary step tracking unit among the remaining step tracking units of the M memory address units generates a memory address offset value for each parallel memory address calculation cycle, A DMA system, wherein M is 2 or greater.

2. 2. The DMA system of claim 1, wherein the step tracking unit is configured to generate the memory address offset values ​​for the tensor elements of the multidimensional tensor by: for each dimension of the multidimensional tensor, (1) determining a step index value for the dimension; and (2) determining a memory address offset value for the dimension based on the step index value, the step index value for the dimension of the multidimensional tensor corresponding to a position of the tensor element within the multidimensional tensor.

3. Each request includes a unique identifier, The response sorting unit is and configured to receive responses from the memory system in any order, each response including the unique identifier of the request to which the response is provided, the response reordering unit further comprising:

3. A DMA system as claimed in claim 1 or 2, configured to release a range of unique identifiers for reuse by the request generator.

4. 4. The DMA system of claim 3, wherein the response reorderer is configured to release the series of unique identifiers when at least a threshold number of consecutive unique identifiers are received in the response.

5. 5. The DMA system of claim 1, wherein the request generator is configured to generate the memory addresses in parallel during one clock cycle, each parallel memory computation occurring during one clock cycle.

6. 6. The DMA system of claim 1, wherein the request generation unit includes M lanes, each of the M lanes includes a step tracking unit and a memory address calculation element, and the step tracking unit and the memory address calculation element of each lane calculate a corresponding memory address in parallel with other lanes.

7. the step tracker is configured to generate the memory addresses of the multidimensional tensor based on a loop nest, the loop nest including, for each dimension of the multidimensional tensor, a loop for traversing a dimension of the multidimensional tensor; 3. The DMA system of claim 2, wherein a number of steps per stride value for each dimension represents a loop range for the loop in that dimension, and wherein the step index value for each dimension represents a loop index for the loop in that dimension.

8. 8. The DMA system of claim 7, wherein each step tracker is configured to update the step index value for each of the dimensions every clock period.

9. 1. A method performed by a DMA system, comprising: The method includes a step of: a request generator generating, for each parallel memory address calculation cycle, (1) M memory addresses for a multidimensional tensor in parallel, and for each memory address (2) generating a request to a memory system to perform a memory operation on the multidimensional tensor, the request generator including M memory address units, M being 2 or greater, each memory address unit including a step tracking unit and a memory address calculation element, the method further comprising: generating memory address offset values ​​for tensor elements of the multi-dimensional tensor for each parallel memory address calculation cycle by the step tracking unit of each memory address unit; The memory address calculation element of each memory address unit generates a memory address of a tensor element of the multidimensional tensor based on the memory address offset value of the tensor element of the multidimensional tensor for each parallel memory address calculation cycle; the memory address calculation element for each memory address portion sending the request to the memory system to perform the memory operation using the memory address; A response sorting unit holds, for each tensor element, a status as to whether or not a memory operation for the tensor element has been performed; and a synchronization update unit providing a plurality of partial updates to the processor core, each of the partial updates specifying an overall status of memory operations performed on the tensor elements of the multi-dimensional tensor; The method of claim 1, wherein the tensor element for which any one of the step tracking units of the M memory address units generates a memory address offset value is different from the tensor element for which any one of the remaining step tracking units of the M memory address units generates a memory address offset value in each parallel memory address calculation cycle.

10. 10. The method of claim 9, wherein generating the memory address offset values ​​for the tensor elements of the multidimensional tensor comprises, for each dimension of the multidimensional tensor, (1) determining a step index value for the dimension, and (2) determining a memory address offset value for the dimension based on the step index value, the step index value for the dimension of the multidimensional tensor corresponding to a position of the tensor element within the multidimensional tensor.

11. Each request includes a unique identifier, The response sorting unit is and configured to receive responses from the memory system in any order, each response including the unique identifier of the request to which the response is provided, the response reordering unit further comprising:

11. The method of claim 9 or 10, configured to release a set of unique identifiers for reuse by the request generator.

12. The method of claim 11 , wherein the response reorderer releases the set of unique identifiers if at least a threshold number of consecutive unique identifiers are received in the response.

13. A method according to any one of claims 9 to 12, wherein the request generator generates the memory addresses in parallel during one clock cycle, each parallel memory computation being performed during one clock cycle.

14. 14. The method according to claim 9, wherein the request generator includes M lanes, each of the M lanes including a step tracker and a memory address calculation element, the step tracker and the memory address calculation element of each lane calculating a corresponding memory address in parallel with other lanes.

15. The step tracking unit generates the memory addresses of the multidimensional tensor based on a loop nest, the loop nest including, for each dimension of the multidimensional tensor, a loop for traversing a dimension of the multidimensional tensor; 11. The method of claim 10, wherein a number of steps per stride value for each dimension represents a loop range for the loop in that dimension, and the step index value for each dimension represents a loop index for the loop in that dimension.

16. The method of claim 15 , wherein each step tracker updates the step index value for each of the dimensions every clock period.

17. 1. A system comprising: one or more processor cores; A memory system; a DMA hardware unit, the DMA hardware unit comprising: a request generator configured to: (1) generate, in parallel, M memory addresses for a multidimensional tensor for each parallel memory address computation cycle; and (2) generate, for each memory address, a request to a memory system to perform a memory operation on the multidimensional tensor; the request generator includes M memory address units, each memory address unit comprising: a step tracking unit configured to generate memory address offset values ​​for tensor elements of the multidimensional tensor during each parallel memory address calculation cycle; a memory address calculation element, the memory address calculation element comprising: generating memory addresses of the tensor elements of the multidimensional tensor based on the memory address offset values ​​during each parallel memory address calculation cycle; and configured to send the request to the memory system to perform the memory operation using the memory address, the DMA hardware unit further comprising: A progress status tracking unit, the progress status tracking unit comprising: A response sorter configured to hold, for each tensor element, a status of whether or not a memory operation of the tensor element has been performed; and a synchronization update unit configured to provide a plurality of subset updates to a processor core, each subset update specifying an overall status of memory operations performed on the tensor elements of the multidimensional tensor; The tensor element for which an arbitrary step tracking unit among the step tracking units of the M memory address units generates a memory address offset value is different from the tensor element for which an arbitrary step tracking unit among the remaining step tracking units of the M memory address units generates a memory address offset value for each parallel memory address calculation cycle, A system, wherein M is 2 or greater.

18. 18. The system of claim 17, wherein the step tracker is configured to generate the memory address offset values ​​for the tensor elements of the multidimensional tensor by: for each dimension of the multidimensional tensor, (1) determining a step index value for the dimension; and based on the step index value, (2) determining a memory address offset value for the dimension, the step index value for the dimension of the multidimensional tensor corresponding to a position of the tensor element within the multidimensional tensor.

19. Each request includes a unique identifier, The response sorting unit is and configured to receive responses from the memory system in any order, each response including the unique identifier of the request to which the response is provided, the response reordering unit further comprising:

20. The system of claim 18, configured to release a set of unique identifiers for reuse by the request generator.

20. 20. The system of claim 19, wherein the response reorderer is configured to release the sequence of unique identifiers if at least a threshold number of consecutive unique identifiers are received in the response.

Images