Hardware double buffering using special purpose computation units

Abstract

To provide a method, a system and a device in which a device is included for transferring data using a plurality of buffers.SOLUTION: A device includes a plurality of memories and one or more processing units. The one or more processing units determine a buffer memory address for a sequence of data elements that are stored in a first data storage location and transferred to a second data storage location, and identify a value of a buffer allocation element switchable between a plurality of values for each group of one or more data elements in the data elements of the sequence. Each of the plurality of values corresponds to different one of the memories. The processing units further determine a buffer memory address for the group of one or more data elements based on the value of the buffer allocation element, and switch the value of the buffer allocation element before determining a buffer memory address for a subsequent group of the one or more data elements in the sequence of the data elements.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] background This specification relates generally to performing machine learning computations using special purpose computing units with hardware double buffers. [Background technology]

[0002] A neural network is a machine learning model that utilizes one or more layers of the model to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an outer layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden or output layer of the network. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters.

[0003] Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has an associated set of kernels. The kernels can be represented as a matrix structure of weight inputs. Each convolutional layer uses kernels to process the inputs to the layer. The set of inputs to the layer can also be represented as a matrix structure. Summary of the Invention [Means for solving the problem]

[0004] overview Described herein are techniques relating to the use of special purpose computational units for double buffering data for N-dimensional tensors.

[0005] In general, one innovative aspect of the subject matter described herein may be embodied in an apparatus for transferring data. The apparatus may include a plurality of memories including at least a first memory and a second memory. The apparatus may also include one or more processing units. The one or more processing units may be configured to determine a buffer memory address of a sequence of data elements stored in a first data storage location and being transferred to a second data storage location. For each group of one or more data elements of the data elements in the sequence, the one or more processing units may identify a value of a buffer allocation element switchable between a plurality of values, each of the plurality of values ​​corresponding to a different one of the memories. The one or more processing units may determine a buffer memory address of the group of one or more data elements based on the value of the buffer allocation element. If the value of the buffer allocation element is a first value corresponding to the first memory, the one or more processing units may allocate the one or more groups of data elements to the first memory by determining the buffer memory address of the one or more groups of data elements based on a combination of a base address of the first memory and a memory offset value of the one or more groups of data elements. If the value of the buffer allocation element is a second value corresponding to the second memory, different from the first value, the one or more processing units may allocate the one or more groups of data elements to the second memory by determining the memory address of the one or more groups of data elements based on a combination of the base address of the first memory, the memory address offset value of the second memory and the memory offset value of the one or more groups of data elements to allocate the one or more groups of data elements to the second memory. The one or more processing units may The value of the buffer allocation element may be toggled before determining the buffer memory addresses of a subsequent group of one or more data elements of the sequence of data elements. The one or more processing units may use the determined buffer memory addresses of each group of one or more data elements to transfer each data element to a respective memory location of the first or second memory.

[0006] Each of these and other implementations may optionally include one or more of the following features: In some aspects, the first memory and the second memory are buffers each having a first data storage capacity. Each of the first data storage location and the second data storage location may include at least a second data storage capacity that is greater than the first data storage capacity.

[0007] In some aspects, the first memory and the second memory are buffers each having a first data storage capacity, and the sequence of data elements may include an amount of data that exceeds the first data storage capacity.

[0008] In some aspects, determining the buffer memory address of the one or more group of data elements based on the value of the buffer allocation element and a memory address offset value of the second memory may include determining the memory offset value of the one or more group of data elements based on a number of iterations of one or more loops used to iterate through the sequence of data elements.

[0009] In some aspects, determining the buffer memory address of the one or more group of data elements based on the value of the buffer allocation element and a memory address offset value of the second memory may include determining as the buffer memory address of the one or more group of data elements a sum of (i) the base address of the first memory, (ii) the memory address offset value of the second memory, and (iii) the memory offset value of the one or more group of data elements each time the value of the buffer allocation element is the second value. Each time the value of the buffer allocation element is the first value, the buffer memory address of the one or more group of data elements may be determined based on the sum of (i) the base address of the first memory, and (ii) the offset value of the one or more group of data elements independent of the memory address value of the second memory.

[0010] In some aspects, the memory offset value of the group of one or more data elements is based on a number of iterations of the loop for each dimension of an N-dimensional tensor.The memory address offset value of the second memory may be based on a difference between a memory address of the first memory and an address of the second memory.

[0011] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages: Using multiple memories allows for faster and more efficient transfer of data that exceeds the storage capacity of a single buffer. For example, if 8 megabytes (MB) of data is to be transferred from a first data storage location to a second data storage location and the buffer has a capacity of only 4 MB, the data can be split between the two 4 MB buffers. Using nested loops to determine memory addresses of multiple buffers can reduce the number of instructions to determine the addresses, resulting in denser encoding, less memory resources used, and / or less memory resources required. Using a buffer allocation element (e.g., a 1-bit toggle counter) that toggles value after each buffer memory address determination allows for faster buffer allocation and reduces the time required to allocate data to multiple buffers. The total number of instructions to allocate data to a buffer can be reduced. Also, by toggling the value of a buffer allocation element rather than determining if a first buffer is full before allocating data to a second buffer, processing can be done more quickly and with less computational demand on the processor. Fewer instructions also result in higher performance because the processing unit processes fewer instructions to determine memory addresses. Encoding the double-buffer instructions in a special purpose hardware unit reduces the number of computational cycles the processor would perform to allocate data to a buffer, thus increasing processor bandwidth for other computational tasks.

[0012] Other implementations of this and other aspects include corresponding systems, devices, and computer programs configured to perform the method operations encoded in a computer storage device. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the system that, when operated, causes the system to perform the operations. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the operations.

[0013] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other possible features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an example computing system. [Diagram 2] 1 shows an example of data transferred using a buffer and an example of buffer allocation elements. [Diagram 3] FIG. 1 is a flow diagram illustrating an example of a process for transferring data using double buffering. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description In general, when a software algorithm processes an N-dimensional tensor, nested loops may be used. Each loop may involve traversing a respective dimension of the N-dimensional tensor. A multidimensional tensor may be a matrix or a multidimensional matrix. For example, a two-dimensional tensor is a matrix, and a three-dimensional tensor is a three-dimensional matrix made up of multiple two-dimensional matrices. Each dimension of an N-dimensional tensor may include one or more elements, and each element may store a respective data value. For example, a tensor may be a variable in a program, and the variable may have three dimensions. The first dimension may be 300 elements long, the second dimension may be 1000 elements long, and the third dimension may be 20 elements long. Of course, other numbers of elements in each dimension are possible.

[0016] Traversing a tensor in a nested loop may include calculating memory address values ​​of elements to load or store corresponding data values ​​of the elements. A for loop is an example of a nested loop, where three loops tracked by three loop index variables (e.g., i, j, and k) may be nested to traverse a three-dimensional tensor. In a neural network, the value of an element may be used in one or more dot product calculations associated with a tensor. For example, the value of the element may be multiplied by a corresponding parameter or weight. The elements of a tensor may be traversed in order using nested for loops to access the elements and perform one or more calculations using the value of the element. Continuing with the three-dimensional tensor example, an outer for loop may be used to traverse the loop tracked by variable i, a middle for loop may be used to traverse the loop tracked by variable j, and an inner for loop may be used to traverse the loop tracked by variable k. In this example, the first element accessed may be (i=0,j=0,k=0), the second element may be (i=0,j=0,k=1), etc.

[0017] As described below, the tensor traversal unit can be used to determine the memory address of each element in turn using nested loops so that a processing unit can access the element's value and perform one or more computations using the element's value. Values ​​of weights or parameters can be similarly accessed using nested for loops. The tensor traversal unit can also be used to determine addresses of weights or parameters for use in a computation and / or for the output of the computation, which may be used as an input to a hidden layer of a neural network.

[0018] The techniques described herein allow loop nests to be encoded to produce and consume data using double buffering. For example, data stored in tensor elements may be transferred from one data storage location to another using double buffering techniques. In a neural network example, activations determined as the output of one hidden layer may be provided as input to another hidden layer and thus may be transferred from an output location, i.e., a memory location where the output of a neural network layer is stored, to an input location, i.e., a memory location where the input to a neural network layer is stored. In another example, data representing the output of a computation may be transferred from a temporary memory location to a more permanent memory location. In each of these examples, data may be transferred more quickly and the data may be prepared for subsequent processing more quickly using double buffering.

[0019] Buffering can be used to collect data for a neural network calculation before the calculation is performed. For example, inputs to a neural network layer may be stored in a specific location for retrieval by a processor performing the calculation. While a neural network calculation is being performed using the data stored in the specific location, data for the next machine learning calculation may be moved to the buffer. Once the previous neural network calculation is completed, the data stored in the buffer may be moved to a specific location for retrieval by the processor.

[0020] Using a double buffer, one or more loops in the loop nest may be used to calculate the buffer memory addresses of tensor elements for which data is being produced or consumed. If the amount of data to be transferred is larger than the storage capacity of a single buffer, multiple buffers may be used. For example, if the amount of data to be transferred is twice the storage capacity of a single buffer, the data may be split between two buffers. In this example, a first portion of the data may be transferred to the first buffer and a second portion of the data may be transferred to the second buffer before being transferred to the second data storage location. In this manner, all of the data may be buffered before being transferred to the second data storage location.

[0021] As an example, the elements of a three-dimensional tensor may represent the features of an image to be classified by a neural network. The first dimension (Z) may represent the width of the image, the second dimension (Y) may represent the height of the image, and the third dimension (X) may represent the RGB values ​​of pixels in the image. To classify an image, each RGB value may be multiplied by a filter value of a convolutional layer to generate an activation map.

[0022] Nested loops can be used to determine memory addresses for accessing each RGB value of the tensor. The nested loops can include a loop for each dimension of the tensor. For example, an outer loop (z) may be used to traverse the Z dimension (width of the image), a central loop (y) may be used to traverse the Y dimension (height of the image), and an inner loop (x) may be used to traverse the X dimension (the three RGB values ​​of each pixel). At each iteration of the inner loop, a memory address is determined for one of the three RGB values ​​of a particular pixel of the image represented by the outer loop z and central loop y values. For example, the memory address of the R value of a pixel of the image represented by Z=0 and Y=0 may be determined during a first iteration of the inner loop x when z=0 and y=0 (e.g., z=0; y=0; x=0). Similarly, the memory address of the G value of a pixel of the image represented by Z=5 and Y=2 may be determined during a third iteration of the inner loop x when z=5 and y=2 (e.g., z=5; y=2; x=2). If the three RGB values ​​of each pixel of the image exceed the capacity of a buffer, the data representing the three RGB values ​​of each pixel of the image can be split between two or more buffers.

[0023] To determine the buffer memory addresses of multiple buffers using nested loops, the value of the buffer allocation element can be switched after (or before) each iteration of the loop used to determine the buffer memory addresses. For example, if two buffers are used and data is split between the two buffers, the buffer allocation value may be switched between two values. A first value (e.g., 0) of the buffer allocation element may be used to assign a data element (or a group of data elements) to the first buffer, and a second value (e.g., 1) of the buffer allocation element may be used to assign a data element (or a group of data elements) to the second buffer. When the value of the buffer allocation element is the first value for an iteration of the loop, the data element corresponding to this iteration of the loop may be assigned to the buffer memory address of the first buffer. Similarly, when the value of the buffer allocation element is the second value for an iteration of the loop, the data element corresponding to this iteration of the loop may be assigned to the buffer memory address of the second buffer. When more than two buffers are used, the buffer allocation element may have more than two values, e.g., one value for each buffer.

[0024] 1 is a block diagram of an example of a computing system 100. In general, the computing system 100 processes an input 104 to generate an output 116. The computing system 100 may be configured to perform linear algebra calculations, neural network calculations, and other calculations. The input 104 may be any suitable data processable by the computing system 100. The computing system 100 includes a processing unit 102, one or more storage media 104, and a tensor traversal unit 106.

[0025] The processing unit 114 may include one or more processors and / or one or more finite-state machines (FSMs). The processor of the processing unit 114 may execute instructions to access a particular element of a tensor. When the processor processes such an instruction, the tensor traversal unit 106 determines a memory address of the particular element of the tensor such that the processing unit can access the storage medium 104 to read data representing the value of the particular element.

[0026] In a processing unit that includes an FSM, the FSM can query the memory addresses of tensor elements from the tensor traversal unit 106. In some implementations, the FSM 108 continually queries the tensor traversal unit 106 for address values ​​of particular elements of a tensor. The FSM then notifies the processor to access the storage medium 104 to retrieve the address values ​​of the particular elements. The received address value may be provided to a processor in the processing unit 102 so that the data can be read.

[0027] For example, the program may include a nested loop, and the processing unit 102 may execute instructions to access an element of the two-dimensional array variable within the nested loop according to a current index variable value associated with the nested loop. Based on the current index variable value associated with the nested loop, the tensor traversal unit 106 may determine an address offset value representing an offset from a memory address of a first element of the two-dimensional array variable. The processing unit 102 may then use the address offset value to access a particular element of the two-dimensional array variable from the storage medium 104.

[0028] The storage medium 104 stores information within the computing system 100. In some implementations, the storage medium 104 is one or more volatile memory units. In some other implementations, the storage medium 104 is one or more non-volatile memory units. The storage medium 104 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive or a tape drive, a flash memory or other similar solid-state memory device, or a variety of devices (including devices in a storage area network or other configuration). The instructions, when executed by the processing unit 102, cause the processing unit 102 to perform one or more tasks.

[0029] In general, the tensor traversal unit 106 determines state associated with one or more tensors. In some implementations, the state may include loop bound values, current loop index variable values, partial address offset values ​​for determining memory address values, and / or program counter values ​​for processing branch loop bounds. The tensor traversal unit 106 may be implemented as an application specific integrated circuit.

[0030] The tensor traversal unit 106 can translate tensor indices into memory addresses. For example, the tensor traversal unit 106 may translate a set of N-dimensional tensor indices into a one-dimensional address space. The tensor traversal unit 106 can perform such a translation by making the memory addresses of the tensor elements a combination (e.g., a linear combination) of the element's dimensional indices.

[0031] The tensor traversal unit 106 may include one or more tensor state elements 122 and a numeric arithmetic unit 124. Each of the tensor state elements 122 may be a storage element, such as a register or any suitable storage circuit. The tensor state elements 122 may include buffer allocation elements, described below. The numeric arithmetic unit 124 may include one or more arithmetic logic units (ALUs) and / or one or more hardware adders. The math unit 124 can be used to calculate memory addresses or memory address offset values ​​of tensor elements based on values ​​stored in tensor state elements, for example. An example of a technique for determining memory addresses using a tensor traversal unit is described in U.S. patent application Ser. No. 15 / 335,769, entitled "Neural Network Computational Tiles," filed Oct. 27, 2016, and U.S. patent application Ser. No. 15 / 014,265, entitled "Accessing Data in Multidimensional Tensors," filed Feb. 3, 2016, the entire contents of which are incorporated herein by reference.

[0032] The tensor traversal unit 106 may also be configured to determine a memory address of a temporary memory location, such as a buffer, for transferring data from one data store to another. For example, the storage medium 104 may include storage location A 112 and storage location B 113. The first data storage location may include multiple data storage locations, including storage location B 114. Each storage location may be different memory addresses in a common memory unit or different memory units. The storage medium 104 may also include multiple temporary memories, including buffer A 116 and buffer B 118. The processing unit 102 may transfer data from a first data storage location (e.g., storage location A 112) to one or more buffers (e.g., buffer A 116 and / or buffer B 118) before transferring the data to a second data storage location (e.g., storage location B 114).

[0033] The tensor traversal unit 106 can use a buffer (e.g., a double buffer) to determine the buffer memory address of the tensor element for which data is being produced or consumed. For example, similar to the techniques described in U.S. patent application Ser. No. 15 / 014,265, entitled “Accessing Data in Multidimensional Tensors,” filed Feb. 3, 2016, the tensor traversal unit 106 can translate the tensor index to a buffer memory address using a base buffer memory address of the tensor and an address offset for each tensor element based on the tensor index of the tensor element.

[0034] For example, as described in U.S. Patent Application Serial No. 15 / 014,265, tensor state elements 122 may include a group of tensor index elements, a group of tensor boundary elements, and a group of dimension multiplier elements. Each group of elements may be arranged as a 2D array having M rows and N columns. Each row of the group may represent tensor index information for a tensor. Each column of the group may represent information about a nested loop index variable value associated with a tensor (e.g., a tensor index value, a tensor boundary value, or a dimension multiplier value). For example, one column in the 2D array of tensor index elements may represent tensor index information for variable i, one column may represent tensor index information for variable i, and one column may represent tensor index information for variable k.

[0035] Each tensor index element may track a nested loop variable of a loop in the nested loop. For example, one tensor index element may be assigned to track nested loop index variable i, one tensor index element may be assigned to track nested loop index variable j, and one tensor index element may be assigned to track nested loop index variable k. Each tensor boundary element has a corresponding element in the tensor index elements. Each tensor boundary element may represent tensor boundary information for a nested loop index variable value associated with a tensor. For example, one tensor boundary element may represent tensor boundary information for nested loop index variable i, one tensor boundary element may represent tensor boundary information for nested loop index variable j, and one tensor boundary element may represent tensor boundary information for nested loop index variable k.

[0036] Each dimension multiplier element may represent a multiplier by which a corresponding element in the tensor index element is multiplied. To determine the memory address of the element, the tensor traversal unit 106 may determine a memory address offset for each nested loop index variable by multiplying the value stored in the tensor index element of the nested loop index variable by the multiplier of the nested loop index variable. The tensor traversal unit 106 may then sum all the multiplication results to determine the memory address corresponding to the element being accessed.

[0037] The tensor traversal unit 106 may update the tensor index element after each iteration of the inner loop of the nested loop. Unit 106 may update the tensor index element of the inner loop, for example by incrementing the tensor index element of the inner loop. If the updated tensor index element of the inner loop is equal to the value stored in the tensor boundary element of the inner loop, the tensor index element may be reset and the tensor index element of the next outer loop in the nesting may be updated. Tensor traversal unit 120 may then determine the memory address of the next element corresponding to this iteration of the inner loop by multiplying the tensor index elements by their corresponding multipliers and summing the results, as described above.

[0038] When more than one buffer is used to transfer data, the tensor traversal unit 106 can also use a buffer allocation element to assign each tensor element or group of tensor elements to one of the buffers. In some implementations, the tensor traversal unit 106 can assign one or more groups of tensor elements to one of the buffers by adding an additional offset to the buffer memory address when the buffer allocation element has a value of one value, and assign one or more groups of tensor elements to a different buffer by not adding an additional offset value to the buffer memory address when the buffer allocation element has a different value.

[0039] For example, the base memory address may correspond to a first memory address of the first buffer. The first memory address of the second buffer may be offset from the base memory address by a certain number of addresses. In this example, to assign a tensor element to a memory address of the first buffer, the tensor traversal unit 106 may combine (e.g., add) the base memory address with a memory offset value of the tensor element. The memory offset value of the tensor element may be determined based on a tensor index of a nested loop used to traverse the tensor, as described in U.S. Patent Application No. 15 / 014,265, entitled “Accessing Data in Multidimensional Tensors,” filed February 3, 2016.

[0040] To assign a tensor element to a memory address of the second buffer, tensor traversal unit 106 may combine (e.g., add) the base memory address with the memory offset value of the tensor element and the memory address offset value of the second buffer (e.g., a particular number of addresses that offset the first memory address of the second buffer from the first memory address of the first buffer).

[0041] The tensor traversal unit 106 may use the value of the buffer allocation element to determine when to allocate the tensor element to the second buffer by determining the buffer memory address of the tensor element by combining the memory address offset value of the second buffer with the base memory address and the memory offset value of the tensor element. For example, if the value of the buffer allocation element is a first value, the tensor traversal unit 106 may allocate the tensor element to the first buffer by not combining the memory address offset value with the base memory address and the memory offset value of the tensor element. If the value of the buffer allocation element is a second value different from the first value, the tensor traversal unit 106 may allocate the tensor element to the second buffer by combining the memory address offset value with the base memory address and the memory offset value of the tensor element.

[0042] In some implementations, the tensor traversal unit 106 may determine the buffer memory addresses of a set of tensor elements in a sequence, for example using nested loops. In this example, the processing unit 102 may determine the buffer memory addresses of a group of one or more tensor elements for each iteration of a particular loop, for example each iteration of the innermost loop. The tensor traversal unit 106 can request the tensor element group from the tensor traversal unit 106. The tensor traversal unit 106 can determine a memory offset value of the group of tensor elements corresponding to the iteration of the loop based on the loop index. The tensor traversal unit 106 can also determine whether to allocate the group of tensor elements to the first buffer or the second buffer (or further buffers, if more than two) based on the value of the buffer allocation element, as described above. The tensor traversal unit 106 can determine a buffer memory address of the group of tensor elements based on the base memory address, the memory offset value of the group of tensor elements, and, depending on the value of the buffer allocation element, the memory offset value of the second buffer.

[0043] After determining the buffer memory addresses of the group of tensor elements in the sequence, the tensor traversal unit can toggle the value of the buffer allocation element. For example, if there are two buffers, the tensor traversal unit 106 may toggle the value between two values ​​after each buffer memory address determination. In this example, the tensor traversal unit 106 may assign the group of tensor elements to the first buffer when the value of the buffer allocation element is 0, and assign the group of tensor elements to the second buffer when the value of the buffer allocation element is 1. For the first buffer memory address determination, the buffer allocation element may have a value of 0. In this example, the tensor traversal unit 106 may assign the first group of tensor elements in the sequence to the first buffer. Then, the tensor traversal unit 106 may toggle the value of the buffer allocation element to 1. Thus, the tensor traversal unit 106 may assign the second group of tensor elements in the sequence to the second buffer. The tensor traversal unit 106 may continue to toggle the value after each buffer memory address determination such that all other groups of tensor elements are assigned to the first buffer.

[0044] In some implementations, coarse-grained toggling is used such that a group of tensor elements (e.g., subtensors of a tensor) are assigned to a buffer for each buffer memory address resolution. In some implementations, fine-grained toggling is used such that each individual tensor element is assigned to a buffer for each memory address resolution.

[0045] Consider an example where a tensor traversal unit has two 1 kilobyte (kB) buffers, and these buffers are used to transfer 4 kB of data. An example of a loop nest may include an outer loop that goes back and forth between the two buffers, and an inner loop may be used to identify each portion of data contained in the buffer. For example, the nested loop may include the following:

[0046]

number

[0047] In this example, the inner loop "j" is used to identify 1kB of data to be contained in the buffer, and the outer loop "i" is used to switch between the two buffers. For example, if "i" has an odd value, a group of 1kB of data elements may be assigned to the first buffer. If "i" has an even value, 1kB of data may be assigned to the second buffer. Thus, in this example, the loop nest alternates between the two buffers based on the value of "i".

[0048] In the case where there are more than two buffers, the tensor traversal unit 106 may switch the buffer allocation element between three or more different values, e.g., a unique value for each buffer. For example, in the case where there are three buffers, the tensor traversal unit 106 may assign a group of tensor elements to a first buffer when the buffer allocation element has a first value, the tensor traversal unit 106 may assign a group of tensor elements to a second buffer when the buffer allocation element has a second value, and the tensor traversal unit 106 may assign a group of tensor elements to a third buffer when the buffer allocation element has a third value.

[0049] In another example, there may be two buffers, each with 1 MB of storage capacity, and 3 MB of data may need to be transferred through the buffers. In this example, the first 1 MB may be allocated to the first of the two buffers, and the second 1 MB may be allocated to the second of the two buffers. Then, after the first 1 MB has been consumed, for example by the processor, the third 1 MB may be moved to the first buffer.

[0050] In some implementations, rather than switching the values ​​of the buffer allocation elements after each buffer memory address determination, the tensor traversal unit 106 can obtain a sequence of alternating buffer allocation values. For example, the sequence of alternating buffer allocation values ​​can be an alternating sequence of 0s and 1s. After each memory address determination, the tensor traversal unit 106 can move to the next value in the sequence and allocate a group of tensor elements to the appropriate buffer based on the next value.

[0051] 2 shows an example of data transferred using buffers and an example of a buffer allocation element. In this example, a sequence of eight groups of data elements, e.g., tensor elements, are transferred from a first data store 205 to a second data store 215 using two buffers. A sequence of buffer allocation values ​​210 is used to assign each group of data elements to one of two buffers. For example, if a group of data elements is in the same position in the sequence as a buffer allocation element having a value of 0, the group of data elements is assigned to a first buffer. If a group of data elements is in the same position in the sequence as a buffer allocation element having a value of 1, the group of data elements is assigned to a second buffer different from the first buffer.

[0052] Thus, in this example, data element groups "0," "2," "4," and "6" are assigned to the first buffer because the first, third, fifth, and seventh buffer allocation values ​​are 0. Similarly, data element groups "1," "3," "5," and "7" are assigned to the second buffer because the second, fourth, sixth, and eighth buffer allocation values ​​are 1. Thus, eight groups of data elements can be buffered using two buffers, each having storage capacity for storing four groups of data elements.

[0053] 3 is a flow diagram illustrating an example of a process 300 for transferring data using double buffering. Process 300 may be performed by one or more computer systems, such as computing system 110 of FIG.

[0054] The system identifies 302 a sequence of data elements designated for double buffering using a first buffer and a second buffer. The sequence of data elements may be a sequence of tensor elements designated for double buffering. A tensor element may be part of an N-dimensional tensor. For example, the tensor may be traversed using nested loops, each loop involved in traversing a respective dimension of the N-dimensional tensor.

[0055] The sequence of data elements may include all tensor elements of a particular dimension designated for double buffering. For example, a program containing nested loops may include code that designates a loop corresponding to a particular dimension as a loop to be double buffered. In a particular three-dimensional tensor example, the tensor may be traversed using three loops with indexes x, y, and z. In this example, the Z dimension of the tensor may be traversed using an outer loop with index z, the Y dimension of the tensor may be traversed using a middle loop with index y, and the X dimension of the tensor may be traversed using an inner loop with index x. The inner loops may be designated for double buffering to quickly buffer data for neural network computations.

[0056] The system determines (304) a buffer memory address for each group of data elements in the sequence. Each group may contain one or more data elements. For example, if fine-grained toggling is used, each group may contain one data element. If coarse-grained toggling is used, each group may contain multiple data elements, for example, up to a certain amount of memory or a certain number of data elements.

[0057] In some implementations, the system determines the buffer memory addresses one at a time. Continuing with the previous example, the system may determine a buffer memory address for each iteration of the inner loop, since each iteration of the inner loop corresponds to a particular tensor element designated for double buffering. The system may determine the buffer memory addresses for a sequence of data elements using configuration operations 306-314.

[0058] The system identifies (306) a buffer allocation element value for a group of data elements in the sequence of data elements. In some implementations, as described above, the system may switch the value of the buffer allocation element after each buffer memory address determination, e.g., after each iteration of a loop specified for double buffering. In this example, the system may identify the current value of the buffer allocation element as the buffer allocation element value for this data element. The buffer allocation element value is used to allocate the group of data elements to the appropriate buffer.

[0059] The system determines (308) a buffer memory address offset value for the group of data elements based on the value of the buffer allocation element and the memory address offset value of the second buffer. As described above, the base memory address of the buffer may correspond to the first memory address of the first buffer. The first memory address of the second buffer may be offset from the base memory address by a specific number of addresses. The memory address offset value of the second buffer may be equal to the specific number of addresses.

[0060] To determine a buffer memory address offset value for the group of data elements, the system determines whether the value of the buffer allocation element is a first value or a second value (or more values ​​if there are more than two buffers). If the buffer allocation element is a first value, the system may allocate the group of data elements to the first buffer by not using the memory address offset value of the second buffer when determining a buffer memory address value for the group of data elements. Instead, the system may allocate the group of data elements to the first buffer by not using the memory address offset value of the second buffer when determining a buffer memory address value for the group of data elements based on a loop index of the nested loop, as described above. The memory offset of the data element can be used.

[0061] When the buffer allocation element is a second value, the system may allocate the group of data elements to the second buffer by combining a memory offset value of the group of data elements with a memory address offset value of the second buffer. For example, the system may determine a buffer memory address offset value for the group of data elements as a sum of the memory offset value of the group of data elements and the memory address offset value of the second buffer.

[0062] In some implementations, the system may calculate the buffer memory address offset value for the group of data elements by ANDing a value of a buffer allocation element having a value of 1, multiplying the result by the memory address offset value of the second buffer, and adding the result to the memory offset value of the group of data elements. In this example, if the buffer allocation element has a value of 0, the buffer memory address offset value for the group of data elements is equal to the memory offset value of the group of data elements. If the buffer allocation element has a value of 1, the buffer memory address offset value for the group of data elements has a value equal to the memory address offset value of the second buffer plus the memory offset value of the group of data elements. In some implementations, a one-bit toggle counter may be used to determine which buffer to use.

[0063] The system determines a buffer memory address for the group of data elements based on a base address of the buffer and a buffer memory address offset value (310). For example, the system may determine the buffer memory address for the group of data elements by adding a base address of the buffer (e.g., a first memory address of a first buffer) to a buffer memory address offset value.

[0064] The system determines whether a buffer memory address has been determined for each data element in the sequence (312). If not, the system switches the value of the buffer allocation element for the next data element, so that the next data element is assigned to a different buffer than the current data element.

[0065] Once the buffer memory address has been determined for each data element in the sequence, the system transfers the data stored in the data element to a buffer based on the determined buffer memory address (314). The data may then be transferred from the buffer to a second data storage location, for example for use in a neural network calculation.

[0066] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in any combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical or electromagnetic signal, generated to encode information for transmission to a suitable receiver device and execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0067] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data to generate output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0068] A computer suitable for executing a computer program may include or be based on, by way of example, a general-purpose or special-purpose microprocessor or both, or other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operatively coupled to receive data from, transmit data to, or transmit data to, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer may be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0069] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example only, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0070] Although the present specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination and may be described as such initially, one or more features from a described combination may in some cases be deleted from the combination, and the described combination may be directed to a subcombination or a variation of the subcombination.

[0071] Similarly, although operations are described in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown to achieve a desired result, or as requiring that all of the shown operations be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Additionally, the separation of various system modules and components in the above embodiments may be implemented in a variety of ways. It should be understood that such separation should not be understood as necessary in all embodiments, and that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0072] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order depicted, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A system comprising: Multiple memories; a first data storage and a second data storage; and a processing circuit, the processing circuit comprising: and arranged to determine buffer memory addresses of data of an N-dimensional tensor stored in the first data storage and being transferred to the second data storage, N being an integer greater than or equal to 2, said determining comprising: selecting a plurality of data elements of the N-dimensional tensor using an inner loop of a loop nest, the selecting including identifying a portion of data of the N-dimensional tensor as the plurality of data elements until a loop boundary of the inner loop is reached, and the determining further comprising: For the plurality of data elements of the N-dimensional tensor, identifying current values ​​of buffer allocation elements, each value of the buffer allocation elements corresponding to a respective memory of the plurality of memories, the current values ​​of the buffer allocation elements being updated each time a buffer memory address is determined for a data element, the determining further comprising: For the plurality of data elements of the N-dimensional tensor, selecting a memory address offset value that corresponds to the current value of the buffer allocation element based on the current value of the buffer allocation element; determining the buffer memory address for each data element of the plurality of data elements based at least on a combination of (i) a base address of the plurality of memory, (ii) the memory address offset value corresponding to the current value of the buffer allocation element, and (iii) a memory offset value of the data element; outputting the determined buffer memory address; and updating the current value of the buffer allocation element with an outer loop of the loop nest each time the loop boundary of the inner loop is reached.

2. 2. The system of claim 1, wherein outputting the determined buffer memory address includes outputting the determined buffer memory address to a processing unit, the processing unit transferring the plurality of data elements to the respective memory corresponding to the current value of the buffer allocation element using the determined buffer memory address, the processing unit including at least one of a processor or a finite state machine.

3. The system of claim 1 or 2, wherein the loop bound of the inner loop corresponds to a maximum data size of an allocated memory.

4. The system of any one of claims 1 to 3, wherein the current value of the buffer allocation element is based on a current index value of the outer loop.

5. The system of claim 1 , wherein the memory offset value for each data element is based on a current index value associated with the loop nest used to traverse the N-dimensional tensor.

6. 1. A method performed by a computing system for transferring data, comprising: selecting a plurality of data elements of an N-dimensional tensor using an inner loop of a loop nest, the selecting comprising identifying a portion of data of the N-dimensional tensor as the plurality of data elements until a loop boundary of the inner loop is reached, the method further comprising: For the plurality of data elements of the N-dimensional tensor, identifying current values ​​of buffer allocation elements, each value of the buffer allocation elements corresponding to a respective memory of a plurality of memories, the current values ​​of the buffer allocation elements being updated each time a buffer memory address is determined for a data element, the method further comprising: For the plurality of data elements of the N-dimensional tensor, selecting a memory address offset value that corresponds to the current value of the buffer allocation element based on the current value of the buffer allocation element; determining a buffer memory address for each data element of the plurality of data elements based at least on a combination of (i) a base address of the plurality of memory, (ii) the memory address offset value corresponding to the current value of the buffer allocation element, and (iii) a memory offset value of the data element; outputting the determined buffer memory address; and updating the current value of the buffer allocation element with an outer loop of the loop nest each time the loop boundary of the inner loop is reached.

7. 7. The method of claim 6, wherein outputting the determined buffer memory address comprises outputting the determined buffer memory address to a processing unit, the processing unit transferring the plurality of data elements to the respective memories corresponding to the current value of the buffer allocation element using the determined buffer memory address, the processing unit comprising at least one of a processor or a finite state machine.

8. The method of claim 6 or 7, wherein the loop bound of the inner loop corresponds to a maximum data size of an allocated memory.

9. The method of any one of claims 6 to 8, wherein the current value of the buffer allocation element is based on a current index value of the outer loop.

10. A method according to any one of claims 6 to 9, wherein the memory offset value for each data element is based on a current index value associated with a loop nest used to traverse the N-dimensional tensor.

11. An apparatus comprising: Multiple memories; and a processing circuit, the processing circuit comprising: A method for transferring data from a first data storage location to a second data storage location, the method comprising: determining a buffer memory address of data for an N-dimensional tensor stored in a first data storage location and being transferred to a second data storage location, N being an integer greater than or equal to 2, said determining comprising: selecting a plurality of data elements of the N-dimensional tensor using an inner loop of a loop nest, the selecting including identifying a portion of data of the N-dimensional tensor as the plurality of data elements until a loop boundary of the inner loop is reached, and the determining further comprising: For the plurality of data elements of the N-dimensional tensor, identifying current values ​​of buffer allocation elements, each value of the buffer allocation elements corresponding to a respective memory of the plurality of memories, the current values ​​of the buffer allocation elements being updated each time a buffer memory address is determined for a data element, the determining further comprising: For the plurality of data elements of the N-dimensional tensor, selecting a memory address offset value that corresponds to the current value of the buffer allocation element based on the current value of the buffer allocation element; determining the buffer memory address for each data element of the plurality of data elements based at least on a combination of (i) a base address of the plurality of memory, (ii) the memory address offset value corresponding to the current value of the buffer allocation element, and (iii) a memory offset value of the data element; outputting the determined buffer memory address; and updating the current value of the buffer allocation element with an outer loop of the loop nest each time the loop boundary of the inner loop is reached.

12. 12. The apparatus of claim 11, wherein outputting the determined buffer memory address comprises outputting the determined buffer memory address to a processing unit, the processing unit transferring the plurality of data elements to the respective memories corresponding to the current value of the buffer allocation element using the determined buffer memory address, the processing unit comprising at least one of a processor or a finite state machine.

13. The apparatus of claim 11 or 12, wherein the loop bound of the inner loop corresponds to a maximum data size of an allocated memory.

14. The apparatus of any one of claims 11 to 13, wherein the current value of the buffer allocation element is based on a current index value of the outer loop.

Images