Transpose operation using array of processing elements

Abstract

The present invention provides systems and methods for transposing a tensor using processing element array operations. In some cases, it may be necessary to transpose the elements of a tensor to perform matrix operations. The tensor may be decomposed into blocks of data elements having dimensions consistent with the dimensions of a systolic array. Identity multiplication may be performed on each block of data elements loaded into the systolic array, and the products may be summed in a column partition of a result buffer. The data elements in the column partition of the result buffer may then be mapped to a row partition of a buffer memory for further processing.

Description

Background Art

[0001] An accelerator is an integrated circuit device optimized for performing computations (e.g., neural network computations) that would otherwise be performed by a general-purpose processor. Data stored in the main memory of a computing system is copied or mapped into memory locations within the accelerator to enable faster access to the data for computations to be performed by the accelerator's execution engine. In some cases, the architecture of an accelerator can be utilized to increase the speed of the computations to be performed by the execution engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 is a block diagram illustrating an example of a host system on which a compiler according to aspects of the present disclosure may execute;

[0004] Figure 2 is a block diagram illustrating an example of an integrated circuit device according to aspects of the present disclosure;

[0005] Figure 3 is a diagram illustrating a simplified example of a process for transposing a matrix using processing element array operations in accordance with aspects of the present disclosure;

[0006] Figure 4 is a diagram illustrating a generalized example of a tensor transpose operation according to aspects of the present disclosure;

[0007] Figure 5 is a flow chart illustrating an example of a method for performing a transpose operation on a decomposed tensor block according to aspects of the present disclosure;

[0008] Figure 6 is a block diagram illustrating an example of an acceleration engine according to aspects of the present disclosure;

[0009] Figure 7 contains a block diagram illustrating an example of a host system according to aspects of the present disclosure; and

[0010] Figure 8 A diagram containing example networks in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0011] The acceleration engine may include one or more accelerators to perform calculations using a processing engine array. The processing engine array may perform complex calculations, such as matrix multiplication calculations or other calculations. The accelerator may also include a memory subsystem. The memory subsystem may include one or more buffer memories that store data, for example, neural network weights and data to be processed by the processing engine array, and intermediate and / or final results of calculations performed by the processing engine array.

[0012] For various processing engine array calculations, data in the form of tensors can be retrieved from host memory (e.g., dynamic random access memory (DRAM)) and stored in a buffer memory. In practical applications, the number of data elements in a tensor can be much larger than the number of data elements that the buffer memory can store at one time and / or the processing engine array can process at one time. Therefore, the tensor can be decomposed into multiple blocks of data elements of appropriate size for storage in the buffer memory. In addition, the number of data elements of the decomposed tensor stored in the buffer memory may be too large for the processing engine array to process at one time. Therefore, the blocks of data elements stored in the buffer memory can be decomposed again for processing by the processing engine array.

[0013] In some cases, it may be necessary to change the way tensors are mapped to buffer memory. For example, in the case of matrix multiplication of weights and inputs of different sizes, the calculation may require matrix transposition. One way to perform a matrix transposition operation is to copy the matrix data from the buffer memory to the host memory (e.g., DRAM), and then write the matrix data from the host memory back to the buffer memory at the transposed position. However, this technique may result in significant external memory access latency and may require host processor intervention that takes up host processor cycles. In order to reduce latency when the processing engine array performs calculations such as matrix transposition operations, internal processing engine array operations can be used to perform matrix transposition operations without writing matrix data to the host memory and reading matrix data from the host memory.

[0014] In the following description, various examples will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without the specific details. In addition, well-known features may be omitted or simplified to avoid obscuring the embodiments being described.

[0015] Aspects of the present disclosure provide systems and methods for performing matrix transpositions on data elements stored in a buffer memory of an accelerator using systolic array operations. Compiler-generated instructions may decompose data elements of a multidimensional tensor into a set of two-dimensional (2D) matrices that exceed the dimensions of the systolic array. A multiply-accumulate systolic array operation may be performed on each of the set of 2D matrices using an identity matrix. Additional instructions generated by the compiler may cause the result of the identity multiplication to be stored in the buffer memory as the transpose of each of the 2D matrices.

[0016] Figure 11 is a block diagram illustrating an example of a host system 100 on which a compiler 130 as described herein may be executed in accordance with aspects of the present disclosure. The illustrated host system 100 is an example of a computing device and includes a processor 102, a processor memory 104, at least one storage device 106, various input / output (I / O) devices 108, and at least one network interface 110. Figure 1 In the example of the host system 100, the host system 100 also includes an acceleration engine 112, which is an integrated circuit device that can accelerate certain operations or calculations performed by the host system 100. In various examples, the host system 100 can be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smart phone, among other examples. In some examples, the operations or components discussed below as being executed or included in the host system 100 can be executed or included in other computer devices. For example, when the acceleration engine 112 is located at a different host system, the compiler 130 can be executed on the host system 100.

[0017] The processor 102 is an integrated circuit device that can execute program code in the form of instructions. The program code can be used in various software applications or tools, such as the computing system 120 or the compiler 130 shown. When the processor 102 is executing a program, the instructions of the program can be stored in the processor memory 104. The instructions can also be stored elsewhere, such as on the storage device 106, and can be loaded into the processor memory 104 when the processor 102 needs it. The processor 102 can also use the processor memory 104 for temporary storage of other data operated by the processor 102. In various examples, alternatively or additionally, although non-volatile memory types can be used for the processor memory 104, the processor memory 104 is a volatile memory type, such as a random access memory type.

[0018] Storage device 106 is an example of a device that may include non-volatile memory. For example, storage device 106 may be a disk drive, a solid-state drive, or an optical drive, among other examples. Storage device 106 may further be non-transitory, such that when storage device 106 is not powered, program code and other data stored on storage device 106 still exist.

[0019] Storage device 106 is an example of a peripheral device. Peripheral devices are components that can be coupled to host system 100 to add functionality to host system 100. Other examples of peripheral devices include input / output devices 108 and network interface 110. Input / output devices 108 can include user input and output devices such as keyboards, mice, touch screens, microphones, display screens, speakers, printers, and scanners, among other examples. Network interface 110, which can be implemented using a network interface card, can provide access to one or more networks. Network interface 110 can include, for example, a physical port for connecting a network cable and / or a wireless antenna to communicate with a Wi-Fi and / or cellular network. Network interface 110 can also be described as an I / O device.

[0020] Acceleration engine 112 is also another type of peripheral device or I / O device. Acceleration engine 112 is a device that is specifically designed to perform certain operations that can be performed by processor 102, but can be performed faster by acceleration engine 112. For example, acceleration engine 112 can be a neural network accelerator, and as such can perform massively parallel computations of neural networks more efficiently than when the computations are performed by processor 102. As another example, acceleration engine 112 can be a graphics processing unit (GPU), and can be optimized to perform the computations required for graphics rendering. Other examples of devices that can be implemented by acceleration engine 112 include cryptographic accelerators, compression and decompression accelerators, 3-D accelerators, regular expression accelerators, security accelerators, etc.

[0021] In various examples, the acceleration engine 112 can implement program code to perform certain operations. For example, when the acceleration engine 112 is a neural network accelerator, the acceleration engine 112 can be programmed to execute a specific neural network, such as a neural network that performs image recognition or a neural network that performs machine translation. As another example, to support the execution of the neural network, the acceleration engine 112 can be programmed to perform operations such as copying data of the neural network from the processor memory 104 (for example) to the acceleration engine 112, copying input data of the neural network from the processor memory 104 to the acceleration engine 112, and / or copying results from the acceleration engine 112 to the processor memory 104, as well as other examples.

[0022] To generate program code for the acceleration engine 112, in various instances, the host system 100 may execute a compiler 130. Generally, a compiler is a software program that translates program code written in a human-readable language into a format (e.g., machine instructions) that can be read and processed by an integrated circuit device. Figure 1In the example of FIG. 1 , the acceleration engine 112 is a neural network accelerator, and the compiler 130 is used to compile the neural network description into instructions to be executed by the acceleration engine 112. When the acceleration engine 112 implements a different type of accelerator, another compiler may be used.

[0023] For example, the compiler 130 may be activated when the computing system 120 receives keyboard, mouse, touch screen, voice command, or other input from the input / output device 108. The input may further include parameters for the compiler 130, such as input code 142 for compilation and configuration options for the compilation process. Once the compiler 130 is activated, the processor 102 may load instructions of the compiler 130 into the processor memory 104 and may execute the instructions.

[0024] exist Figure 1 In the example of , compiler 130 includes a first stage 132, a second stage 136, and a third stage 140, each of which performs a different operation to produce compiled code 144. In other examples, compiler 130 may combine the operations of first stage 132, second stage 136, and / or third stage 140 into fewer stages, or may divide the operations of one of the stages into multiple stages.

[0025] The first stage 132 can receive and process input code 142. The input code 142 can describe a program in a high-level programming language, such as Java, C++, or Tensorflow, as well as many other examples. The input code 142 can describe, for example, steps for performing image recognition, speech recognition, machine translation, or other operations. The input code 142 can be obtained, for example, from the storage device 106. Alternatively, although not shown here, the input code 142 can be located in the processor memory 104, or can be obtained from a network location using the network interface 110. The processing of the input code 142 can include classifying the operations described in the input code 142 into layers, where the output of one layer provides the input to the next layer. The processing can also include identifying steps to be performed by the processor 102 instead of the acceleration engine 112. For example, by executing the driver 122, the processor 102 may need to perform steps such as configuring a direct memory access (DMA) descriptor to move data into or out of the acceleration engine 112, as well as other examples.

[0026] The output 134 of the first level 132 can be organized, for example, in layers, nodes, and connections between nodes of a neural network. The second level 136 can perform intermediate processing on this output 134. For example, the operations performed in any one layer or at any one node in the layer may be too many for the acceleration engine 112 to perform simultaneously. The acceleration engine 112 may, for example, have a limited amount of regional storage space for the data required for the calculation, or the calculation may be more than the acceleration engine 112 can perform at one time. In this example, the first level 132 can decompose the operation of the layer or node into smaller operations, which can fit the local memory of the acceleration engine and / or can fit the computing power of the acceleration engine 112. The processing performed on the output 134 of the first level 132 may include other steps, such as scheduling or determining the order in which the acceleration engine 112 and / or the processor 102 will perform the operations, as well as other examples.

[0027] In various examples, the output 138 of the second stage 136 includes various steps to be performed by the components of the acceleration engine 112 in the order in which the steps are to be performed. The output 138 can be represented as, for example, a data flow graph, where the nodes in the graph represent memory operations, calculations, and other operations, and the edges or connections between the nodes represent dependencies between the nodes, such as data dependencies, memory dependencies, or operation dependencies, among other examples.

[0028] The third stage 140 can operate on the output 138 of the second stage 136 and perform various steps before generating instructions to be executed by the acceleration engine 112. These steps can include, for example, removing redundant dependencies, resolving or handling dependencies between nodes by inserting synchronization instructions into the code, identifying possible optimizations in memory usage or memory bandwidth usage, and other operations. In some examples, the third stage 140 can include a data scheduler 150.

[0029] The output of the third stage 140 is a compiled code 144, which may include machine instructions in a binary format. In some instances, the compiled code 144 may be stored in the processor memory 104. Alternatively or in addition, the compiled code 144 may be copied to the storage device 106 or copied to a network location. As described above, the acceleration engine 112 may be located at a different host system, in which case the compiled code 144 may be sent to another host system via the network interface 110.

[0030] exist Figure 1In the example of , the host system 100 can execute a driver 122 that manages the acceleration engine 112, which can also be referred to as a device driver or a runtime driver. The driver 122 can provide an interface between an application executed by the host system 100 (or another host system) and the acceleration engine 112. For example, the driver 122 can provide an application program interface (API) that defines functions for feeding input data to the acceleration engine 112 and defining operations performed on the input data. In this example and other examples, the driver 122 can configure the acceleration engine 112 to perform operations. For example, the driver 122 can identify a neural network to be executed by the acceleration engine 112, and the location of the compiled code 144 for the neural network in the processor memory 104 or on the storage device 106. The driver 122 can further load into the acceleration engine 112 or cause the acceleration engine 112 to load the compiled code 144, can load or cause the acceleration engine 112 to load the input data to be operated by the neural network, and / or can cause the acceleration engine 112 to execute the input data. Once the acceleration engine 112 has completed, the acceleration engine 112 may notify the driver 122, and the driver 122 may deliver the results back to the application that requested the results.

[0031] Figure 2 is a block diagram illustrating an example of an integrated circuit device that may include an array of processing engines for remapping tensors to a state buffer according to aspects of the present disclosure. Figure 2 The example accelerator 202 is shown in the example. In various examples, for a set of input data (such as input data 250), the accelerator 202 can use the processing engine array 210, the activation engine 216 and / or the pooling engine 218 to perform calculations. In some examples, the example accelerator 202 can be an integrated circuit component of a processor, such as a neural network processor. The processor can have other integrated circuit components, including additional accelerator engines.

[0032] In various embodiments, the memory subsystem 204 may include multiple storage bodies 214. In these embodiments, each storage body 214 may be independently accessible, which means that the reading of one storage body is independent of the reading of another storage body. Similarly, writing to one storage body does not affect or limit writing to a different storage body. In some cases, each storage body may be read and written at the same time. Various techniques may be used to have independently accessible storage bodies 214. For example, each storage body may be a physically separated memory component whose address space is separate and independent of the address space of each other storage body. In this example, each storage body may have at least one read channel and may have at least one separate write channel that may be used simultaneously. In these examples, the memory subsystem 204 may allow access to the read channel or write channel of multiple storage bodies at the same time. As another example, the memory subsystem 204 may include arbitration logic so that, for example, arbitration between the outputs of multiple storage bodies 214 may produce outputs using more than one storage body. In these and other examples, although managed globally by the memory subsystem 204, each storage body may be operated independently of any other storage body.

[0033] Making the memory banks 214 independently accessible can improve the efficiency of the accelerator 202. For example, multiple values ​​can be read simultaneously and provided to each row of the processing engine array 210 so that the entire processing engine array 210 can be used in one clock cycle. As another example, the memory banks 214 can be read while the results calculated by the processing engine array 210 are being written to the memory subsystem 204. In contrast, a single memory may only be able to service one read or write at a time. In the case of a single memory, multiple clock cycles may be required, for example, for reading the input data for each row of the processing engine array 210, before the processing engine array 210 can be started.

[0034] In various embodiments, the memory subsystem 204 can be configured to simultaneously serve multiple clients, including the processing engine array 210, the activation engine 216, the pooling engine 218, and any external client that accesses the memory subsystem 204 through the communication structure 220. In some embodiments, being able to serve multiple clients can mean that the memory subsystem 204 has at least as many storage bodies as clients. In some cases, each row of the processing engine array 210 can be counted as a separate client. In some cases, each column of the processing engine array 210 can output a result, so that each column can be counted as a separate write client. In some cases, the output from the processing engine array 210 can be written to the storage body 214, and then the input data can be provided to the processing engine array 210. As another example, the activation engine 216 and the pooling engine 218 can include multiple execution channels, each of which can be a separate memory client. The storage body 214 can be implemented, for example, using static random access memory (SRAM).

[0035] In various embodiments, the memory subsystem 204 may include control logic. The control logic may, for example, track the address space of each of the memory banks 214, identify the memory bank 214 to be read from or written to, and / or move data between the memory banks 214. In some embodiments, the memory banks 214 may be hardwired to a particular client. For example, a group of memory banks 214 may be hardwired to provide values ​​to the rows of the processing engine array 210, with one memory bank servicing each row. As another example, a group of memory banks may be hardwired to receive values ​​from the columns of the processing engine array 210, with one memory bank receiving data for each column.

[0036] The processing engine array 210 is a computational matrix of the example accelerator 202. The processing engine array 210 can, for example, perform parallel integration, convolution, correlation, and / or matrix multiplication, etc. The processing engine array 210 includes a plurality of processing engines 211 arranged in rows and columns, so that a result output by one processing engine 211 can be directly input into another processing engine 211. Therefore, a processing engine 211 that is not located on the outer edge of the processing engine array 210 can receive data from other processing engines 211 instead of from the memory subsystem 204 for operation.

[0037] In various examples, the processing engine array 210 uses a systolic execution in which data arrives at each processing engine 211 from different directions at regular intervals. In some examples, input data can flow into the processing engine array 210 from the left and weight values ​​can be loaded at the top. In some examples, weights and input data can flow from the left, and partial sums can flow from top to bottom. In these and other examples, multiplication and accumulation operations move through the processing engine array 210 in a diagonal wavefront, with data moving right and downward across the array. Control signals can be input to the left at the same time as weights, and can flow through and flow downward with the calculations.

[0038] In various embodiments, the number of columns in processing engine array 210 determines the computational power of processing engine array 210, and the number of rows determines the memory bandwidth required to achieve maximum utilization of processing engine array 210. Processing engine array 210 may, for example, have 64 columns and 428 rows, or some other number of columns and rows.

[0039] exist Figure 2 An example of a processing engine 211 is illustrated in FIG. As shown in this example, the processing engine 211 may include a multiplier-accumulator circuit. The input from the left may include, for example, input data i and weight values ​​w, where the input data is a value obtained from a set of input data or a set of intermediate results, and the weight values ​​are from a set of weight values ​​that connect one layer of the neural network to the next layer. A set of input data may be, for example, an image submitted for identification or object recognition, an audio clip provided for speech recognition, a text string for natural language processing or machine translation, or the current state of a game that needs to be analyzed to determine the next move, etc. In some examples, the input data and weight values ​​are output to the right for input to the next processing engine 211.

[0040] In the example shown, the input from above may include a partial sum p_in, which is provided by another processing engine 211 or by a previous round of calculations in the processing engine array 210. When starting calculations on a new set of input data, the top row of the processing engine array 210 may receive a fixed value for p_in, such as 0. As shown in this example, i and w are multiplied, and the result is summed with p_in to produce a new partial sum p_out, which may be input into another processing engine 211. Various other embodiments of the processing engine 211 are possible.

[0041] The output from the last row in the processing engine array 210 may be temporarily stored in the result buffer 212. The result may be an intermediate result that may be written to the memory bank 214 to be provided to the processing engine array 210 for further computation. Alternatively, the result may be a final result that, once written to the memory bank 214, may be read from the memory subsystem 204 via the communication fabric 220, the final result being output by the system.

[0042] In some embodiments, accelerator 202 includes activation engine 216. In these embodiments, activation engine 216 can combine results from processing engine array 210 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other instances, it may be necessary to accumulate results from one or more columns in processing engine array 210 to produce an output activation for a single node in the neural network. In some instances, activation engine 216 can be bypassed.

[0043] In various examples, activation engine 216 may include multiple separate execution channels. In these examples, an execution channel may correspond to a column of processing engine array 210, and operations may be performed on the output of the column, the results of which may be stored in memory subsystem 204. In these examples, activation engine 216 may be capable of performing between 1 and n parallel calculations, where n is equal to the number of columns in processing engine array 210. In some cases, one or more of the calculations may be performed simultaneously. Examples of calculations that each execution channel may perform include exponential, square, square root, identity, binary step, bipolar step, S-shaped, and ramp, among other examples.

[0044] In some embodiments, accelerator 202 may include pooling engine 218. Pooling is a combination of the outputs of the columns of processing engine array 210. Combination may include, for example, calculating maximum, minimum, mean, median, summation, multiplication or another logical or mathematical combination. In various instances, pooling engine 218 may include multiple execution channels, which may calculate the values ​​of the corresponding columns of processing engine array 210. In these instances, pooling engine 218 may be able to perform parallel calculations between 1 and n times, where n equals the number of columns in processing engine array 210. In various instances, the execution channels of pooling engine 218 may be parallel and / or simultaneously calculated. In some instances, pooling engine 218 may be bypassed.

[0045] Activation engine 216 and pooling engine 218 may be collectively referred to herein as an execution engine. Processing engine array 210 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine, which may be located outside of accelerator 202.

[0046] Input data 250 may arrive via communication structure 220. Communication structure 220 may connect accelerator 202 to other components of the processor, such as a DMA engine that may obtain input data 250 from an input / output (I / O) device, a storage drive, or a network interface. Input data 250 may be, for example, one-dimensional data, such as a string or a sequence of numbers, or two-dimensional data, such as an array of pixel values ​​of an image or frequency values ​​and amplitude values ​​of an audio signal over time. In some instances, input data 250 may be three-dimensional, such as may be the case of contextual information or virtual reality data used, for example, by an autonomous vehicle. In some embodiments, memory subsystem 204 may include a separate buffer for input data 250. In some embodiments, when input data 250 is received by accelerator 202, input data 250 may be stored in storage body 214.

[0047] In some examples, accelerator 202 may implement a neural network processing engine. In these examples, for a set of input data 250, accelerator 202 may execute a neural network to perform the task for which the neural network was trained. Executing a neural network on a set of input data may be referred to as inference or performing inference.

[0048] The weights of the neural network can be stored in the memory subsystem 204 along with the input data 250 that the neural network will perform operations on. The neural network can also include instructions that can program the processing engine array 210 to perform various calculations on the weights and input data. The instructions can also be stored in the memory subsystem 204, the storage body 214, or a separate instruction buffer. The processing engine array 210 can output intermediate results that represent the outputs of various layers of the neural network. In some cases, the activation engine 216 and / or the pooling engine 218 can be enabled to perform calculations required by certain layers of the neural network. The accelerator 202 can store the intermediate results in the memory subsystem 204 for input into the processing engine array 210 to calculate the results of the next layer of the neural network. The processing engine array 210 can further output the final result from the last layer of the neural network. The final result can be stored in the memory subsystem 204 and then copied to the host processor memory or another location.

[0049] Accelerator 202 may use an array of processing engines to perform computations, such as a systolic array. The array of processing engines may use systolic execution, in which data arrives at each processing engine in the array of processing engines from different directions at regular intervals. For example, referring to Figure 2 , input data can flow into the processing engine array from the left (e.g., through a data bus), and weight values ​​can be loaded from the top (e.g., through a different data bus). The processing engine array may also be referred to as a processing element array in this article. The processing element array may have a fixed size, for example, an array of 128 rows by 64 columns of processing elements or another size. The processing element array may perform calculations such as matrix multiplication and / or other calculations.

[0050] The accelerator may also include a memory subsystem on the integrated circuit (also referred to herein as on-chip memory). The on-chip memory may include a buffer memory for storing data, such as Figure 2 The state buffer shown. The buffer memory can store data, such as neural network weights and data to be processed by the processing element array, as well as intermediate and / or final results of calculations performed by the processing element array. The buffer memory can have a fixed size, such as 128 rows by 128 columns of memory elements or an array of another size. A partition of the buffer memory can be defined as a row of the buffer memory. Thus, a buffer memory with 128 rows can have 128 partitions, with the number of elements in each partition equal to the number of columns in the buffer memory. Each partition is a logical portion of the memory that is independent of other portions of the buffer memory.

[0051] For various processing element array calculations, data in the form of tensors can be retrieved from host memory (e.g., dynamic random access memory (DRAM)) and stored in buffer memory. Tensors are multidimensional arrays. Two-dimensional (2D) matrices can be considered simple tensors, but in general, tensors can have any number of dimensions. When the processing element array loads data to perform calculations, the tensors are stored in the accelerator's buffer memory to reduce latency.

[0052] In practical applications, the number of data elements in a tensor can be much larger than the number of data elements that can be stored in the buffer memory at one time. In this case, the compiler may cause the tensor to be decomposed into smaller blocks of data elements whose size is less than or equal to the dimension of the buffer memory. For example, where the dimension of the buffer memory is p rows by q columns, and the dimension of the tensor is m rows by n columns, where m>p and n>q, the tensor may be decomposed by the compiler into k blocks of data elements, where the number of blocks k is given by Equation 1:

[0053] k=[(m-1) / / 128+1]·[(n-1) / / 128+1] (1)

[0054] Where “ / / ” represents integer or floor division. The compiler can map each block of data elements of the decomposed tensor to p partitions (i.e., rows) of the buffer memory, with q data elements in each partition.

[0055] The number of data elements in the block of the decomposed tensor stored in the buffer memory can be greater than the number of data elements that the array of processing elements can process at one time. For example, the size of the buffer memory can be 128 rows by 128 columns, while the size of the array of processing elements can be 128 rows by 64 columns. In this case, the compiler can further decompose the data block into two data sub - blocks of 128 rows by 64 columns. Each sub - block can be separately loaded into the array of processing elements and processed by the array of processing elements.

[0056] In some cases, for example, when the intermediate results computed by the array of processing elements are stored in the buffer memory, it is necessary to change the way the tensor is mapped to the buffer memory for subsequent computation. For example, it may be necessary to transpose the data elements in one of the matrices to perform matrix multiplication. According to aspects of the present disclosure, the speed of the computation process can be increased by using the operations of the array of processing elements to change the way a tensor (e.g., a matrix) is mapped to the buffer memory. Using the operations of the processing array elements can avoid latency issues caused by uploading data from the buffer memory to the host memory and then re - loading the data into the buffer memory in a properly mapped form.

[0057] As an example, a 2D tensor (i.e., a matrix) A of shape m rows and n columns can be mapped to a buffer memory having p < m rows and q < n columns. As described above, to make matrix A the left operand of a matrix multiplication (e.g., A×B = C), matrix A can be decomposed into k blocks of data elements. Then, each block can be mapped to the buffer memory across p partitions, with q elements in each partition. In the case where matrix A needs to be transposed (i.e., A T ) to be the left operand of a matrix multiplication, each block k can be transposed such that each block is mapped to the buffer memory across q partitions, with p elements in each partition. Matrix A can be, for example, the result of a previous computation from the array of processing elements stored back in the buffer memory.

[0058] Figure 3 is a diagram showing a simplified example of the process of transposing a matrix using the operations of an array of processing elements according to aspects of the present disclosure. The matrix transpose operation can be performed by an accelerator (e.g., Figure 2 the accelerator 202 in Figure 3In the example using a square matrix is ​​shown for ease of explanation, however, the matrix transposition operation is not limited to a square matrix. It should be understood that although the examples of the present invention are explained using a 2D matrix, the 2D matrix can be a block or sub-block of data elements of a larger decomposed multidimensional tensor.

[0059] refer to Figure 3 , the data elements of the decomposed tensor forming the 2D matrix A 311 may be mapped to locations in a buffer memory, for example, a state buffer 310. The state buffer 310 may be an on-chip memory for an accelerator. Figure 3 In the example of , the data elements of matrix A 311 can be mapped to partitions 0-3 312 of the state buffer 310. Each partition 312 of the state buffer 310 can be a logical portion of the memory independent of other portions. Figure 3 310, each partition 312 of the state buffer 310 may be a row of the state buffer 310 with a number of columns equal to the column dimension of the state buffer 310. Data elements 1, 2, 3, and 4 of the matrix A 311 may be mapped to partition 0 of the state buffer 310 starting at a first address (e.g., address 0). Data elements 5-8 of the matrix A 311 may be mapped to partition 1 of the state buffer 310, data elements 9-12 of the matrix A 311 may be mapped to partition 2 of the state buffer 310, and data elements 13-16 of the matrix A 311 may be mapped to partition 3 of the state buffer 310. The data elements of the decomposed tensors stored in the state buffer 310 may be, for example, neural network weights that are results from previous calculations performed by the PE array 320.

[0060] Data elements 1-16 of matrix A 311 stored in state buffer 310 may be loaded into PE array 320. PE array 320 is a systolic array, and data elements 1-16 may be loaded into the systolic array as weights. Figure 2, data elements 1-16 may be loaded from the top into the PE array (e.g., via the first data bus). Instructions generated by the compiler, such as load weight instructions, may cause each of the data elements to be loaded sequentially into a different processing element in the PE array 320. Data elements from each partition 312 in the state buffer 310 may be loaded into a corresponding row of the PE array 320. For example, data elements 1-4 in partition 0 of the state buffer 310 may be loaded into row 0 of the PE array 320, data elements 5-8 in partition 1 of the state buffer 310 may be loaded into row 1 of the PE array 320, data elements 9-12 in partition 2 of the state buffer 310 may be loaded into row 2 of the PE array 320, and data elements 13-16 in partition 3 of the state buffer 310 may be loaded into row 3 of the PE array 320. Thus, data elements 1-16 of matrix A 311 may be loaded into PE array 320 in the same row-column format as they are stored in state buffer 310 , where the data elements in each row of PE array 320 correspond to a partition in state buffer 310 .

[0061] After the data elements (e.g., weights) of matrix A 311 are loaded into the processing elements of PE array 320, a matrix multiplication operation may be performed by the systolic array using identity matrix 330. The matrix multiplication operation may multiply the data elements in the columns of the identity matrix by the weights in the columns of the PE array elements element by element, and accumulate (i.e., sum) the results of the multiplication. Identity matrix 330 may also be stored in state buffer 310. Alternatively, identity matrix 330 may be constructed by a compiler from the "0" data elements and the "1" data elements stored in state buffer 310. Identity matrix 330 may flow into PE array 320 from the left (e.g., via a second data bus).

[0062] It should be noted in the following description that rows 0 341 to 3 344 are identified for the PSUM buffer 340 for convenience of explanation only. Rows 0 341 to 3 344 identified for the PSUM buffer 340 do not correspond to the row partitions in the state buffer 310. When the identity matrix 330 flows from the state buffer 310 to the PE array 320 through the second data bus, during the first cycle of the systolic array, column 3 334 of the identity matrix 330 will be multiplied by column 0 322 of the PE array 320, and the results of the multiplication are summed (i.e., accumulated) in the result buffer (e.g., PSUM buffer) 340. The first element (e.g., weight) in column 0 321 of the PE array 320 will be multiplied by the first element in column 3 334 of the identity matrix 330, the second element in column 0 321 of the PE array 320 will be multiplied by the second element in column 3 334 of the identity matrix 330, the third element in column 0 321 of the PE array 320 will be multiplied by the third element in column 3 334 of the identity matrix 330, the fourth element in column 0 321 of the PE array 320 will be multiplied by the fourth element in column 3 334 of the identity matrix 330, and the results of the multiplications will be summed at partition 0 of the PSUM buffer 340 at row 0 341. The partitions of the PSUM buffer 340 may be arranged in columns rather than rows as in the state buffer 310. The partitions of the PSUM buffer 340 may correspond to the columns of the systolic array PE array 320. Since the first element in column 3 334 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications in row 0 341 of partition 0 stored in the PSUM buffer 340 will be 1.

[0063] During the second cycle of the systolic array, column 3 334 of the identity matrix 330 will be multiplied by column 1 322 of the PE array 320, and column 2 333 of the identity matrix will be multiplied by column 0 321 of the PE array 320, and the results of the multiplications may be summed (i.e., accumulated) in the PSUM buffer. The first element in column 1 322 of the PE array 320 will be multiplied by the first element in column 3 334 of the identity matrix 330, the second element in column 1 322 of the PE array 320 will be multiplied by the second element in column 3 334 of the identity matrix 330, the third element in column 1 322 of the PE array 320 will be multiplied by the third element in column 3 334 of the identity matrix 330, the fourth element in column 1 322 of the PE array 320 will be multiplied by the fourth element in column 3 334 of the identity matrix 330, and the results of the multiplications will be summed in row 0 341 of partition 1 of the PSUM buffer 340. Since the first element in column 3 334 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications in row 0 341 of partition 1 stored in the PSUM buffer 340 will be 2.

[0064] Similarly, during the second cycle of the systolic array, the first element in column 0 321 of the PE array 320 will be multiplied by the first element in column 2 333 of the identity matrix 330, the second element in column 0 321 of the PE array 320 will be multiplied by the second element in column 2 333 of the identity matrix 330, the third element in column 0 321 of the PE array 320 will be multiplied by the third element in column 2 333 of the identity matrix 330, the fourth element in column 0 321 of the PE array 320 will be multiplied by the fourth element in column 2 333 of the identity matrix 330, and the results of the multiplications will be summed in row 1 342 of partition 0 in the PSUM buffer 340. Since the second element in column 2 333 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications stored in row 1 342 of partition 0 in the PSUM buffer 340 will be 5.

[0065] During the third cycle of the systolic array, column 3 334 of the identity matrix 330 will be multiplied by column 2 323 of the PE array 320, column 2 333 of the identity matrix will be multiplied by column 1 322 of the PE array 320, and column 1 332 of the identity matrix 330 will be multiplied by column 0 321 of the PE array 320, and the results of the multiplications may be summed (i.e., accumulated) in the PSUM buffer.

[0066] The first element in column 2 323 of the PE array 320 will be multiplied by the first element in column 3 334 of the identity matrix 330, the second element in column 2 323 of the PE array 320 will be multiplied by the second element in column 3 334 of the identity matrix 330, the third element in column 2 323 of the PE array 320 will be multiplied by the third element in column 3 334 of the identity matrix 330, the fourth element in column 2 323 of the PE array 320 will be multiplied by the fourth element in column 3 334 of the identity matrix 330, and the results of the multiplications will be summed in row 0 341 of partition 2 of the PSUM buffer 340. Since the first element in column 3 334 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications in row 0 341 of partition 2 stored in the PSUM buffer 340 will be 3.

[0067] Similarly, during the third cycle of the systolic array, the first element in column 1 322 of the PE array 320 will be multiplied by the first element in column 2 333 of the identity matrix 330, the second element in column 1 322 of the PE array 320 will be multiplied by the second element in column 2 333 of the identity matrix 330, the third element in column 1 322 of the PE array 320 will be multiplied by the third element in column 2 333 of the identity matrix 330, the fourth element in column 1 322 of the PE array 320 will be multiplied by the fourth element in column 2 333 of the identity matrix 330, and the results of the multiplications will be summed in row 1 342 of partition 1 in the PSUM buffer 340. Since the second element in column 2 333 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications stored in row 1 342 of partition 1 in the PSUM buffer 340 will be 6.

[0068] Furthermore, during the third cycle of the systolic array, the first element in column 0 321 of the PE array 320 will be multiplied by the first element in column 1 332 of the identity matrix 330, the second element in column 0 321 of the PE array 320 will be multiplied by the second element in column 1 332 of the identity matrix 330, the third element in column 0 321 of the PE array 320 will be multiplied by the third element in column 1 332 of the identity matrix 330, the fourth element in column 0 321 of the PE array 320 will be multiplied by the fourth element in column 1 332 of the identity matrix 330, and the results of the multiplications will be summed in row 2 343 of partition 0 in the PSUM buffer 340. Since the third element in column 1 332 of the identity matrix 330 is “1” and the remaining elements are “0”, the sum of the multiplications stored in row 2 343 of partition 0 in the PSUM buffer 340 will be 9.

[0069] During the fourth cycle of the systolic array, column 3 334 of the identity matrix 330 will be multiplied by column 3 324 of the PE array 320, column 2 333 of the identity matrix will be multiplied by column 2 323 of the PE array 320, column 1 332 of the identity matrix 330 will be multiplied by column 1 322 of the PE array 320, column 0 331 of the identity matrix 330 will be multiplied by column 0 321 of the PE array 320, and the results of the multiplications may be summed (i.e., accumulated) in the PSUM buffer.

[0070] The first element in column 3 324 of the PE array 320 will be multiplied by the first element in column 3 334 of the identity matrix 330, the second element in column 3 324 of the PE array 320 will be multiplied by the second element in column 3 334 of the identity matrix 330, the third element in column 3 324 of the PE array 320 will be multiplied by the third element in column 3 334 of the identity matrix 330, the fourth element in column 3 324 of the PE array 320 will be multiplied by the fourth element in column 3 334 of the identity matrix 330, and the results of the multiplications will be summed in row 0 341 of partition 3 in the PSUM buffer 340. Since the first element in column 3 334 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications in row 0 341 of partition 3 stored in the PSUM buffer 340 will be 4.

[0071] Similarly, during the fourth cycle of the systolic array, the first element in column 2 323 of the PE array 320 will be multiplied by the first element in column 2 333 of the identity matrix 330, the second element in column 2 323 of the PE array 320 will be multiplied by the second element in column 2 333 of the identity matrix 330, the third element in column 2 323 of the PE array 320 will be multiplied by the third element in column 2 333 of the identity matrix 330, the fourth element in column 2 323 of the PE array 320 will be multiplied by the fourth element in column 2 333 of the identity matrix 330, and the results of the multiplications will be summed in row 1 342 of partition 2 in the PSUM buffer 340. Since the second element in column 2 333 of the identity matrix 330 is “1” and the remaining elements are “0”, the sum of the multiplications stored in row 1 342 of partition 2 in the PSUM buffer 340 will be 7.

[0072] Furthermore, during the fourth cycle of the systolic array, the first element in column 1 322 of the PE array 320 will be multiplied by the first element in column 1 332 of the identity matrix 330, the second element in column 1 322 of the PE array 320 will be multiplied by the second element in column 1 332 of the identity matrix 330, the third element in column 1 322 of the PE array 320 will be multiplied by the third element in column 1 332 of the identity matrix 330, the fourth element in column 1 322 of the PE array 320 will be multiplied by the fourth element in column 1 332 of the identity matrix 330, and the results of the multiplications will be summed in row 2 343 of partition 1 in the PSUM buffer 340. Since the third element in column 1 332 of the identity matrix 330 is “1” and the remaining elements are “0”, the sum of the multiplications stored in row 2 343 of partition 1 in the PSUM buffer 340 will be 10.

[0073] Finally, during the fourth cycle of the systolic array, the first element in column 0 321 of the PE array 320 will be multiplied by the first element in column 0 331 of the identity matrix 330, the second element in column 0 321 of the PE array 320 will be multiplied by the second element in column 0 331 of the identity matrix 330, the third element in column 0 321 of the PE array 320 will be multiplied by the third element in column 0 331 of the identity matrix 330, the fourth element in column 0 321 of the PE array 320 will be multiplied by the fourth element in column 0 331 of the identity matrix 330, and the results of the multiplications will be summed in row 3 344 of partition-0 in the PSUM buffer 340. Since the fourth element in column 0 331 of the identity matrix 330 is "1" and the remaining elements are "0", the sum of the multiplications stored in row 3 344 of partition 0 in the PSUM buffer 340 will be 13.

[0074] The process continues with identity matrix 330 propagating across PE array 320 until all columns of identity matrix 330 are multiplied by all columns of data elements (e.g., weights) in PE array 320, resulting in values ​​displayed in PSUM buffer 340. The data elements accumulated in column partitions 0-3 of PSUM buffer 340 may be, for example, activated by an engine ( Figure 2 216) or the pooling engine ( Figure 2 The element 218 in is transferred back to the status buffer 310.

[0075] The accelerator hardware configuration is capable of mapping data elements from column partitions 346 in the PSUM buffer 340 to row partitions 314 of the state buffer 310. When the data elements in the PSUM buffer 340 are transferred to the state buffer 310, the data elements in column partitions 0 to 346 in the PSUM buffer 340 may be mapped to row partitions 0 to 314 at a second address (e.g., address N) of the state buffer 310, respectively. Thus, row partition 0 of state buffer 310 contains data elements 1, 5, 9, and 13 (from column partition 0 of PSUM buffer 340), row partition 1 of state buffer 310 contains data elements 2, 6, 10, and 14 (from column partition 1 of PSUM buffer 340), row partition 2 of state buffer 310 contains data elements 3, 7, 11, and 15 (from column partition 2 of PSUM buffer 340), and row partition 3 of state buffer 310 contains data elements 4, 8, 12, and 16 (from column partition 3 of PSUM buffer 340). Thus, matrix A 311 can be remapped into state buffer 310 as transposed matrix A T 314.

[0076] Figure 4is a diagram illustrating a generalized example of a tensor transpose operation according to aspects of the present disclosure. In this example, a block of data elements of a decomposed tensor has dimensions of 128 rows×64 columns and is transposed into a block of data elements having dimensions of 64 rows×128 columns. Figure 4 , a block 410 of the decomposed tensor 410 may be stored in an accelerator (eg, Figure 2 The data elements of the decomposed tensor block 410 stored in the buffer memory 420 may be, for example, neural network weights that are the result of a previous calculation performed by the systolic array 430. The data elements of the decomposed tensor block 410 may be mapped into 128 rows (i.e., 128 partitions 422) of the buffer memory 420, where each row has 64 elements (i.e., 64 columns). Thus, the decomposed tensor block 410 may occupy 128 rows (i.e., rows 0-127) by 64 columns (i.e., columns 0-63) of the buffer memory 420. The size of the decomposed tensor block 410 may correspond to the size of the systolic array 430, in this example 128 rows by 64 columns of processing elements, and may be determined by the compiler. The data elements of the decomposed tensor block 410 may be represented by a subscript indicating the row / column position of the data element, such as the data elements initially stored in the buffer memory 420. For example, Figure 4 As shown, a 0,0 represents the data element initially stored at row 0 column 0 in buffer memory 420, and a 127,63 Represents the data element initially stored in buffer memory 420 at row 127 column 63 .

[0077] The data element a of the decomposed tensor block 410 0,0 -a 127,63 The instructions generated by the compiler may be loaded into the processing elements of the systolic array 430 through a first data bus (not shown) so that all data elements a of the decomposed tensor block 410 are 0,0 -a 127,63 Each data element a of the decomposed tensor block 410 is loaded in one operation. 0,0 -a 127,63 can be loaded into different processing elements of the systolic array 430 so that the data element a in the systolic array 430 0,0 -a 127,63 The row / column position corresponds to the data element a as initially stored in the buffer memory 420. 0,0 -a 127,63 For example, if Figure 4 As shown, data element a is initially mapped to row 0 (i.e., partition 0) column 0 in buffer memory 420.0,0 is loaded into the processing element at row 0, column 0 of the systolic array 430 and the data element a originally mapped to row 127 (i.e., partition 127) column 63 in the buffer memory 420 127,63 is loaded into the processing element at row 127 column 63 of the systolic array 430. Thus, the data elements in each row partition 422 of the buffer memory 420 are loaded into the corresponding row of the systolic array 430.

[0078] After the instruction to load the data elements into the systolic array 430 is completed, the systolic array 430 may begin performing multiply-accumulate operations on the data elements and the identity matrix 440. The identity matrix 440 may be a square matrix having dimensions corresponding to the number of partitions in the buffer memory 420. In this example, since it is a square matrix, the identity matrix 440 may have dimensions of 128 rows by 128 columns. The identity matrix 440 may be stored in the buffer memory 420.

[0079] The identity matrix 440 may flow into the systolic array 430 one column at a time via a second data bus (not shown) during each processing cycle of the systolic array 430. During each processing cycle, as the identity matrix 440 propagates through the systolic array 430, data elements (e.g., weights) stored in a column of processing elements of the systolic array 430 are multiplied by a column of elements in the identity matrix 440, and the products are accumulated in the result buffer 450. For example, during a first processing cycle of the systolic array 430, data element a stored in column 0 432 of the systolic array 430 is multiplied by a column of elements in the identity matrix 440. 0,0 -a 127,0 Multiplying the elements in column 127 442 of the identity matrix 440 element by element makes the data element a in column 0 432 of the systolic array 430 0,0 Multiply by the first element of column 127 442 of identity matrix 440 (i.e., a 0,0 ×1=a 0,0 ). Similarly, data element a in column 0 432 of systolic array 430 1,0 Multiply by the second element in column 127 442 of identity matrix 440 (i.e., a 1,0 ×0=0), and for data element a in column 0 432 of systolic array 430 2,0 -a 127,63 The same is true for elements 2-127 in columns 127-442 of the identity matrix 440. Since all elements of columns 127-442 of the identity matrix 440 are zero except for the first element, when the products are accumulated (i.e., summed), only data element a is included. 0,0 Will be retained. Data element a 0,0may be stored at row 0 452 of partition 0 in result buffer 450. Partitions 458 of result buffer 450 may be arranged in columns rather than rows as in buffer memory 420. Partitions 458 of result buffer 450 may correspond to columns of systolic array 430.

[0080] The multiplication and accumulation operations move through the systolic array 430 with each processing cycle. With each cycle of the systolic array 430, the accumulated products are stored in different locations in the result buffer 450. For example, during a second processing cycle of the systolic array 430, the data elements in column 1 434 of the systolic array 430 are multiplied by the elements in column 127 442 of the identity matrix 440, and the data elements in column 0 432 of the systolic array 430 are multiplied by the elements in column 126 444 of the identity matrix 440. Since all other products are zero, the accumulation of the products of column 1 434 of the systolic array 430 multiplied by column 127 442 of the identity matrix 440 will only produce data element a. 0,1 The cumulative product (i.e., a 0,1 ) can be stored at row 0452 of partition 1 in result buffer 450.

[0081] Similarly, the product-accumulation of column 0 432 of systolic array 430 multiplied by column 1 26 444 of identity matrix 440 will produce only data element a since all other products are zero. 1,0 The accumulated product (i.e., a 1,0 ) may be stored at row 1 454 of partition 0 in result buffer 450. As the multiply-accumulate operation moves through systolic array 430 during each processing cycle, the accumulated product of each column of systolic array 430 performing the multiply-accumulate operation is stored in a subsequent row in the partition corresponding to the particular column of systolic array 430. For example, during a first processing cycle of systolic array 430, the accumulated product a 0,0 stored in row 0 of partition 0 of result buffer 450. During the second processing cycle of systolic array 430, the product a is accumulated. 1,0 Stored in row 1 of partition 0 of result buffer 450, and so on.

[0082] During the last processing cycle of the systolic array 430, the data elements in column 63 436 of the systolic array 430 are multiplied by the elements in column 0 446 of the identity matrix 440. Since all other products are zero, the accumulated product of the multiplications produces only a 127,63 , and accumulate the product a 127,63456 of partition 63 in the result buffer 450. Thus, using the multiply-accumulate operation of the systolic array 430, the data elements of the block 410 of the decomposed tensor 410 loaded into the processing elements of the systolic array 430 are transferred to the result buffer 450 in the same row / column arrangement as they were loaded into the systolic array 430, but the partitions are arranged in columns rather than rows.

[0083] Once the data elements of the decomposed tensor block 410 are stored in the column partition 458 of the result buffer 450, the compiler generates instructions to cause the data elements a 0,0 -a 127,63 The result buffer 450 is transmitted back to the buffer memory 420 by another hardware execution engine of the accelerator, such as a pooling engine or an activation engine, starting from the second memory address (e.g., address N). The hardware of the accelerator can be configured so that the column partition 458 in the result buffer 450 is mapped to the row partition 422 of the buffer memory 420. For example, starting from address N in the buffer memory 420, the data element a of the decomposed tensor block 410 0,0 The data element a of the decomposed tensor block 410 may be mapped to column 0 424 of partition 0 423 1,0 can be mapped to column 1 426 of partition 0 423, and so on, until data element a of the decomposed tensor block 410 127,0 Column 127428 may be mapped to partition 0 423 of the buffer memory 420 .

[0084] Similarly, the data element a of the decomposed tensor block 410 0,63 The data element a of the decomposed tensor block 410 may be mapped to column 0 424 of partition 63 427 1,63 can be mapped to column 1 426 of partition 63 427, and so on, up to data element a of the decomposed tensor block 410. 127,63 428 may be mapped to columns 127 428 of partition 63 427 of buffer memory 420. Thus, using the multiply-accumulate operation of systolic array 430, the decomposed tensor block 410 that was originally mapped into a 128-row partition with 64 columns of buffer memory 420 may be transposed in buffer memory 420 into a tensor block that is mapped into a 64-row partition with 128 columns.

[0085] In some embodiments, when the accumulated products (i.e., data elements of decomposed tensor block 410) fill a row across all column partitions in result buffer 450, the compiler may generate instructions to map one data element in each column partition to a corresponding row partition in buffer memory determined by the compiler. In some embodiments, after all accumulated products (i.e., all data elements of decomposed tensor block 410) have been stored in column partitions of result buffer 450, the compiler may generate instructions to map all data elements in each column partition of result buffer 450 to a corresponding row partition in buffer memory 420 determined by the compiler.

[0086] Figure 5 is a flow chart illustrating an example of a method 500 for performing a transpose operation on a decomposed tensor block according to aspects of the present disclosure. These methods can be implemented by the above-mentioned system, for example, Figure 1 The compilers shown and Figure 2 The accelerator shown.

[0087] refer to Figure 5 At block 510, the compiler may receive program code to cause the execution engine to perform a set of operations. For example, the program code may cause various elements of the execution engine (e.g., systolic arrays, buffer memories, result buffers, etc.) to perform data processing operations.

[0088] At block 515, it may be determined that the data processing operation includes a matrix transpose operation. In some embodiments, to perform a matrix transpose operation, it may be necessary to decompose the multidimensional tensor into smaller blocks of data elements having dimensions compatible with the dimensions of the execution engine's buffer memory.

[0089] At block 520, the compiler may generate instructions for causing the execution engine to perform data processing operations. For example, the compiler may generate a set of instructions to decompose a multidimensional tensor into blocks of data elements whose dimensions are compatible with the dimensions of the buffer memory, and cause the execution engine (i.e., the accelerator) to perform a matrix transpose operation on the decomposed multidimensional tensor blocks. The compiler may also generate instructions to implement the matrix transpose operation using identity matrix multiplication.

[0090] Starting at block 525, instructions generated by the compiler may be executed. A multidimensional tensor may be decomposed into smaller blocks of data elements. A tensor may be a multidimensional tensor initially stored in a host memory (e.g., DRAM). In practical applications, the number of data elements in a tensor may be much larger than the number of data elements that may be stored in an integrated circuit, such as a buffer memory in an accelerator, at one time. The compiler may generate instructions to decompose the tensor into smaller blocks of a size compatible with the buffer memory. For example, where the dimension of the buffer memory is p rows by q columns, the dimension of the tensor is m rows by n columns, where m>p and n>q, the tensor may be decomposed into k blocks by instructions generated by the compiler, where the number of blocks k is given by equation 1 above. In some cases, a block of data elements may be the result of a previous calculation stored in a buffer memory.

[0091] At block 530, one or more blocks of the decomposed tensor may be mapped into row partitions of the buffer memory. The buffer memory may be configured to have partitions corresponding to the storage rows in the buffer memory. Each partition is a logical portion of the memory that is independent of other portions of the buffer memory. For example, the dimensions of the buffer memory may be 128 rows by 128 columns. Each row of the buffer memory may be a partition, and each partition may contain 128 columns. Blocks of decomposed tensors of the same or smaller dimensions may be mapped into the buffer memory.

[0092] At block 535, the decomposed tensor block may be loaded from the buffer memory into the systolic array. In some cases, the number of data elements in the decomposed tensor block stored in the buffer memory may be greater than the number of data elements that the processing element array can process at one time. For example, the size of the buffer memory may be 128 rows by 128 columns, while the size of the processing element array may be 128 rows by 64 columns. In this case, the compiler may generate instructions to further decompose the data block into two 128 rows by 64 columns of data sub-blocks.

[0093] The data elements of the decomposed tensor block or sub-block may be loaded into the processing elements of the systolic array via the first data bus via compiler generated instructions such that all data elements of the decomposed tensor block are loaded in one operation. Each data element of the decomposed tensor block may be loaded into a different processing element of the systolic array such that the row / column position of the data element in the systolic array corresponds to the row / column position of the data element as originally stored in the buffer memory. Thus, the data elements in each row (i.e., partition) of the buffer memory may be loaded into a corresponding row of the systolic array.

[0094] At block 540, a matrix multiplication operation may be performed on the identity matrix, and the products accumulated in a result buffer. For each processing cycle, as the identity matrix flows into the systolic array on the second data bus, the systolic array may perform multiplications between one or more columns of the identity matrix and one or more columns of data elements stored in the processing elements of the systolic array. Since each column of the identity matrix contains only a "1" in one position and 0s in all other positions, the column multiplication may produce only one non-zero product, which is the data element corresponding to the "1" position in the column of the identity matrix.

[0095] At block 545, the accumulated products are stored in column partitions of the result buffer. The partitions of the result buffer may be arranged in columns, rather than rows as in the buffer memory. Each column partition of the result buffer may correspond to a column of the systolic array. As the multiply-accumulate operation moves through the systolic array during each processing cycle, the accumulated products of each column of the systolic array that performs the multiply-accumulate operation are stored in subsequent rows in the partition corresponding to the particular column of the systolic array.

[0096] For example, refer to Figure 4 During the first processing cycle of the systolic array, the accumulated product of column 0 of the data elements in the systolic array multiplied by column 127 of the identity matrix may be data element a 0,0 Data element a 0,0 may be stored in row 0 of partition 0 in the result buffer. During the second processing cycle of the systolic array, the data element in column 1 of the systolic array is multiplied by the element in column 127 of the identity matrix, and the data element in column 0 of the systolic array is multiplied by the element in column 126 of the identity matrix. Since all other products are 0, the product accumulation of column 1 of the systolic array multiplied by column 127 of the identity matrix will produce only data element a 0,1 ,. The cumulative product (i.e., a 0,1 ) can be stored at row 0 of partition 1 in the result buffer. Since all other products are 0, the product-accumulation of column 0 of the systolic array and column 126 of the identity matrix will produce only data element a 1,0 The accumulated product (i.e., a 1,0 ) may be stored at row 1 of partition 0 in the result buffer. The process may continue until all columns of the identity matrix are propagated through and multiplied by all columns of data elements in the systolic array. Using the multiply-accumulate operation of the systolic array, the data elements of the decomposed tensor blocks loaded into the processing elements of the systolic array are transferred to the result buffer in the same row / column arrangement as they were loaded into the systolic array, but arranged in columns rather than rows in the partition.

[0097] return Figure 5, at block 550, the stored product may be mapped from the column partition of the result buffer to the row partition of the buffer memory. Once the data elements of the decomposed tensor block are stored in the column partition of the result buffer, the compiler-generated instructions may cause the data elements to be mapped back to the buffer memory starting at a second memory address (e.g., address N) through another hardware execution engine of the accelerator, such as a pooling engine or an activation engine. The hardware of the accelerator may be configured to cause the column partition in the result buffer to be transposed to the row partition of the buffer memory. Thus, using the multiplication-accumulation operation of the systolic array, the decomposed tensor block with 128 row partitions of 64 columns initially stored in the buffer memory may be transposed to a tensor block with 64 row partitions of 128 columns in the buffer memory.

[0098] In some embodiments, when each partition in the result buffer becomes full, that is, when the data elements occupy all rows in the column partition, the compiler can generate instructions to map the data elements in the column partition to the row partition determined by the compiler in the buffer memory. By using the identity matrix multiplication operation in the execution engine as described above to implement the matrix transposition operation, the matrix transposition operation can be performed without having to write data to the host memory (e.g., DRAM) and read back the data from the host memory. In other words, using the above technology, the host memory can be bypassed when performing the matrix transposition operation. Therefore, compared with the implementation scheme that requires the use of the host memory, the calculation time for performing the matrix transposition operation can be reduced. In addition, the matrix transposition operation can be performed in the execution engine to release the processing cycle of the host processor.

[0099] It should be understood that Figure 5 The specific steps shown provide a specific method for performing a transpose operation on a decomposed tensor block according to an embodiment. According to alternative embodiments, other step sequences may also be performed. For example, alternative embodiments may perform the steps outlined above in a different order. In addition, Figure 5 Each step shown in the embodiment may include multiple sub-steps, which may be performed in various orders suitable for each step. In addition, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0100] The method for performing a transpose operation on a decomposed tensor block may be embodied on a non-transitory computer-readable medium known to those skilled in the art, in which a program containing computer-executable instructions for causing a processor, computer or other programmable device to perform the operations of the method has been stored.

[0101] Figure 6600 is an example of an integrated circuit that may include one or more accelerators 602a-602n, which may be similar to Figure 2 The accelerator shown.

[0102] exist Figure 6 In the example of, acceleration engine 600 comprises multiple accelerators 602a-602n, and each accelerator in the accelerator can perform a group of operations. In each example, accelerators 602a-602n are used for the operation of a specific type, so that accelerators 602a-602n can perform these operations at a much faster speed than when similar operations are performed by a general-purpose processor. In each example, in order to perform a group of operations, the input data to be operated on must first be moved to accelerators 602a-602n. In addition, in some cases, program code is also moved to accelerators 602a-602n, and the program code programs the operation performed by accelerators 602a-602n on the data. In the example shown, acceleration engine 600 comprises n accelerators 602a-602n. The example of the accelerator that can be included in acceleration engine 600 comprises graphics accelerators, floating point accelerators, neural network accelerators, etc. In various examples, accelerators 602a-602n can each be identical (eg, each is a graphics accelerator) or can be different (eg, accelerators 602a-602n include a graphics accelerator, a floating point accelerator, and a neural network accelerator).

[0103] The example acceleration engine 600 further includes DRAM controllers 642a-642k that communicate with external memory. In this example, the external memory is implemented using DRAM 630. In the example shown, the acceleration engine 600 includes k DRAM controllers 642a-642k, each of which can be able to communicate with a group of independent DRAM access bodies. In other examples, other types of RAM technologies can be used for external memory. DRAM controllers 642a-642k can also be referred to as memory controllers.

[0104] In various instances, input data and / or program code for the accelerators 602a-602n may be stored in the DRAM 630. Different programs may cause the accelerators 602a-602n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program may configure the neural network accelerator to perform speech recognition, while another program may configure the neural network accelerator to perform image recognition. In various instances, different accelerators 602a-602n may be programmed with different programs so that each accelerator performs a different set of operations. In various instances, the processors 648a-648s may manage the movement of program code from the DRAM 630 to the accelerators 602a-602n.

[0105] The example acceleration engine 600 further includes an I / O controller 644a-644p that communicates with an I / O device 632 in the system. The acceleration engine 600 can communicate with the I / O device through, for example, a processor bus. In some instances, the processor bus can be implemented using a variant of a peripheral component interconnect (PCI) and / or PCI bus protocol. The processor bus can connect the acceleration engine 600 to an I / O device, such as an input and output device, a memory controller, a storage device, and / or a network interface card, etc. In some instances, the I / O controller 644-644p can enable the acceleration engine 600 to act as an I / O device of a host processor. For example, the acceleration engine 600 can be a recipient of input data from a host processor and a command indicating an operation (e.g., a specific calculation or analysis) to be performed on the input data. In the example shown, the acceleration engine 600 includes p I / O controllers 644a-644p, each of which can include a separate root complex and can communicate with a set of separate I / O devices 632. In other examples, other standardized bus protocols such as Ultra Path Interconnect (UPI) can be used for the host bus. In other examples, a proprietary bus protocol can be used.

[0106] Data movement within the acceleration engine 600 may be managed by one or more processors 648a-648s, which may also be referred to as data management processors. Figure 6In an example of, the acceleration engine 600 includes s processors 648a-648s incorporated into the device (e.g., on the same silicon chip). In other examples, the processors 648a-648s may be external to the acceleration engine 600 (e.g., on different dies and / or in different packages). In some examples, the processors 648a-648s may manage the movement of data from the I / O device 632 to the accelerators 602a-602n or DRAM 630. For example, input data may be located at the I / O device 632 or in the processor memory, and the processors 648a-648s may move the input from the I / O device 632 or the processor memory to the accelerator or DRAM 630. As another example, the program code of the accelerators 602a-602n may be located on the I / O device 632 or in the processor memory.

[0107] The example acceleration engine 600 further includes a DMA engine 646a-646d that can move data between accelerators 602a-602n, DRAM controllers 642a-642k and I / O controllers 644a-644p. In the example shown, the acceleration engine 600 includes d DMA engines 646a-646d. In some embodiments, the DMA engines 646a-646d can be assigned with specific tasks, such as moving data from DRAM controllers 642a-642d to accelerators 602a-602n, or moving data between I / O controllers 644a-644p and accelerators 602a-602n. For example, these tasks can be assigned by queuing descriptors with DMA engines 646a-646d, wherein the descriptor identifies the address of a data block and the operation to be performed (e.g., read or write). For example, a descriptor can manage a DMA engine to instruct a DMA controller to read a data block from DRAM 630. As another example, the descriptor may instruct the DMA engine to write data read by the DMA controller to the accelerator. Additional descriptors may be used to move data from the accelerator to the DRAM 630.

[0108] In various instances, each processor in processors 648a-648s can be responsible for managing the data movement of different accelerators. In some instances, a processor can manage the data movement of more than one accelerator. Similarly, in various instances, each processor in processors 648a-648s can be assigned to one or more DMA engines 646a-646d. In these and other instances, the association between processors 648a-648s, accelerators 602a-602n and DMA engines 646a-646d is determined by the program code executed by each corresponding processor.

[0109] In the example acceleration engine 600, the various components may communicate via a chip interconnect 620. The chip interconnect 620 primarily includes wiring for routing data between the components of the acceleration engine 600. In some cases, the chip interconnect 620 may include a minimal amount of logic, such as multiplexers to control the direction of data, flip-flops to handle clock domain crossings, and timing logic.

[0110] Figure 7 A block diagram is included showing an example of a host system 700 in which an acceleration engine 760 may be used according to aspects of the present disclosure. Figure 7 The acceleration engine 760 may include Figure 6 An example of an arrangement of one or more accelerators is shown. Figure 7 The example host system 700 includes an acceleration engine 760, a host processor 772, DRAM 730 or processor memory, I / O devices 732, and a support system 774. In various embodiments, the host system 700 may include other hardware not shown here.

[0111] Host processor 772 is a general purpose integrated circuit capable of executing program instructions. In some instances, host processor 772 may include multiple processing cores. A multi-core processor may include multiple processing units within the same processor. In some instances, host system 700 may include more than one host processor 772. In some instances, host processor 772 and acceleration engine 760 may be a chip, such as one or more integrated circuits in the same package.

[0112] In various examples, the host processor 772 can communicate with other components in the host system 700 through one or more communication channels. For example, the host system 700 can include a host processor bus, and the host processor 772 can use, for example, the host processor bus to communicate with the DRAM 730. As another example, the host system 700 can include an I / O bus, such as a PCI-based bus, and the host processor 772 can use, for example, the I / O bus to communicate with the acceleration engine 760 and / or the I / O device 732. In various examples, alternatively or additionally, the host system 700 can include other communication channels or buses, such as a serial bus, a power management bus, a storage device bus, etc.

[0113] In some instances, a software program executed on the host processor 772 can receive or generate input to be processed by the acceleration engine 760. In some instances, the program can select an appropriate neural network to execute a given input. For example, the program can be used for language translation, and one or more neural networks capable of speech recognition and / or machine translation can be selected. In these and other instances, the program can configure the acceleration engine 760 with a neural network for execution, and / or a neural network processing engine previously configured to execute the desired neural network can be selected on the acceleration engine 760. In some instances, once the acceleration engine 760 begins to infer the input data, the host processor 772 can manage the movement of data (such as weights, instructions, intermediate results, results of conditional layers, and / or final results) into or out of the acceleration engine 760.

[0114] In some examples, a software program using acceleration engine 760 to perform inference can read results from a conditional layer from acceleration engine 760 and / or from a storage location such as DRAM 730. In these examples, the program can determine what action the neural network should take next. For example, the program can decide to terminate the inference. As another example, the program can determine to change the direction of the inference, which can be translated to the next layer by the lower level code and / or neural network processor for execution. In these and other examples, the execution flow of the neural network can be coordinated by software.

[0115] DRAM 730 is a memory used by host processor 772 to store program code being executed by host processor 772 and values ​​to be operated on. In some examples, data for a neural network (e.g., weight values, instructions, and other data) may be stored in whole or in part in DRAM 730. DRAM is a general term for processor memory, and while DRAM is volatile memory, processor memory may be volatile and / or non-volatile. Although not shown here, host system 700 may include other volatile memory and non-volatile memory for other purposes. For example, host system 700 may include a read-only memory (ROM) that stores boot code and / or basic input / output system (BIOS) code that boots host system 700 when powered on.

[0116] Although not shown here, DRAM 730 may store instructions for various programs that may be loaded into and executed by host processor 772. For example, DRAM 730 may store instructions for a computing system, one or more data stores, one or more application programs, one or more drivers, and / or services for implementing the features disclosed herein.

[0117] The computing system can manage and orchestrate the overall computing of the host system 700, such as scheduling tasks, executing applications, and / or overall computing of controller peripherals and other computing. In some instances, the host system 700 can host one or more virtual machines. In these instances, each virtual machine can be configured to execute its own computing system. Examples of computing systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. Alternatively or in addition, the computing system can be a proprietary computing system.

[0118] The data storage area may contain permanent data or temporary data used and / or operated by a computing system, an application program or a driver. Examples of such data include web pages, video data, audio data, images, user data, etc. In some instances, the information in the data storage area may be provided to a user device via a network. In some cases, the data storage area may additionally or alternatively contain stored applications and / or drivers. Additionally or alternatively, the data storage area may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data storage area may be machine-readable object code, source code, interpretation code, or intermediate code.

[0119] Drivers may include programs that provide communication between components in the host system 700. For example, some drivers may provide communication between the computing system and peripheral devices or I / O devices 732. Alternatively or in addition, some drivers may provide communication between applications and the computing system, and / or between applications and peripheral devices accessible by the host system 700. In many cases, drivers may include drivers that provide well-understood functionality (e.g., printer drivers, display drivers, hard disk drivers, solid-state device drivers, etc.). In other cases, drivers may provide proprietary or specialized functionality.

[0120] I / O devices 732 may include hardware connected to user input and output devices, such as a keyboard, mouse, pen, tablet computer, voice input device, touch input device, display or monitor, speaker, and printer, among other devices. I / O devices 732 may also include a storage drive and / or a network interface for connecting to network 780. For example, in various examples, host system 700 may communicate with storage devices, user terminals, other computing devices or servers, and / or other networks using a network interface.

[0121] In various examples, one or more of the I / O devices 732 may be storage devices. In these examples, the storage device includes a non-volatile memory and may store program instructions and / or data. Examples of storage devices include magnetic storage devices, optical disks, solid-state disks, flash memories, and / or tape storage devices, etc. The storage device may be housed in the same rack as the host system 700 or may be located in an external hard disk enclosure. The storage device may be fixed (e.g., connected by screws) or removable (e.g., with a physical release mechanism and a possible hot-swap mechanism).

[0122] The storage device, DRAM 730, and any other memory components in the host system 700 are examples of computer-readable storage media. A computer-readable storage medium is a physical medium that can store data in a format that can be read by a device such as the host processor 772. A computer-readable storage medium can be non-transitory. A non-transitory computer-readable medium can retain the data stored thereon when the medium is not powered. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tapes, optical disks, flash memory devices, and solid-state drives, among others. As used herein, a computer-readable storage medium does not include a computer-readable communication medium.

[0123] In various examples, data stored on a computer-readable storage medium may include program instructions, data structures, program modules, libraries, other software program components, and / or other data that may be transmitted within a data signal, such as a carrier wave or other transmission. Additionally or alternatively, a computer-readable storage medium may include documents, images, video, audio, and other data that may be operated on or manipulated using a software program.

[0124] In various examples, one or more of the I / O devices 732 may be PCI-based devices. In these examples, the PCI-based I / O device includes a PCI interface that communicates with the host system 700. The term "PCI" or "PCI-based" may be used to describe any protocol in the PCI bus protocol family, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP) and PCI-Express (PCIe) or any other improved or derived protocol based on the PCI protocol discussed herein. The PCI-based protocol is a standard bus protocol for connecting devices such as local peripheral devices to a host device. The standard bus protocol is a data transmission protocol whose specifications have been defined and adopted by various manufacturers. The manufacturer ensures that the compatible device is compatible with the computing system that implements the bus protocol, and vice versa. As used herein, the PCI-based device also includes a device that communicates using fast non-volatile memory (NVMe). NVMe is a device interface specification for accessing a non-volatile storage medium connected to a computing system using PCIe.

[0125] A PCI-based device may include one or more functions. "Function" describes the hardware and / or software of operations that may be provided by a PCI-based device. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers, etc. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some embodiments, a PCI-based device may have up to eight functions.

[0126] In some instances, a PCI-based device may include single root I / O virtualization (SR-IOV). SR-IOV is an extended function that may be included in a PCI-based device. SR-IOV allows a physical resource (e.g., a single network interface controller) to be displayed as multiple virtual resources (e.g., sixty-four network interface controllers). Therefore, a PCI-based device (e.g., a network interface controller) that provides a certain function may appear to be multiple devices that provide the same function for a device using a PCI-based device. The functions of a storage adapter device that supports SR-IOV can be classified as a physical function (PF) or a virtual function (VF). A physical function refers to a function that can be discovered, managed, and manipulated in a device. A physical function has configuration resources that can be used to configure or control a storage adapter device. A physical function contains the same configuration address space and memory address space as a non-virtualized device. A physical function may have multiple virtual functions associated with it. A virtual function is similar to a physical function, but is a lightweight function that may generally lack configuration resources and is generally controlled by the configuration of its underlying physical function. Each of the physical function and / or virtual function may be assigned to a corresponding execution thread (e.g., a virtual machine) running on a host device.

[0127] In various embodiments, the support system 774 may include hardware for coordinating the operations of the acceleration engine 760. For example, the support system 774 may include a microprocessor that coordinates the activities of the acceleration engine 760, including data moving around on the acceleration engine 760. In this example, the microprocessor may be an integrated circuit that can execute microcode. Microcode is a program code that can give an integrated circuit some flexibility in the operations that the integrated circuit can perform, but because the program code uses a limited set of instructions, the microprocessor may have more limited capabilities than the host processor 772. In some examples, the program executed by the microprocessor is stored on the hardware of the microprocessor, or on a non-volatile memory chip in the host system 700. In some examples, the microprocessor and the acceleration engine 760 can be on a chip, such as an integrated circuit on the same die and in the same package.

[0128] In some instances, the support system 774 may be responsible for obtaining instructions from the host processor 772 when a program executed on the host processor 772 requests execution of a neural network. For example, the host processor 772 may provide the support system 774 with a set of input data and a task to be performed on the set of input data. In this instance, the support system 774 may identify a neural network that can perform the task and may program the acceleration engine 760 to execute the neural network on the set of input data. In some instances, the support system 774 may only need to select an appropriate neural network processing engine of the neural network processor. In some instances, the support system 774 may need to load data for the neural network onto the acceleration engine 760 before the acceleration engine 760 can begin executing the neural network. In these and other instances, the support system 774 may further receive the output of executing the neural network and provide the output back to the host processor 772.

[0129] In some instances, the operations of support system 774 may be handled by host processor 772. In these instances, support system 774 may not be required and may be omitted from host system 700.

[0130] In various examples, host system 700 may include a combination of host systems, processor nodes, storage subsystems, and I / O chassis representing a user device, a service provider computer, or a third party computer.

[0131] The user device may include a computing device for accessing an application (e.g., a web browser or a mobile device application). In some instances, the application may be hosted, managed, and / or provided by a computing resource service or service provider. The application may enable a user to interact with a service provider computer to, for example, access network content (e.g., web pages, music, videos, etc.). The user device may be a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a netbook computer, a desktop computer, a thin terminal device, a tablet computer, an e-book reader, a game console, etc. In some instances, the user device may communicate with the service provider computer through one or more networks. In addition, the user device may be part of a distributed system managed, controlled, or otherwise part of a service provider computer (e.g., a console device integrated with a service provider computer).

[0132] The host system 700 may also represent one or more service provider computers. The service provider computer may provide a local application configured to run on a user device, and the user may interact with the local application. In some instances, the service provider computer may provide computing resources, such as but not limited to client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, etc. The service provider computer may also be operable to provide network hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, etc. In some instances, the service provider computer may provide one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that are quickly provided and released. These computing resources may include computing, network and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. The service provider computer may include one or more servers, which may be arranged in a cluster, as a server group, or as separate servers that are unrelated to each other, and may host applications and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some cases, additionally or alternatively, the service provider computer may include a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a netbook computer, a server computer, a thin terminal device, a tablet computer, a gaming console, etc. In some instances, the service provider computer may communicate with one or more third-party computers.

[0133] Figure 8 A diagram of an example network 800 including one or more host systems, such as Figure 7The host system shown. For example, Figure 8 The example network 800 includes a plurality of nodes 802a-802h, wherein one or more of the plurality of nodes may be Figure 7 The host system shown. Other nodes in nodes 802a-802h may be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing instructions, and a network interface for connecting to network 800.

[0134] In various examples, network 800 can be used to process data. For example, input data can be received at one of nodes 802a-802h or received from other networks 808 with which network 800 can communicate. In this example, input data can be directed to a node in the network 800 that includes an acceleration engine, for the acceleration engine to run and generate results. Then, the result can be transmitted to a node or other network that receives the input data. In various examples, input data can be accumulated from various sources, including one or more nodes in nodes 802a-802h and / or a computing device located in other networks 808, and the accumulated input data can be directed to one or more host systems in network 800. Then, the result from the host system can be distributed back to the source of the collected input data.

[0135] In various instances, one or more of nodes 802a-802h may be responsible for operations such as accumulating input data for host system operations, tracking which host systems are busy and which host systems can accept more work, determining whether host systems are operating correctly and / or most efficiently, monitoring network security and / or other management operations.

[0136] exist Figure 8 In the example of , nodes 802a-802h are connected to each other using a switching fabric with point-to-point links. The switching fabric includes multiple switches 804a-804d, which can be arranged in a multi-layer network such as a Clos network. A network device that filters and forwards data packets between local area network (LAN) segments can be referred to as a switch. Switches typically operate at the data link layer (layer 2) of the open system interconnection (OSI) reference model, and sometimes operate at the network layer (layer 3), and can support several data packet protocols. Figure 8 The switches 804a-804d may be connected to the nodes 802a-802h and provide multiple paths between any two nodes.

[0137] The network 800 may also include one or more network devices, such as routers 806, for connecting to other networks 808. Routers use headers and forwarding tables to determine the best path to forward packets, and use protocols such as the Internet Control Message Protocol (ICMP) to communicate with each other and configure the best route between any two devices. Figure 8 The router 806 can be used to connect to other networks 808, such as a subnet, a LAN, a wide area network (WAN), and / or the Internet.

[0138] In some examples, the network 800 may include any one or combination of many different types of networks, such as a wired network, the Internet, a wireless network, a cellular network, and other private and / or public networks. If present, the interconnected switches 804a-804d and routers 806 may be referred to as a switching fabric 810, a fabric, a network fabric, or simply a network. In the context of computer networks, the terms "fabric" and "network" may be used interchangeably herein.

[0139] Nodes 802a-802h may be any combination of host systems, processor nodes, storage subsystems, and I / O chassis representing user devices, service provider computers, or third-party computers.

[0140] The user device may include a computing device for accessing an application 832 (e.g., a web browser or a mobile device application). In some aspects, the application 832 may be hosted, managed, and / or provided by a computing resource service or service provider. The application 832 may allow a user to interact with a service provider computer to, for example, access network content (e.g., web pages, music, videos, etc.). The user device may be a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a netbook computer, a desktop computer, a thin terminal device, a tablet computer, an e-book reader, a game console, etc. In some instances, the user device may communicate with the service provider computer via other networks 808. In addition, the user device may be part of a distributed system managed, controlled, or otherwise part of a service provider computer (e.g., a console device integrated with a service provider computer).

[0141] Figure 8The node may also represent one or more service provider computers. One or more service provider computers may provide local applications configured to run on a user device, and the user may interact with the local application. In some instances, the service provider computer may provide computing resources, such as but not limited to client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, etc. The service provider computer may also be operable to provide network hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, etc. to users. In some instances, the service provider computer may provide one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that are quickly provided and released. These computing resources may include computing, network and / or storage devices. The hosted computing environment may also be referred to as a cloud computing environment. The service provider computer may include one or more servers, which may be arranged in a cluster, as a server group, or as separate servers that are unrelated to each other, and may host applications 832 and / or cloud-based software services. These servers may be configured as part of an integrated distributed computing environment. In some aspects, additionally or alternatively, the service provider computer may include a computing device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a desktop computer, a netbook computer, a server computer, a thin terminal device, a tablet computer, a gaming console, etc. In some cases, the service provider computer may communicate with one or more third-party computers.

[0142] In one example configuration, nodes 802a-802h may include at least one memory 818 and one or more processing units (or processors 820). Processor 820 may be implemented in hardware, computer executable instructions, firmware, or a combination thereof. Computer executable instructions or firmware implementations of processor 820 may include computer executable instructions or machine executable instructions written in any suitable programming language for performing the various functions described.

[0143] In some cases, the hardware processor 820 can be a single-core processor or a multi-core processor. A multi-core processor can include multiple processing units in the same processor. In some instances, the multi-core processor can share certain resources, such as a bus and a second-level cache or a third-level cache. In some cases, each core in a single-core processor or a multi-core processor can also include multiple execution logical processors (or execution threads). In such cores (e.g., cores with multiple logical processors), several levels of the execution pipeline and lower-level caches can also be shared.

[0144] The memory 818 can store program instructions that can be loaded and executed on the processor 820 and the data generated during the execution of these programs. Depending on the configuration and type of the nodes 802a-802h, the memory 818 can be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). The memory 818 can include an operating system 828, one or more data storage areas 830, one or more application programs 832, one or more drivers 834, and / or services for implementing the features disclosed herein.

[0145] The computing system 828 can support the basic functions of the nodes 802a-802h, such as scheduling tasks, executing applications and / or controlling peripheral devices. In some embodiments, the service provider computer can host one or more virtual machines. In these embodiments, each virtual machine can be configured to execute its own computing system. Examples of computing systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. The computing system 828 can also be a proprietary computing system.

[0146] The data storage area 830 may include permanent data or temporary data used and / or operated by the computing system 828, the application 832, or the driver 834. Examples of such data include web pages, video data, audio data, images, user data, etc. In some embodiments, the information in the data storage area 830 may be provided to the user device via the network 808. In some cases, additionally or alternatively, the data storage area 830 may include stored applications and / or drivers. Additionally or alternatively, the data storage area 830 may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data storage area 830 may be machine-readable object code, source code, interpretation code, or intermediate code.

[0147] Driver 834 includes programs that can provide communication between components in the node. For example, some drivers 834 can provide communication between computing system 828 and other storage devices 822, network devices 824 and / or I / O devices 826. Alternatively or in addition, some drivers 834 can provide communication between application program 832 and computing system 828, and / or communication between application program 832 and peripheral devices accessible by service provider computer. In many cases, driver 834 can include drivers (for example, printer driver, display driver, hard disk driver and solid state device driver) that provide easy-to-understand functions. In other cases, driver 834 can provide proprietary or specialized functions.

[0148] The service provider computer or server may also include another storage device 822, which may include a removable storage device and / or a non-removable storage device. Another storage device 822 may include a magnetic storage device, an optical disk, a solid state disk, a flash memory, and / or a tape storage device. Another storage device 822 may be contained in the same rack as the node 802a-802h or may be located in an external hard disk box. Storage device 818 and / or another storage device 822 and its associated computer-readable medium may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computing devices. In some embodiments, storage device 818 may include multiple different types of storage devices, such as SRAM, DRAM, or ROM.

[0149] Removable and non-removable memory 818 and additional storage device 822 are examples of computer-readable storage media. For example, a computer-readable storage medium may be included in a volatile or non-volatile removable or non-removable medium implemented in a method or technology for storing information, and the information includes, for example, computer-readable instructions, data structures, program modules, or other data. Memory 818 and additional storage device 822 are examples of computer storage media. Other types of computer storage media that may be present in nodes 802a-802h may include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical storage devices, cassette devices, tape devices, disk storage devices or other magnetic storage devices, solid-state drives, or some other media that may be used to store desired information and may be accessed by nodes 802a-802h. Computer-readable media also include a combination of any of the above-mentioned media types, including multiple units of a media type.

[0150] Alternatively or additionally, computer readable communication media may include computer readable instructions, program modules, or other data transmitted within a data signal, such as a carrier wave or other transmission. However, as used herein, computer readable storage media does not include computer readable communication media.

[0151] Nodes 802a-802h may also include I / O devices 826, such as keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, etc. Nodes 802a-802h may also include one or more communication channels 836. Communication channels 836 may provide a medium through which the various components of nodes 802a-802h may communicate. One or more communication channels 836 may take the form of a bus, a ring, a switching structure, or a network.

[0152] The nodes 802a - 802h may also contain a network device 824 that allows the nodes 802a - 802h to communicate with a stored database, another computing device or server, a user terminal, and / or other devices on the network 800 .

[0153] In some embodiments, the network device 824 is a peripheral device, such as a PCI-based device. In these embodiments, the network device 824 includes a PCI interface for communicating with a host device. The term "PCI" or "PCI-based" can be used to describe any protocol in the PCI bus protocol family, including the original PCI standard, PCI-X, accelerated graphics port (AGP) and PCI-Express (PCIe) or any other improvement or derivative protocol based on the PCI protocol discussed herein. The PCI-based protocol is a standard bus protocol for connecting devices such as local peripheral devices to a host device. The standard bus protocol is a data transmission protocol whose specifications have been defined and adopted by various manufacturers. The manufacturer ensures that the compatible device is compatible with the computing system implementing the bus protocol, and vice versa. As used herein, the PCI-based device also includes a device for communicating using fast non-volatile memory (NVMe). NVMe is a device interface specification for accessing a non-volatile storage medium connected to a computing system using PCIe. For example, the bus interface module may implement NVMe and the network device 824 may connect to the computing system using a PCIe interface.

[0154] A PCI-based device may include one or more functions. A "function" describes an operation that may be provided by the network device 824. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers, etc. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers to control two different storage resources. In some embodiments, a PCI-based device may have up to eight functions.

[0155] In some embodiments, the network device 824 may include single root I / O virtualization (SR-IOV). SR-IOV is an extended function that can be included in a PCI-based device. SR-IOV allows physical resources (e.g., a single network interface controller) to be displayed as multiple resources (e.g., sixty-four network interface controllers). Therefore, a PCI-based device (e.g., a network interface controller) that provides a certain function may appear to be multiple devices that provide the same function for a device using a PCI-based device. The functions of a storage adapter device that supports SR-IOV can be classified as physical functions (PF) or virtual functions (VF). A physical function refers to a function that can be discovered, managed, and manipulated in a device. A physical function has configuration resources that can be used to configure or control a storage adapter device. A physical function contains the same configuration address space and memory address space as a non-virtualized device. A physical function may have multiple virtual functions associated with it. A virtual function is similar to a physical function, but is a lightweight function that may generally lack configuration resources and is generally controlled by the configuration of its underlying physical function. Each of a physical function and / or a virtual function may be assigned to a corresponding execution thread (e.g., a virtual machine) running on a host device.

[0156] Embodiments of the present disclosure may be described in terms of the following:

[0157] Clause 1. A method for transposing a tensor using a processing element array operation, the method comprising:

[0158] mapping a block of data elements of the tensor to a plurality of row partitions of a state buffer of the integrated circuit, each row partition having a plurality of columns;

[0159] loading the data elements into corresponding processing element rows and columns in an array of processing elements of the integrated circuit;

[0160] performing a series of multiplication operations using an identity matrix such that each column of the identity matrix is ​​sequentially multiplied with each column of data elements in the array of processing elements;

[0161] summing the products of multiplying each column of the processing element array by a column of the identity matrix after each multiplication operation;

[0162] storing the summed products for each column of the processing element array performing the multiplication operation in a corresponding column partition of a result buffer having the same number of column partitions as the number of columns in the processing element array, wherein the summed products for subsequent multiplication operations are stored in subsequent rows of each corresponding column partition, and wherein the result buffer has the same number of rows as the number of rows of the processing element array; and

[0163] The summed products in the column partitions stored in the result buffer are loaded into a corresponding number of row partitions in the state buffer.

[0164] Clause 2. The method of clause 13, further comprising decomposing the tensor into a plurality of blocks of data elements, each block of data elements having a size corresponding to a size of the state buffer.

[0165] Clause 3. The method of any preceding clause, wherein each summed product stored in the result buffer corresponds to one of the data elements stored in the processing element array.

[0166] Clause 4. A method according to any of the preceding clauses, wherein when the block of data elements of the tensor is loaded into the processing element array, the sum of the products in the column partition of the result buffer reproduces the block of data elements in the same row position and the same column position in the result buffer.

[0167] Clause 5. A computer-implemented method comprising:

[0168] receiving, by the compiler, a set of operations to be performed by the execution engine;

[0169] Determining, by the compiler, that the set of operations includes performing a matrix transpose operation on a tensor;

[0170] converting, by the compiler, the matrix transposition operation into an identity matrix multiplication operation; and

[0171] A set of instructions is generated by the compiler, the set of instructions being operable to transpose the tensor by: loading data elements of the tensor from a buffer memory of the execution engine into a systolic array of the execution engine; performing an identity multiplication on the data elements in the systolic array; storing a result of the identity multiplication in a result buffer of the execution engine; and loading the result from the result buffer into the buffer memory.

[0172] Clause 6. The computer-implemented method of Clause 5, wherein loading the data element of the tensor comprises loading the data element into the systolic array over a first data bus in response to a first instruction in the set of instructions.

[0173] Clause 7. A computer-implemented method as described in Clause 5 or 6, wherein performing the identity multiplication includes propagating columns of the identity matrix from the buffer memory to the systolic array one column at a time through a second data bus during each processing cycle of the systolic array by a second instruction in the set of instructions.

[0174] Clause 8. The computer-implemented method of any of Clauses 5 to 7, wherein the result buffer comprises a set of column partitions, and each column partition in the result buffer corresponds to a column of processing elements in the systolic array.

[0175] Clause 9. A computer-implemented method according to any one of clauses 5 to 8, wherein:

[0176] The buffer memory includes a set of row partitions, and

[0177] A first instruction in the set of instructions maps each column partition in the result buffer to a row partition of the buffer memory.

[0178] Clause 10. A computer-implemented method according to any one of clauses 5 to 9, wherein the identity multiplication comprises multiplying the data elements of the tensor stored in each column of a set of processing element columns of the systolic array by each column of an identity matrix, and

[0179] Wherein multiplication of a column of data elements of the tensor with a column of the identity matrix generates at most one non-zero value.

[0180] Clause 11. The computer-implemented method of any of Clauses 5 to 10, wherein storing the result of the identity multiplication in the result buffer comprises summing the products from each column multiplication performed by the systolic array, and

[0181] wherein the value of each summed product is equal to the data element of the tensor corresponding to the non-zero element in the column of the identity matrix.

[0182] Clause 12. A computer-implemented method according to any one of clauses 5 to 11, wherein when the data elements of the tensor are loaded into the execution engine, the result of the identity multiplication is stored in the result buffer and the data elements are reproduced in the same row position and the same column position in the result buffer.

[0183] Clause 13. The computer-implemented method of any one of clauses 5 to 12, further comprising:

[0184] Prior to transposing the tensor, decomposing the tensor into blocks of data elements having a dimension of the tensor not exceeding a dimension of the buffer memory by the set of instructions generated by the compiler; and

[0185] Blocks of data elements are stored in the buffer memory.

[0186] Clause 14. The computer-implemented method of any one of clauses 5 to 13, wherein the computer-implemented method further comprises:

[0187] decomposing, by the set of instructions generated by the compiler, the block of data elements of the tensor stored in the buffer memory into sub-blocks of data elements having a dimension not exceeding a dimension of the systolic array; and

[0188] Sub-blocks of data elements are loaded into the systolic array.

[0189] Clause 15. A computer-implemented method according to any one of clauses 5 to 14, wherein:

[0190] The buffer memory comprises a set of rows,

[0191] Each row of the buffer memory comprises a partition,

[0192] mapping the data elements in the sub-block of the tensor to the partitions in the buffer memory, and

[0193] The data elements stored in each partition in the buffer memory are loaded into corresponding rows in the systolic array.

[0194] Clause 16. A computer-implemented method as described in any of Clauses 5 to 15, wherein the data elements of the tensor are intermediate results of calculations performed by the execution engine.

[0195] Clause 17. A non-transitory computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to execute a compiler that performs operations comprising:

[0196] receiving a set of operations to be performed by an execution engine;

[0197] determining that the set of operations includes performing a matrix transpose operation on the tensor;

[0198] converting the matrix transposition operation into an identity matrix multiplication operation; and

[0199] A set of instructions is generated, the set of instructions being operable to transpose the tensor by: loading data elements of the tensor from a buffer memory of the execution engine into a systolic array of the execution engine; performing an identity multiplication on the data elements in the systolic array; storing a result of the identity multiplication in a result buffer of the execution engine; and loading the result from the result buffer into the buffer memory.

[0200] Clause 18. The non-transitory computer-readable medium of Clause 17, wherein the result buffer comprises a set of column partitions, and each column partition in the result buffer corresponds to a column of processing elements in the systolic array;

[0201] The buffer memory includes a set of row partitions, and

[0202] A first instruction in the set of instructions maps each column partition in the result buffer to a row partition of the buffer memory.

[0203] Clause 19. The non-transitory computer-readable medium of clause 17 or 18, further comprising instructions that, when executed by the one or more processors, cause the compiler to perform operations comprising:

[0204] Before transposing the tensor, decomposing the tensor into blocks of data elements whose dimensions do not exceed the dimensions of the buffer memory; and

[0205] Blocks of data elements are stored in the buffer memory.

[0206] Clause 20. The non-transitory computer-readable medium of any one of clauses 17 to 19, further comprising instructions that, when executed by the one or more processors, cause the compiler to perform operations comprising:

[0207] decomposing the block of data elements of the tensor stored in the buffer memory into sub-blocks of data elements having a dimension not exceeding a dimension of the systolic array; and

[0208] Sub-blocks of data elements are loaded into the systolic array.

[0209] Clause 21. An integrated circuit device comprising:

[0210] Status buffer;

[0211] an array of processing elements coupled to the status buffer; and

[0212] a result buffer coupled to the array of processing elements,

[0213] wherein the integrated circuit device is configured to:

[0214] receiving a block of data elements of a tensor into a plurality of row partitions of the state buffer of the integrated circuit, each row partition having a plurality of columns;

[0215] loading the data elements into corresponding processing element rows and columns in the array of processing elements of the integrated circuit;

[0216] performing a series of multiplication operations using an identity matrix such that each column of the identity matrix is ​​sequentially multiplied with each column of data elements in the array of processing elements;

[0217] After each multiplication operation, summing the products of the multiplication operations performed on each column of the processing element array and a column of the identity matrix;

[0218] storing the summed products for each column of the processing element array performing the multiplication operation in a corresponding column partition of a result buffer having the same number of column partitions as the number of columns in the processing element array, wherein the summed products for subsequent multiplication operations are stored in subsequent rows of each corresponding column partition, and wherein the result buffer has the same number of rows as the number of rows of the processing element array; and

[0219] The summed products in the column partitions stored in the result buffer are loaded into a corresponding number of row partitions in the state buffer.

[0220] Clause 22. The integrated circuit device of Clause 21, wherein each summed product stored in the result buffer corresponds to one of the data elements stored in the processing element array.

[0221] Clause 23. An integrated circuit device according to clause 21 or 22, wherein when the block of data elements of the tensor is loaded into the processing element array, the sum of the products in the column partition of the result buffer reproduces the block of data elements in the same row position and the same column position in the result buffer.

[0222] The modules described herein may be software modules, hardware modules, or a suitable combination thereof. If the module is a software module, the module may be embodied on a non-transitory computer-readable medium and processed by a processor of any of the computer systems described herein. It should be noted that the described processes and architectures may be executed in real time or in an asynchronous mode prior to any user interaction. The modules may be configured in the manner suggested in the aforementioned figures, and / or the functions described herein may be provided by one or more modules that exist as separate modules, and / or the module functions described herein may be distributed over multiple modules.

[0223] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

[0224] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated examples thereof are shown in the accompanying drawings and have been described in detail above. However, it should be understood that it is not intended to limit the present disclosure to the specific form or forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions and equivalents that fall within the spirit and scope of the present disclosure as defined by the appended claims.

[0225] Unless otherwise indicated herein or clearly contradicted by context, the terms "a / an" and "the" and similar referents used in the context of describing the disclosed examples (especially in the context of the following claims) should be interpreted as covering both the singular and the plural. The terms "comprising", "having", "including" and "containing" should be interpreted as open-ended terms (i.e., meaning "including but not limited to") unless otherwise indicated herein or clearly contradicted by context. The term "connected" should be interpreted as partially or completely contained within, attached to, or joined together, even in the presence of intermediates. Unless otherwise indicated herein, the recitation of ranges of values ​​herein is intended to serve only as a shorthand method of referring individually to each individual value falling within the range, and each individual value is incorporated into the specification as if the individual value were recited individually herein. Unless otherwise indicated herein or clearly contradicted by context, all methods described herein can be performed in any suitable order. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate examples of the present disclosure and is not intended to limit the scope of the present disclosure. No language in this specification should be construed as indicating any unclaimed element as essential to the practice of the present disclosure.

[0226] Unless expressly stated otherwise, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood in the context of being generally used to express that an item, term, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain instances require that at least one of X, at least one of Y, or at least one of Z each be present.

[0227] Various examples of the present disclosure are described herein, including the best modes known to the inventors for implementing the present disclosure. After reading the above description, variations of these examples will be apparent to those of ordinary skill in the art. The inventors anticipate that the technicians may adopt such variations when appropriate, and the inventors intend to practice the present disclosure in other ways different from the ways specifically described herein. Therefore, where permitted by applicable law, the present disclosure includes all modifications and equivalents of the subject matter recited in the attached claims. In addition, unless otherwise indicated herein or clearly contradictory to the context, the present disclosure encompasses any combination of the above elements in all possible variations thereof.

Claims

1. A computer-implemented method comprising: receiving, by the compiler, a set of operations to be performed by the execution engine; Determining, by the compiler, that the set of operations includes performing a matrix transpose operation on a tensor; converting, by the compiler, the matrix transposition operation into an identity matrix multiplication operation; and A set of instructions is generated by the compiler, the set of instructions being operable to transpose the tensor by: loading data elements of the tensor from a buffer memory of the execution engine into a systolic array of the execution engine; performing identity multiplication on the data elements in the systolic array; storing a result of the identity multiplication in a result buffer of the execution engine; and loading the result from the result buffer into the buffer memory, wherein the identity multiplication comprises multiplying the data elements of the tensor stored in each column of a set of processing element columns of the systolic array by each column of an identity matrix, and Wherein multiplication of a column of data elements of the tensor with a column of the identity matrix generates at most one non-zero value.

2. The computer-implemented method of claim 1 , wherein loading the data element of the tensor comprises loading the data element into the systolic array over a first data bus in response to a first instruction in the set of instructions.

3. The computer-implemented method of claim 1 , wherein performing the identity multiplication comprises propagating columns of an identity matrix from the buffer memory to the systolic array one column at a time via a second data bus during each processing cycle of the systolic array by a second instruction in the set of instructions.

4. The computer-implemented method of claim 1 or 2, wherein the result buffer comprises a set of column partitions, and each column partition in the result buffer corresponds to a column of processing elements in the systolic array.

5. The computer-implemented method of claim 1 or 2, wherein: The buffer memory includes a set of row partitions, and A first instruction in the set of instructions maps each column partition in the result buffer to a row partition of the buffer memory.

6. The computer-implemented method of claim 1 or 2, wherein said storing the result of said identity multiplication in said result buffer comprises summing products from each column multiplication performed by said systolic array, and wherein the value of each summed product is equal to the data element of the tensor corresponding to the non-zero element in the column of the identity matrix.

7. The computer-implemented method of claim 1 , wherein when the data element of the tensor is loaded into the execution engine, the result of the identity multiplication is stored in the result buffer, the data element is reproduced in the same row position and the same column position in the result buffer.

8. The computer-implemented method of claim 1 or 2, further comprising: Prior to transposing the tensor, decomposing the tensor by the set of instructions generated by the compiler into blocks of data elements whose dimensions do not exceed the dimensions of the buffer memory; and Blocks of data elements are stored in the buffer memory.

9. The computer-implemented method of claim 1 or 2, wherein the computer-implemented method further comprises: decomposing, by the set of instructions generated by the compiler, the block of data elements of the tensor stored in the buffer memory into sub-blocks of data elements having a dimension not exceeding a dimension of the systolic array; and Sub-blocks of data elements are loaded into the systolic array.

10. The computer-implemented method of claim 9, wherein: The buffer memory comprises a set of rows, Each row of the buffer memory comprises a partition, mapping the data elements in the sub-block of the tensor to the partitions in the buffer memory, and The data elements stored in each partition in the buffer memory are loaded into corresponding rows in the systolic array.

11. The computer-implemented method of claim 1 or 2, wherein the data elements of the tensor are intermediate results of computations performed by the execution engine.

12. An integrated circuit device comprising: Status buffer; an array of processing elements coupled to the state buffer; as well as a result buffer coupled to the array of processing elements, wherein the integrated circuit device is configured to: receiving a block of data elements of a tensor into a plurality of row partitions of the state buffer of the integrated circuit, each row partition having a plurality of columns; loading the data elements into corresponding processing element rows and columns in the array of processing elements of the integrated circuit; performing a series of multiplication operations using an identity matrix such that each column of the identity matrix is ​​sequentially multiplied with each column of data elements in the array of processing elements; After each multiplication operation, summing the products of the multiplication operations performed on each column of the processing element array and a column of the identity matrix; storing the summed products for each column of the processing element array performing the multiplication operation in a corresponding column partition of a result buffer having the same number of column partitions as the number of columns in the processing element array, wherein the summed products for subsequent multiplication operations are stored in subsequent rows of each corresponding column partition, and wherein the result buffer has the same number of rows as the number of rows of the processing element array; and The summed products in the column partitions stored in the result buffer are loaded into a corresponding number of row partitions in the state buffer.

13. The integrated circuit device of claim 12, wherein each summed product stored in the result buffer corresponds to one of the data elements stored in the processing element array.

14. An integrated circuit device according to claim 12 or 13, wherein when the block of data elements of the tensor is loaded into the processing element array, the sum of products in the column partition of the result buffer reproduces the block of data elements in the same row position and the same column position in the result buffer.

Images