Neural network instruction set architecture

Abstract

The invention relates to a neural network instruction set architecture. A computer-implemented method includes receiving, by a processing unit, an instruction specifying a data value for performing a tensor calculation. In response to receiving an instruction, the method may include performing, by a processing unit, a tensor calculation by executing a loop nest comprising a plurality of loops, where a structure of the loop nest is defined based on one or more of data values of the instruction. The tensor calculation can be at least part of the calculation of the neural network layer. The data values specified by the instructions can include at least a value specifying a type of neural network layer, and the structure of the loop nest can be defined at least in part by the type of neural network layer.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201710909908.8, filed on September 29, 2017. Technical Field

[0003] This specification relates to a set of instructions for computation in deep neural networks (“DNNs”). Background Technology

[0004] Neural networks are machine learning models that use one or more layers to generate outputs from received inputs, such as for classification. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network—that is, the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter array.

[0005] Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has a set of associated kernels. Each kernel includes values ​​established by a neural network model created by the user. In some implementations, kernels identify specific image contours, shapes, or colors. Kernels can be represented as a matrix structure of weight inputs. Each convolutional layer can also process a set of activation inputs. This set of activation inputs can also be represented as a matrix structure. Summary of the Invention

[0006] One inventive approach to the subject matter disclosed in this specification can be embodied in a computer-implemented method. The method includes receiving, by a processing unit, instructions specifying data values ​​for performing tensor computations. The method may include, in response to receiving the instructions, the processing unit performing the tensor computations by executing nested loops comprising multiple loops, wherein the structure of the nested loops is defined based on one or more of the data values ​​of the instructions. These and other implementations may each selectively include one or more of the following features. For example, the tensor computation may be at least part of the computation of a neural network layer. The data values ​​specified by the instructions include at least one value specifying the type of the neural network layer, and wherein the structure of the nested loops is defined at least in part by the type of the neural network layer. Thus, executing nested loops comprising multiple loops may refer to traversing the elements of a tensor in an order specified by the structure of the nested loops, wherein the structure is specified, for example, by the depth of the nested loops, and the start and end indices, stride, and direction of each loop.

[0007] In some implementations, tensor computation is at least part of the computation of a neural network layer. In some implementations, the data value specified by the instructions includes at least one value specifying the type of the neural network layer, and the nested loop structure is defined at least in part by the type of the neural network layer. In some implementations, the instructions cause a processing unit to access at least one element of a dimension of the tensor, said element being part of at least one index used during the performance of tensor computation and when performing nested loops. In some implementations, the instructions cause a processing unit to access at least one memory address of an array in a storage medium, said memory address including variables read by the processing unit during the performance of tensor computation.

[0008] In some embodiments, performing tensor computation includes providing at least one control signal to a tensor traversal unit (TTU) via a processing unit, causing the TTU to issue a loop index used during the tensor computation and when performing nested loops. In some embodiments, the method further includes providing at least one control signal to the TTU via the processing unit, causing an array reference of the TTU to generate an address for a referenced array element used during the tensor computation and when performing nested loops. In some embodiments, instructions instruct a first TTU counter to be added to a second TTU counter to generate an address for the array reference associated with the TTU.

[0009] In some embodiments, performing tensor computation includes: executing a first synchronization procedure via a processing unit to manage one or more operands associated with performing tensor computation, wherein managing the operands includes halting one or more nested loops based on a synchronization flag condition. In some embodiments, performing tensor computation includes: executing a second synchronization procedure via a processing unit to manage a counter associated with incrementing and nested loop characteristics.

[0010] Another inventive approach to the subject matter described in this specification can be implemented in an electronic system comprising: a processing unit disposed in the electronic system, the processing unit including one or more processing means; and one or more machine-readable storage means for storing instructions executable by the one or more processing means to perform operations, the operations including: receiving by the processing unit an instruction specifying data values ​​for performing tensor computations; and, in response to receiving the instruction, performing tensor computations by the processing unit by executing nested loops comprising multiple loops, wherein the structure of the nested loops is defined based on one or more data values ​​of the instruction.

[0011] Another inventive approach to the subject matter described in this specification can be implemented in a non-transitory computer-readable storage medium comprising executable instructions by a processing unit that, when executed, cause the processing unit to perform operations including: receiving instructions by the processing unit specifying data values ​​for performing tensor computations; and, in response to the received instructions, performing tensor computations by the processing unit by executing nested loops comprising multiple loops, wherein the structure of the nested loops is defined based on one or more of the data values ​​of the instructions.

[0012] Other implementations of this and other solutions include systems, devices, and computer programs configured to perform actions of methods encoded on computer storage devices. Systems of one or more computers can be configured in this way, by means of software, firmware, hardware, or a combination thereof installed on the system, which, during operation, cause the system to perform the actions. One or more computer programs can be configured in this way by means of instructions having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0013] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The processing unit can initiate a program with a single instruction to repeat deeply nested loops. The processing unit can execute received instructions encoded with data values ​​for performing tensor computations. By reducing the number of instructions that the processor needs to execute when traversing tensors to perform one or more computations, the computational bandwidth of the processing unit is increased. Instructions for performing tensor computations for a given neural network layer can be encoded and distributed across one or more computing systems in the example hardware computing system. Distributing encoded instructions across various computing systems allows for increased computational bandwidth within a single system. Because a single system is only responsible for a subset of the total computations required for a given tensor, the number of instructions implemented in the computing system is reduced.

[0014] Details of one or more embodiments of the subject matter described in this specification are given in the following drawings and description. Other potential features, solutions, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0015] Figure 1 This is a block diagram of an example computing system.

[0016] Figure 2 Example tensor traversal cell is shown.

[0017] Figure 3 The diagram shows the operation code and at least partially the method. Figure 1 Example tables show the corresponding operations performed by the example computing system.

[0018] Figure 4AExample activation tensor, example weight tensor, and example output tensor are shown.

[0019] Figure 4B It shows that it can be passed Figure 1 The example of nested loops is executed by the processing unit of the example computing system.

[0020] Figure 5 This is a flowchart illustrating an example of performing tensor computation using a computer-implemented method.

[0021] The same reference numerals and symbols in the various figures indicate the same elements. Detailed Implementation

[0022] The subject matter described in this specification relates to a computer implementation of a process or method to accelerate machine learning inference workloads, enabling the processing unit performing the workload to achieve high performance when performing computations. The processes and methods described in this specification can be executed within a hardware computing system organized as a collection of tiles, each of which is independent and capable of independently performing at least a portion of the computations required by neural network layers.

[0023] The computational processing performed within a neural network can consist of multiplying the input tensor with parameter / weight tensors over one or more cycles, and accumulating the products over many cycles. The input weight multiplication can then be written as the sum of the products of each weight element multiplied by a row in the input tensor or matrix. Deep neural networks (DNNs) consist of multiple layers that feed into each other. The methods described in this specification accelerate inference workloads involving tensor computations for a given layer. The topics described in this specification can also relate to image recognition and / or classification methods / systems using the disclosed techniques, as well as hardware using neural network instruction set architectures.

[0024] Typically, when software algorithms perform tensor computations by processing N-dimensional tensors, nested loops can be used, where each loop iterates through each dimension of the N-dimensional tensor. A multidimensional tensor can be a matrix or a multidimensional matrix. Each dimension of an N-dimensional tensor can include one or more elements, where each element can store a corresponding data value. For example, a tensor can be a variable in a program, where the variable can have three dimensions. The first dimension could have a length of three hundred elements, the second dimension could have a length of thousands of elements, and the third dimension could have a length of twenty elements.

[0025] Traversing a tensor within a nested loop requires calculating the memory address of each element to load or store its corresponding data value. For example, a for-loop is a nested loop where three loops, tracked by three loop index variables, can be nested to traverse a three-dimensional tensor. In some cases, the processor may need to execute a loop boundary condition, such as setting the loop boundary of the inner loop using the outer loop index variable. For instance, when determining whether to exit the innermost loop of a nested loop, the program can compare the current value of the innermost loop's loop index variable with the current value of the outermost loop's loop index variable.

[0026] Computational tasks associated with traversing a three-dimensional tensor can require a large number of instructions, such as branch instructions and integer arithmetic instructions. When each loop boundary is small and the number of loops is large, memory address calculations and tensor dimension traversal computations can account for a large portion of the total execution time, severely degrading overall performance. As described in more detail below, when performing one or more computations by traversing a tensor, a method executable by a processing unit that receives instructions encoded with data values ​​used to perform tensor computations can increase the computational bandwidth of the processing unit. In some implementations, computational bandwidth refers, for example, to the maximum speed at which operations can be performed on a tensor or the number of operations that can be performed on a tensor.

[0027] Instructions for performing tensor computations on a given layer can be encoded and distributed across one or more computing modules (hereinafter described as computing modules) within the example hardware computing system. Distributing encoded instructions to different computing modules allows for increased computational bandwidth within a single module. Furthermore, by distributing computations across multiple computing modules, the processing units of the computing modules consume a reduced number of instructions (compared to current systems) when traversing tensors. Therefore, consuming instructions involves, for example, executing one or more instructions through the processing units of the computing modules. Because a single module is only responsible for a subset of the total computations required for a given tensor, the number of instructions within the computing module can be reduced.

[0028] Additional details and descriptions relating to a hardware computing system for accelerating tensor computations in neural networks are described in U.S. Patent Application No. 15 / 335,769, entitled "Neural Network Compute Tile," including dot products and matrix multiplications for neural network inference workloads. The entire disclosure of U.S. Patent Application No. 15 / 335,769 is incorporated herein by reference.

[0029] Figure 1A block diagram of an example computing system 100 is shown, which is used to traverse one or more tensors to perform computations for a neural network layer. As shown, the computing system 100 includes a processing unit 102, a storage medium 104, a tensor traversal unit (TTU) 106, a multiply-accumulate (MAC) arithmetic unit 108, and an activation unit 110. Although a single TTU is shown, the system 100 can include two or more TTUs 106 to handle tensor traversal operations and memory access operations when using one or more computing modules to process tensor computations for a given neural network layer. Similarly, although a single MAC is shown, the system 100 can include multiple MAC arithmetic units 108.

[0030] Processing unit 102 is configured to process instructions for execution within computing system 100, including instructions or program code stored in storage medium 104 or other instructions / code stored in another storage device. Processing unit 102 may include one or more processors. Storage medium 104 may include one or more repositories or units, including a first repository 112 for storing activations and a second repository 114 for storing weights. In some embodiments, storage medium 104 is a volatile memory unit. In some other embodiments, storage medium 104 is a non-volatile memory unit or a unit such as, for example, read-only memory (ROM) and / or electrically erasable programmable read-only memory (EEPROM). Storage medium 104 may also be another form of computer-readable medium that includes devices in a storage area network or other configuration, such as a floppy disk device, hard disk device, optical disk device or magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices.

[0031] When the processing unit 102 executes the instruction set 103, the instruction set 103 causes the processing unit 102 to perform one or more tasks, such as storing activations at memory address locations in the first library 112 and storing weights at memory address locations in the second library 114. Furthermore, the instruction 103 also enables the processing unit 102 to access stored data from the first library 112 and the second library 114 and provide the accessed data to the MAC arithmetic unit 108. As described in more detail below, the MAC arithmetic unit 108 is capable of performing multiplication operations, which may include multiplying activations by weights to produce a partial sum or to produce output activations (multiples) provided to the activation unit 110.

[0032] Typically, computing system 100 processes one or more instructions included in instruction set 103 to generate output 116. Instruction set 103 may include at least one of tensorop instructions or direct memory access operation (DMAop) instructions. Computing system 100 may be configured to execute at least one tensorop and at least one DMAop instruction to perform tensor computations including multiple linear algebra computations.

[0033] In some implementations, performing tensor computation includes executing a DMAop instruction to access memory address locations corresponding to elements of a specific dimension of the tensor, wherein at least one memory address location stores data values ​​for the elements of the tensor. Tensor computation can also include executing tensorop to traverse elements along a specific dimension of the tensor. In some implementations, performing tensor computation corresponds to multiplying the data values ​​of elements activating the tensor with the data values ​​of elements for parameter or weight tensors to produce the output values ​​of a neural network layer.

[0034] As shown in the figure, TTU 106 can include instruction buffer 124. Although a single instruction buffer 124 is shown, computing system 100 can include multiple instruction buffers 124. Furthermore, system 100 can include multiple TTUs 106, and each TTU 106 can include one or more instruction buffers 124. In some embodiments, the different instruction buffers 124 within system 100 can include tensorop instruction buffers and DMAop instruction buffers. As described in more detail below, instructions received by system 100 and executed by processing unit 102 can include one of tensorop instructions and DMAop instructions. In some embodiments, the received instructions are stored in instruction buffer 124 before execution by processing unit 102.

[0035] As described in more detail below with reference to FIG4, activations can be preloaded into the first library 112 at the corresponding memory address locations corresponding to the elements of the input tensor. Similarly, weights can be preloaded into the second library 114 at the corresponding memory address locations corresponding to the elements of the weight tensor. In some embodiments, instructions, activations, and weights are provided to the system 100 from an external control device or a higher-level control device associated with the neural network hardware computing system.

[0036] Typically, when processing unit 102 executes one or more instructions for accessing specific elements of a tensor, tensor traversal unit 106 determines the memory address of the element, enabling processing unit 102 to access storage medium 104 to read data representing the value of the specific element. For example, the instructions stored in storage medium 104 may include program code for a nested loop program that can be executed by processing unit 102.

[0037] Nested loop programs can include multiple variables that can have a range of values ​​determined based on data values ​​provided by instruction set 103. Processing unit 102 can execute program code for the nested loop program to enable TTU 106 to access memory address locations corresponding to corresponding elements of at least two multidimensional arrays. For example, the multiple variables can include data values ​​used to exemplify an example deep loop nesting containing multiple inner loops and multiple outer loops. In some implementations, the multiple variables and data values ​​enable TTU 106 to exemplify deep loop nesting, as shown below. Figure 4B The description of nested loops (e.g., deep nested loop 402).

[0038] Based on the current index variable value associated with the nested loop, TTU 106 can determine an offset value representing the offset from the first element of the multidimensional array variable. Processing unit 102 can then use this offset value to access a specific element of the two-dimensional array variable from the storage medium.

[0039] Tensor traversal unit 106 is configured to determine states associated with one or more tensors. These states can include loop boundary values, current loop index variable values, dimension multipliers used to calculate memory address values, and / or program counter values ​​used to process branch loop boundaries. Tensor traversal unit 106 includes tensor state elements 122 and arithmetic logic units. Each of the tensor state elements 122 can be a storage element, such as a register or any other suitable storage circuit. In some embodiments, the tensor state elements 122 can be arranged into different groups, either physically or logically.

[0040] In some implementations, the groups of tensor state elements 122 can be arranged into a multidimensional array, either physically or logically. For example, each group in tensor state elements 122 can be arranged into a two-dimensional array, either physically or logically. The arithmetic logic unit includes one or more arithmetic logic units (ALUs) configured to perform arithmetic and logical operations. For example, an arithmetic logic unit may include digital circuitry configured to perform addition, subtraction, multiplication, division, logical sum, logical OR, and any other suitable arithmetic and logical operations.

[0041] Figure 2An example set of tensor state elements 200 of a tensor traversal unit is shown. Tensor state elements 200 may correspond to tensor state elements 122 of tensor traversal unit 106. Tensor traversal unit 200 includes groups of tensor index elements 202, groups of tensor boundary elements 204, and groups of dimension multiplier elements 206. The tensor index elements 202 can be arranged physically or logically as a two-dimensional array with M rows and N columns, where M and N are integers greater than or equal to 1. In some embodiments, each row of tensor index elements 202 may represent tensor index information for the tensor. For example, if a program defines two array variables V1 and V2, the tensor traversal unit may assign rows 202a and 202b to track the tensor index information for array variables V1 and V2, respectively. In some embodiments, each column of tensor index elements 202 may represent tensor index information for the values ​​of nested loop index variables associated with the tensor. For example, if a program defines nested loops with three loops for accessing variable V1, where each loop in the nested loop is indexed by the nested loop index variables i, j, and k, then the tensor traversal unit can assign the tensor index element X. 1,1 X 1,2 and X 1,3 Track the nested loop index variables i, j, and k respectively.

[0042] Tensor boundary elements 204 can be arranged as a two-dimensional array with the same dimensions as tensor index elements 202, either physically or logically, where each element of tensor boundary elements 204 has a corresponding element in tensor index elements 202. In some implementations, each row of tensor boundary elements 204 can represent tensor boundary information for the tensor. For example, if the program defines two array variables V1 and V2, then the tensor traversal unit can allocate rows 204a and 204b to track the tensor boundary information of array variables V1 and V2, respectively.

[0043] The dimension multiplier elements 206 can be arranged, either physically or logically, into a two-dimensional array having the same dimensions as the tensor index elements 202, wherein each element of the dimension multiplier elements 206 has a corresponding element in the tensor index elements 202. In some implementations, each row of the dimension multiplier elements 206 can represent dimension multiplier information for the tensor. For example, if the program defines two array variables V1 and V2, then the tensor traversal unit can allocate rows 206a and 206b to track the dimension multiplier information of array variables V1 and V2, respectively.

[0044] In some implementations, instead of using multipliers to multiply tensor index values ​​with dimension multipliers, alternative embodiments can include a TTU 106 that utilizes an adder function instead of a multiplier function. In these embodiments, processing unit 102 can execute program code to provide the TTU 106 with the numbers that have been multiplied (e.g., the start, end, and step values ​​for a particular loop).

[0045] Deep neural networks (DNNs) comprise multiple layers that feed into each other. In some implementations, an example neural network can include one or more of the following layer types: convolutional layers (e.g., in the depth direction), reduced layers, fully connected (FC) layers, classification layers, element-wise multiplication layers, or pooling layers (e.g., averaging and maximizing). In some implementations, computations with small (e.g., one or two) input and output depths can be associated with layers having shallow kernel properties, which can be properties of existing layers such as fully connected or convolutional layers. Typically, tensor computations occurring within system 100 for a first layer (e.g., a convolutional layer) can produce an output activation, which is then provided as the input activation of a second layer (e.g., a pooling layer) different from the first layer.

[0046] In some implementations, two or more computing systems 100 can be configured as a collection of computing modules. In these implementations, each computing module independently performs the portion of the computation required to process the inference workload for a given layer. The output of the computation is then moved to another module that needs the output to perform subsequent computations for the given layer. Furthermore, neural network layers can be partitioned in a manner desired by the system designer, and each partition can be mapped to a module such that a portion of the computation corresponds to a specific layer partition.

[0047] Refer to Figure 1 In some implementations, system 100 receives via instruction set 103 at least one instruction encoding a plurality of operations to be performed by system 100, to at least perform a portion of the tensor computation required for a given neural network layer. The instructions and encoded operations can specify the type of layer, the structure of the activation tensor, and the structure of the weight tensor. Furthermore, the encoded operations can cause processing unit 102 to execute program code to create a nested loop structure at least partially defined by the layer type of the neural network. Regarding the encoded operations, instruction set 103 can include at least one of tensorop instructions or direct memory access operation (DMAop) instructions. Instruction set 103 can also include one or more other instructions encoding various other operations, such as synchronization operations, obstacle operations, and fence operations (described in more detail below).

[0048] Figure 3Table 300 is shown, including opcodes and corresponding operations performed at least partially by system 100. Table 300 includes operations 302, 304, and 308. As shown by table 300, system 100 can perform multiple operations encoded by a single instruction, for example, provided by instruction set 103. Typically, instruction set 103 can include a standardized structure using opcodes to define the types of operations to be performed within system 100 for a particular computation sequence. A unique opcode can correspond to a specific operation type. In some embodiments, encoded operations in the form of one or more instructions, input activations, and weights are provided to system 100 from external input / output (I / O) devices or higher-level controller devices associated with the neural network hardware computing system. Instructions, activations, and weights can be transferred from example I / O devices or between multiple computing modules (e.g., multiple systems 100) of the hardware computing system using one or more data buses.

[0049] As shown in Table 300, operation 302 is an encoded tensor operation that can include one or more tensors with corresponding opcodes. In some implementations, a single instruction or set of instructions can include multiple sub-instructions that define encoding in the form of opcodes indicating a specific tensorop. For example, sub-instructions can be distributed to one or more computation modules to perform tensorops on computations corresponding to specific neural network layers. For example, tensorop sub-instructions can include opcodes for at least one of the aforementioned layers: convolution, fully connected (FC), pooling, shrinking, or element-wise multiplication. Tensorop sub-instructions can be configured such that all tensor computations occurring within the computation module / system 100 have the desired encoding from the tensorop sub-instructions.

[0050] Typically, tensor computations occurring within system 100 will have the required encoding specified by instructions received from instruction set 103. When multiple systems 100 are arranged as a collection of computation modules, tensorop sub-instructions and encodings allow different computation modules to execute different instructions. In some implementations, different sub-instructions can correspond to tensor computations for a specific partition of a neural network layer. Tensorop encoding also allows TTU 106 to issue a loop index to processing unit 102 via a main TTU loop nesting region indicating the desired TTU tensor operation. Tensorop encoding also enables TTU 106 to generate memory addresses for specific array / tensor elements based on a TTU loop mask indicating which counters are added together to compute the address. Tensorop encoding also provides multiple TTU loop nesting instructions executable by processing unit 102 for traversing the input activation tensor, weight tensor, and output activation tensor.

[0051] In some implementations, when two or more computing systems 100 are configured as a collection of computing modules, each computing module can be an independent unit that executes instructions independently. Typically, computing modules execute instructions that define two types of operations: tensorops and DMAops. In some implementations, independent control units within the computing system 100, whose data is synchronized via synchronization flags, can perform different operations. However, while the control units are independent, there may be dependencies that need to be maintained; for example, a tensorop does not necessarily execute a loop within a nested loop unless it has the data required by the nested loop in a memory location, for example, in storage medium 104. The required data can be placed in storage medium 104 via DMAops or other operations.

[0052] Typically, data synchronization within system 100 occurs via synchronization flag operations. In some implementations, system 100 will have a register file containing synchronization flag entries for each virtual write port. For example, TensorOp can have two or more tensor references, which can be partial sums (written to the second library 114) and activations (written to the first library 112). In this implementation, a synchronization flag register can exist for each partial sum and activation. Furthermore, during the execution of a given instruction set, processing unit 102 can select the dimension of the tensor reference that will trigger the incrementing of a particular synchronization flag register. Additionally, processing unit 102 can select the depth of the nested loops for waiting for the desired count on the selected synchronization flag register. This ensures that data dependencies are not violated. Furthermore, if the desired synchronization flag count on the selected synchronization flag register has not been reached, one or more threads will be stalled.

[0053] For example, TensorOp coding allows nested loops to be halted based on one or more synchronization flag conditions. In some implementations, TTU 106 can be halted due to: 1) a predetermined synchronization flag condition is not met (e.g., the synchronization flag count is not greater than a threshold count); or 2) a danger is detected at a specific stage of the instruction pipeline. Additionally, TensorOp coding allows the MAC operator 108 to be configured to receive specific data values ​​(e.g., activations or weights) of a desired bit width, and allows selection of the desired function to be performed by the activation unit 110.

[0054] The synchronization flag conditions defined by tensorop encoding enable the processing unit 102 of system 100 to perform at least one of a synchronization observer operation and a synchronization generator operation. During the execution of tensorop by processing unit 102, the synchronization observer operation can be used to indicate to processing unit 102 which loop iteration to synchronize, the synchronization flag count required before the loop iteration is allowed, and how much the expected synchronization flag count should be incremented for the next iteration.

[0055] As shown in Table 300, the encoded direct memory access operation instructions can include one or more DMAops. The DMA operation specified by instruction set 103 causes processing unit 102 to move data as needed to complete the execution of a specific tensorop through system 100. In some implementations, if data is moved between two memory resources of unequal width (e.g., a wide memory cell and a narrow memory cell), the data transfer size must be a multiple of the wider resource. For example, given a DMAop that moves data to / from wide memory (W-mem), the transfer size must be a multiple of the wide memory width. Similarly, for a DMAop that moves data to / from narrow memory (N-mem), the data transfer size must be a multiple of the narrow memory width.

[0056] Throughout this specification, the designations of "wide" and "narrow" used to describe the attributes of memory cells generally refer to an approximate size in terms of the width (bits / byte) of one or more memory cells. In some embodiments, the memory width of the first library 112 and the memory width of the second library 114 are not defined by a specific number of bits (such as, for example, 16 bits). Typically, two different types of memory exist in each embodiment of system 100, and narrow / wide is a relative term used to indicate the characteristics of the memory, and it is not absolute. In some example embodiments, the first library 112 includes one or more narrow memory cells, and the second library 114 includes one or more wide memory cells.

[0057] As described above, the instructions received by system 100 and executed by processing unit 102 can include tensorop instructions and DMAop instructions. The received instructions can be stored in instruction buffer 124 before execution by processing unit 102. As shown in Table 300, operation 304 is a DMA operation that can include instruction subtypes (e.g., instructions indicating "write to" buffer locations), including: 1) inbound instruction buffer; 2) outbound instruction buffer; 3) narrow-to-wide DMA instruction buffer; 4) wide-to-narrow DMA instruction buffer; and 5) ring bus DMA instruction buffer.

[0058] Referring now to Table 300, the DMAop.in operation is executed by processing unit 102, causing system 100 to receive input activation and other data from instruction set 103 (via example data bus) and store the activation in first library 112. As described in more detail below, the input activation is stored in first library 112 at the memory address corresponding to the corresponding element of the activation tensor. The DMAop.out operation is executed by processing unit 102, causing system 100 to read data (e.g., output activation) from first library 112 and write the data to example data bus to output data from system 100. Processing unit 102 can execute a DMAop.NW operation to move data from first library 112 (narrow memory) to second library 114 (wide memory). Similarly, processing unit 102 can execute a DMAop.WN operation to move data from second library 114 (wide memory) to first library 112 (narrow memory).

[0059] After retrieving data from instruction set 103 (via example data bus), processing unit 102 can perform a DMAop.R-bus operation to write data (e.g., weights / parameters) to second library 114. Execution of the DMAop.R-bus operation also enables processing unit 102 to read / access data (e.g., weights) stored in memory address locations of second library 114 and write the data to the example data bus, allowing data to be output from system 100. In some embodiments, the example data bus is a ring bus connecting multiple computing systems 100 in a ring configuration. In addition to connecting multiple systems 100 in a ring configuration, the ring bus can also connect multiple systems 100 to external I / O devices or higher-level control devices. Additional details and descriptions of a hardware computing system for accelerating neural network tensor computation are also described in U.S. Patent Application No. 15 / 335,769, filed October 27, 2016, entitled "Neural Network Compute Tile," which includes data bus options for connecting the computing module.

[0060] In some implementations, for a given set of computational modules, at least one of these modules can be a classifier module. The classifier block can be configured to apply a function to data values ​​associated with tensor computations already performed for a specific layer of the neural network. For example, an FC layer can be partitioned across modules, so that each module produces a subset of outputs, which are stored in a second library 114. Typically, linear outputs from memory address locations in the second library 114 can be moved from the first computational module to the second module (e.g., the classifier module) via a data ring bus using DMAop. The classifier module then applies at least one function (e.g., a softmax function) to each linear output and provides the post-function result to a higher-level control device or external I / O device.

[0061] Referring again to Table 300, in some embodiments, DMAop.infeed and DMAop.outfeed correspond to operations associated with both the external controller device and system 100. Typically, the DMAop.infeed operation is initiated at the external controller, which then provides both input activations and weights to an example data bus (e.g., a ring bus). The data bus can be configured to multicast an instruction to which compute module in a multi-module arrangement should consume the data stream of activations and weights (e.g., a bitmap). Therefore, the processing unit 102 of a particular receiving module can thus execute a portion of the DMAop.infeed operation to move the activations and weights / parameters from the external controller to the memory bank of the compute module (system 100). Similarly, in some embodiments, the processing unit 102 can then execute a portion of the DMAop.outfeed operation to move the results (outputs) of the example inference workload from the memory banks 112, 114 in the compute module to the classifier module, and from the classifier module to the external controller. Furthermore, the external controller can also execute a portion of the DMAop.outfeed operation to move the inference workload results to an example external I / O device.

[0062] Operation 306 involves the `tilefenceop` and `scalarfenceop` operations, executable by processing unit 102, to manage the issuance of new instructions while other operations are performed by system 100. Executing the `tilefenceop` instruction by processing unit 102 causes system 100 to block or prevent the issuance of any new instructions until all pending instructions in the corresponding instruction buffer of system 100 are released. Furthermore, while processing unit 102 executes pending instructions, the `scalarfenceop` instruction cooperates with the `tilefenceop` instruction to avoid the issuance of new instructions. For example, the `tilefenceop` instruction enables processing unit 102 to monitor the synchronization flag count metric associated with `tilefenceop`. Processing unit 102 can execute the `scalarfenceop` instruction to prevent the issuance of new instructions until a threshold `tilefenceop` count metric is reached.

[0063] Figure 4A Example input activation tensor 404, example weight tensor 406, and example output tensor 408 are shown. Figure 4B An example of a deeply nested loop 402 is shown that can be executed by processing unit 102 to perform tensor calculations related to dot product calculations or matrix multiplications. For example... Figure 4A As shown, the computation can include multiplying the activation tensor 404 with the parameter / weight tensor 406 over one or more computation cycles to produce an output / result in the form of an output tensor 408.

[0064] Each tensor 404, 406, 408 can include multiple data elements arranged in a row-column format across the x, y, and z-dimensional planes. Activation can be read from the corresponding memory address location in the first library 112 corresponding to the element of the activation tensor 404. Similarly, weights can be read from the corresponding memory address location in the second library 114 corresponding to the elements of the weight tensor. In some embodiments, the processing unit 102 can execute program code stored in certain memory cells (e.g., ROM / EEPROM memory cells) of the storage medium 104 for nested loop programs, causing the TTU 106 to access memory address locations corresponding to the corresponding elements of the activation tensor 404 and the weight tensor 406. The processing unit 102 can perform one or more tensor calculations by partially executing nested loop programs to executor multiple loops traversing tensor elements for a given dimension.

[0065] As TTU 106 traverses the activation tensor 404 and the weight tensor 406, it performs multiplication operations on individual activations from x-dimensional data elements in tensor 404, such as multiplying them individually by a row of weights (multiple data elements) in the kx-dimensional dimension of a particular weight tensor 406. These multiplication operations produce an output row corresponding to the first x-row in the zo-dimensional dimension of the output tensor 408. Typically, MAC unit 108 is capable of performing tensor computations including arithmetic operations related to dot product calculations. Input activations are provided one by one from the first library 112 for corresponding access by the individual MAC unit 108. MAC unit 108 multiplies the input activations by weights accessed from the second library 114 to produce individual output activation values ​​corresponding to the individual data elements of the output tensor 408.

[0066] exist Figure 4B In the example, a deep loop nesting 402 is shown and includes the MAC operation in the innermost loop (zo). The input size corresponding to the activation tensor 404 is represented as [x, y, zi], the filter size corresponding to the weight tensor 406 is represented as [x, y, zi, zo], and the output size corresponding to the output tensor 408 is represented as [x, y, zo]. Therefore, the deep loop nesting 402 specifies the output tensor 408 indicated by the output [x][y][zo] based on the computation across different dimensions of tensors 404 and 406. Typically, the deep loop nesting 402 provides a software algorithm that can be executed by system 100 to perform tensor computation by traversing N-dimensional tensors through nested loops. In an example computational process, each loop can be responsible for traversing a specific dimension of the corresponding N-dimensional tensor. Therefore, the deep loop nesting 402 provides expressions for one or more tensor computations that can be performed simultaneously within multiple computation modules.

[0067] In alternative implementations, other orderings of the deep loop nesting 402 that produce changes in the output tensor 408 are possible. System 100 is capable of receiving instructions, including a range of data values, to exemplify the deep loop nesting 402, allowing traversal of the activation tensor 404 and weight tensor 406 in any order. One or more loops can be spatially expanded in system 100. In some implementations, the xx and yy loop nesting can be expanded across one or more computation modules, allowing different computation modules to handle inference workloads associated with different layer partitions. In the example deep loop nesting 402, the activation read for each loop, and the zi... kx The ky loop reads a portion of the sum.

[0068] A nested loop structure (e.g., # for inner and outer loops) can be defined based on one or more of the data values ​​of instructions provided to system 100 via instruction set 103. Example computations occurring within a convolutional layer can be used as the basis for executing instructions describing the encoded operations to compute inference workloads for a given layer. Convolutional layer computations can include two-dimensional (2D) spatial convolutions of three-dimensional (3D) tensors. Computations for a convolutional layer can involve performing a two-dimensional spatial convolution between a three-dimensional activation tensor 404 and at least one three-dimensional filter (weight tensor 406).

[0069] For this calculation, there will be a sum of dot products along the depth dimension. Convolving a 3D filter 406 with the 3D input produces a 2D spatial plane. As shown, a 3D input tensor 404 can exist using multiple 3D filters that need to be convolved. The calculation will result in a 3D tensor output 408. Although the calculation for the convolutional layer has been described, it is also possible to generate one or more instructions encoded for computational operations in other neural network layers based on the teachings described in this specification.

[0070] Typically, when a single computation module comprises multiple MAC arithmetic units 108, the arithmetic units collectively provide Single Instruction Multiple Data (SIMD) functionality by sharing a single activation input to perform their respective computations. SIMD generally means that all parallel units (multiple MAC arithmetic units 108 in a single system 100) share the same instructions (based on deep loop nesting), but each MAC arithmetic unit 108 executes instructions for different data elements of tensors 404 and 406. In a basic example, in one cycle, adding arrays [1, 2, 3, 4] to [5, 6, 7, 8] element-wise to obtain array [6, 8, 10, 12] typically requires four arithmetic units to perform the operation on each element. By using SIMD, the four units are able to share the same instructions (e.g., "add") and perform computations in parallel. It is possible to provide one or more systems 100 with a single instruction encoded in a loop-nested manner for tensorops to compute inference workloads associated with certain data elements of multidimensional tensors. Therefore, compared with existing methods, System 100 provides better acceleration and parallel processing in tensor computation.

[0071] The accelerator functionality of system 100 corresponds to at least two types of parallel processing. The first type of parallel processing is manifested through the spatial expansion of the innermost loop of a deeply nested loop 402. Because system 100 includes SIMD functionality, the innermost loop boundary of the deeply nested loop 402 can be fixed as the number of MAC operators 108 in a linear unit (e.g., a unit of multiple MACs). The second aspect of parallel processing is manifested by a configuration in which each of the multiple computation modules performs the deeply nested loop 402, but performs tensor computations for a specific neural network layer on a different part. The number of modules in the accelerator device can be configurable, and the number of MAC operators in each module can be configurable.

[0072] In some implementations, programmers or developers may choose to describe or construct specific nested loops in detail, for example, to reduce activation input bandwidth and increase partial and read bandwidth. This can be accomplished by writing or constructing deep nested loops 402 in an alternative manner. The deep nested loops described in this specification for accelerating tensor computations are highly flexible and can be organized as needed to suit specific problem sets. The neural network and computation module (system 100) described in this specification can support computational structures with up to eight deep nested loops, thus allowing developers to tiled inputs and outputs as needed to best utilize locality and reduce overall bandwidth requirements.

[0073] Figure 5 This is a flowchart illustrating an example process 500 for performing tensor computation by using computing system 100 to execute one or more operations encoded by instructions. At block 502, processing unit 102 receives instructions specifying data values ​​for performing tensor computation in system 100. At block 504, in response to receiving the instructions, processing unit 102 performs tensor computation by executing nested loops comprising multiple loops. As described above, processing unit 102 is able to access instructions (different from those associated with instruction set 103) to execute program code for the nested loop procedure. Processing unit 102 is then able to execute the program code to cause TTU 106 to access memory address locations corresponding to the respective elements of the activation tensor 404 and the weight tensor 406.

[0074] At block 506, for an implementation where nested loops have a layer-type defined structure, instruction set 103 can include a specific data value (e.g., one or more opcodes) indicating the layer type and the nested loop structure for that layer. For example, as Figure 3As shown in operation 302, instruction set 103 can include one or more opcodes (e.g., opcode 0) corresponding to tensorops used for tensor computations in convolutional and fully connected layers. Therefore, using data values ​​associated with the tensorop instructions provided by instruction set 103, processing unit 102 can then define a nested loop structure based on the data values ​​of the instructions. Once the nested loop structure is defined, processing unit 102 performs one or more tensor computations, in part, by executing the nested loop procedure, to executor a loop traversing tensor elements for a given dimension of activation tensor 404 and weight tensor 406.

[0075] At block 508, processing unit 102 causes output activation 116 to be written to the corresponding data element of output tensor 408. Additionally, at block 508, processing unit 102 can perform a DMAop.outfeed operation to write the result of tensor computation to one or more other computation modules for subsequent tensor computation. Alternatively, the DMAop.outfeed operation can move the result (output) of the inference workload from memory banks 112, 114 in the first computation module to a classifier module (e.g., a second computation module), and from the classifier module to an external controller and to an external I / O device.

[0076] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in computer software or firmware tangibly implemented, in computer hardware (including the structures disclosed herein and their equivalents), or in combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier) for execution by a data processing device or to control the operation of that data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals generated as encoded signals for transmission to a suitable receiver device for execution by the data processing device). Computer storage media can be machine-readable storage devices, machine-readable storage substrates, random or serial access storage devices, or combinations thereof.

[0077] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program suitable for a computing environment or as a module, component, subroutine, or other unit. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on one or more computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0078] The processing and logic flows described in this specification can be executed by one or more programmable computers through one or more computer programs to perform functions by manipulating input data and producing outputs. The processing and logic flows can also be implemented by special-purpose logic circuitry (such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), GPGPUs (General-Purpose Graphics Processing Units), or some other type of processor), and the device can also be implemented as special-purpose logic circuitry.

[0079] As an example, a computer suitable for executing computer programs includes a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include, or be operatively connected to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or transfer data to them, or both. However, a computer does not need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash memory drive), to name just a few.

[0080] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0081] Embodiments of the subject matter described herein can be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers with graphical user interfaces or web browsers from which users can interact with embodiments of the subject matter described herein), or any combination of these backend, middleware, and frontend components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., communication networks). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0082] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through communication networks. The relationship between clients and servers arises from computer programs running on their respective computers and the client-server relationship between them.

[0083] Further implementation methods are summarized in the following examples: Example 1: A computer implementation method includes: receiving, by a processing unit, an instruction specifying data values ​​for performing tensor computation; and, in response to receiving the instruction, performing the tensor computation by the processing unit by executing a nested loop comprising a plurality of loops, wherein the structure of the nested loops is defined based on one or more of the data values ​​of the instruction.

[0084] Example 2: According to the method of Example 1, the tensor computation is at least part of the computation of the neural network layer.

[0085] Example 3: According to the method of Example 2, wherein the data value specified by the instruction includes at least one value specifying the type of the neural network layer, and wherein the nested structure is defined at least in part by the type of the neural network layer.

[0086] Example 4: According to the method of Example 2 or 3, wherein the instruction causes the processing unit to access at least one element of the dimension of the tensor, the element being a portion of at least one index used when performing the nested loops during the tensor computation.

[0087] Example 5: According to the method of Example 2 or 3, wherein the instructions cause the processing unit to access at least one memory address of an array in a storage medium, the memory address of the array including variables read by the processing unit during the tensor computation.

[0088] Example 6: According to the method of Example 5, wherein performing the tensor computation includes: providing at least one control signal to the tensor traversal unit (TTU) through the processing unit, such that the TTU issues a loop index used during the execution of the loop nesting during the tensor computation.

[0089] Example 7: The method according to Example 5 further includes: providing at least one control signal to the TTU via the processing unit to cause the array reference of the TTU to generate addresses of the referenced array elements for use during the execution of the nested loops during the tensor computation.

[0090] Example 8: According to the method of Example 7, wherein the instruction instructs a first TTU counter to be added to a second TTU counter to generate an address of an array reference associated with the TTU.

[0091] Example 9: According to any one of Examples 1 to 8, performing the tensor computation includes: executing a first synchronization procedure through the processing unit to manage one or more operands associated with performing the tensor computation, wherein managing the operands includes pausing one or more nested loops based on synchronization flag conditions.

[0092] Example 10: According to the method of Example 9, performing the tensor computation includes: executing a second synchronization procedure by the processing unit to manage a counter associated with the nested loop characteristic.

[0093] Example 11: An electronic system includes: a processing unit, a processor disposed in the electronic system, the processing unit including one or more processing means; and one or more machine-readable storage means for storing instructions executable by the one or more processing means to perform operations, the operations including: receiving by the processing unit an instruction specifying data values ​​for performing tensor computation; and, in response to receiving the instruction, performing the tensor computation by the processing unit by executing nested loops comprising a plurality of loops, wherein the structure of the nested loops is defined based on one or more of the data values ​​of the instruction.

[0094] Example 12: The electronic system according to Example 11, wherein the tensor computation is at least part of the computation of the neural network layer.

[0095] Example 13: An electronic system according to Example 12, wherein the data value specified by the instruction includes at least one value specifying the type of the neural network layer, and wherein the nested structure is defined at least in part by the type of the neural network layer.

[0096] Example 14: An electronic system according to Example 12 or 13, wherein the instructions cause the processing unit to access at least one element of a dimension of a tensor, the element being a portion of at least one index used when performing the nested loops during the tensor computation.

[0097] Example 15: An electronic system according to Example 12 or 13, wherein the instructions cause the processing unit to access at least one memory address of an array in a storage medium, the memory address of the array including variables read by the processing unit during the tensor computation.

[0098] Example 16: According to the electronic system of Example 15, performing the tensor computation includes providing at least one control signal to a tensor traversal unit (TTU) via the processing unit to cause the TTU to issue a loop index used during the execution of the loop nesting during the tensor computation.

[0099] Example 17: A non-transitory computer-readable storage medium includes instructions executable by a processing unit, the instructions causing the processing unit to perform operations, the operations including: receiving instructions specifying data values ​​for performing tensor computations by the processing unit; and, in response to receiving the instructions, performing the tensor computations by the processing unit by executing nested loops comprising a plurality of loops, wherein the structure of the nested loops is defined based on one or more of the data values ​​of the instructions.

[0100] Example 18: A non-transitory computer-readable storage medium according to Example 17, wherein the tensor computation is at least part of the computation of a neural network layer.

[0101] Example 19: A non-transitory computer-readable storage medium according to Example 18, wherein the data value specified by the instructions includes at least one value specifying the type of the neural network layer, and wherein the nested structure is defined at least in part by the type of the neural network layer.

[0102] Example 20: A non-transitory computer-readable storage medium according to Example 18, wherein the instructions cause the processing unit to access at least one element of a dimension of a tensor, the element being a portion of at least one index used during the execution of the tensor computation and when the nested loops are performed.

[0103] While this specification contains numerous specific implementation details, these details should not be construed as limiting any feature or the scope of the claim, but merely as descriptions of features specific to particular embodiments. In the context of a single embodiment, certain features described herein can also be implemented in combination in that single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented independently or in any suitable sub-combination in multiple embodiments. Furthermore, while multiple features may be described above as functioning in certain combinations, or even claimed in this manner initially, in some cases, one or more features of the claimed combination can be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0104] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0105] Specific embodiments of the subject matter have been described. Other embodiments fall within the scope of the appended claims. For example, the actions listed in the claims can be performed in different orders and still achieve the desired result. As an example, the processing shown in the figures does not necessarily require the specific or sequential order shown to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for accelerating tensor computation in a neural network comprising multiple neural network layers, the method comprising: A single instruction is obtained from the controller of the hardware computing module, and the single instruction encodes multiple data values; The controller identifies an opcode in the single instruction, the opcode specifying a tensor operation for a layer type of a neural network layer among the plurality of neural network layers. This tensor operation uses a nested loop instruction set to traverse a multidimensional input tensor, wherein two or more instructions in the nested loop instruction set can be executed independently by the controller to perform tensor computations, including: Based on the single instruction, multiple elements are traversed along multiple dimensions of the multidimensional input tensor to obtain the input stored at the corresponding memory location corresponding to the multiple elements; Execute a corresponding subset of the operations in the nested loop instruction set to process the input stored at the corresponding memory location through the neural network layer; and The hardware computing module generates the output of the neural network layer based on the plurality of data values, the layer type specified by the opcode, and the input corresponding to the plurality of elements of the multidimensional input tensor.

2. The method of claim 1, wherein generating the output comprises: The first part of the tensor computation is executed based on the single instruction; In response to the first part of performing the tensor computation, the input set is processed through the neural network layer including the layer type specified by the opcode; as well as The output is generated based on the input set processed by the neural network layer.

3. The method according to claim 2, wherein: The input set is derived from the multidimensional input tensor; and Each of the plurality of elements corresponds to a corresponding input in the input set.

4. The method of claim 2, further comprising: Based on the opcode, it is determined that the layer type of the neural network layer is a convolutional layer type corresponding to the convolutional neural network layer; and Based on the opcode, it is determined that the tensor computation is a convolution operation performed at the convolutional neural network layer.

5. The method according to claim 2, wherein performing the tensor computation comprises: Based on the single instruction, traverse the first plurality of elements at the first dimension of the multidimensional input tensor.

6. The method of claim 5, wherein the multidimensional input tensor is an activation tensor, and the method further comprises: Based on the single instruction, multiple activations of the activation tensor are preloaded into the first memory of the computing unit that receives the single instruction.

7. The method of claim 6, wherein traversing the first plurality of elements in the first dimension comprises: Based on the single instruction, multiple address locations in the first memory are accessed. Each of the plurality of address locations corresponds to a corresponding element of the activation tensor along the first dimension of the activation tensor.

8. The method of claim 6, wherein the computation unit is instructed based on the single instruction to perform only a subset of the total computation required to traverse the multidimensional input tensor.

9. The method of claim 1, wherein the plurality of elements along the plurality of dimensions of the multidimensional input tensor comprises: The first or more elements at the x-dimensional dimension of the multidimensional input tensor; The second or more elements at the y-dimensional dimension of the multidimensional input tensor; as well as The third or more elements at the z-dimensional dimension of the multidimensional input tensor.

10. The method of claim 2, wherein the single instruction comprises a plurality of opcodes, each opcode indicating that the operation type is tensor operation.

11. The method of claim 2, wherein the single instruction comprises a plurality of opcodes, each opcode indicating an operation type of direct memory access (DMA) operation.

12. A system for accelerating tensor computation in a neural network having multiple neural network layers, the system comprising: processor; as well as A non-transitory storage medium for storing instructions, which are executable by the processor to cause operations to be performed, the operations including: A single instruction is obtained from the controller of the hardware computing module, and the single instruction encodes multiple data values; The controller identifies an opcode in the single instruction, the opcode specifying a tensor operation for a layer type of a neural network layer among the plurality of neural network layers. This tensor operation uses a nested loop instruction set to traverse a multidimensional input tensor, wherein two or more instructions in the nested loop instruction set can be executed independently by the controller to perform tensor computations, including: Based on the single instruction, multiple elements are traversed along multiple dimensions of the multidimensional input tensor to obtain the input stored at the corresponding memory location corresponding to the multiple elements; Execute a corresponding subset of the operations in the nested loop instruction set to process the input stored at the corresponding memory location through the neural network layer; and The hardware computing module generates the output of the neural network layer based on the plurality of data values, the layer type specified by the opcode, and the input corresponding to the plurality of elements of the multidimensional input tensor.

13. The system of claim 12, wherein generating the output comprises: The first part of the tensor computation is executed based on the single instruction; In response to the first part of performing the tensor computation, the input set is processed through the neural network layer including the layer type specified by the opcode; as well as The output is generated based on the input set processed by the neural network layer.

14. The system according to claim 13, wherein: The input set is derived from the multidimensional input tensor; and Each of the plurality of elements corresponds to a corresponding input in the input set.

15. The system of claim 13, wherein the operation further comprises: Based on the opcode, it is determined that the layer type of the neural network layer is a convolutional layer type corresponding to the convolutional neural network layer; and Based on the opcode, it is determined that the tensor computation is a convolution operation performed at the convolutional neural network layer.

16. The system according to claim 13, wherein performing the tensor computation includes: Based on the single instruction, traverse the first plurality of elements at the first dimension of the multidimensional input tensor.

17. The system of claim 16, wherein the multidimensional input tensor is an activation tensor, and the operation further comprises: Based on the single instruction, multiple activations of the activation tensor are preloaded into the first memory of the computing unit that receives the single instruction.

18. The system of claim 17, wherein traversing the first plurality of elements in the first dimension comprises: Based on the single instruction, multiple address locations in the first memory are accessed. Each of the plurality of address locations corresponds to a corresponding element of the activation tensor along the first dimension of the activation tensor.

19. The system of claim 17, wherein the computation unit is instructed based on the single instruction to perform only a subset of the total computation required to traverse the multidimensional input tensor.

20. The system of claim 12, wherein the plurality of elements along the plurality of dimensions of the multidimensional input tensor comprise: The first or more elements at the x-dimensional dimension of the multidimensional input tensor; The second or more elements at the y-dimensional dimension of the multidimensional input tensor; as well as The third or more elements at the z-dimensional dimension of the multidimensional input tensor.

21. The system according to claim 13, wherein: The single instruction includes a plurality of opcodes, each opcode indicating that the operation type is tensor operation; and The single instruction includes a second set of opcodes, each opcode indicating that the operation type is a direct memory access (DMA) operation.

Images