Hardware transpose neural network matrix

Abstract

This disclosure relates to transposing a neural network matrix in hardware. The disclosure relates to methods, systems, and apparatus for transposing a neural network matrix in hardware, including computer programs encoded on a computer storage medium. In one aspect, a method includes the actions of: receiving a request on a hardware circuit having a matrix computation unit to perform computation on a neural network, the request specifying a transpose operation to be performed on a first neural network matrix; and generating instructions, when executed by the hardware circuit, to transpose the first neural network matrix by performing a first operation, wherein the first operation includes repeatedly performing the following second operations: updating the first neural network matrix by swapping the upper right quadrant and lower left quadrant of each current submatrix for a current subdivision of the first neural network matrix that divides it into one or more current submatrixes; and updating the current subdivision by subdividing each current submatrix into a corresponding new submatrix.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201810196453.4, filed on March 9, 2018. Technical Field

[0003] This specification relates to transposing neural network matrices in hardware. Background Technology

[0004] A neural network is a machine learning model that uses one or more layers to generate outputs (e.g., classification) from received inputs. Some neural networks include one or more hidden layers in addition to output layers. The output of each hidden layer is used as input to another layer in the network (e.g., the next hidden layer or output layer of the network). Each layer of the network generates an output from the received inputs according to the current values ​​of a corresponding set of parameters. Summary of the Invention

[0005] In general, this specification describes a dedicated hardware circuit for computing neural network inference.

[0006] An innovative aspect of the subject matter described in this specification can be embodied in a method of action comprising the following steps: receiving a request to perform computation on a neural network on a hardware circuit having a matrix computation unit, the request specifying a transpose operation to be performed on a first neural network matrix associated with the neural network; and generating instructions that, when executed by the hardware circuit, cause the hardware circuit to transpose the first neural network matrix by performing a first operation, wherein the first operation includes repeatedly performing the following second operation: for a current subdivision of the first neural network matrix that divides the first neural network matrix into one or more current submatrices, updating the first neural network matrix by exchanging the upper right quadrant and lower left quadrant of each current submatrix in the current subdivision using the matrix computation unit, and subdividing each current submatrix in the current submatrix into a corresponding plurality of new submatrices to update the current subdivision, each of the corresponding plurality of new submatrices being a corresponding quadrant of the current submatrix.

[0007] Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform actions of the method. A system of one or more computers may be configured to perform a specific operation or action using software, firmware, hardware, or any combination thereof installed on the system that can cause the system to perform actions during operation. One or more computer programs may be configured to perform a specific operation or action using instructions including those that cause the apparatus to perform actions when executed by a data processing apparatus.

[0008] Embodiments of this aspect may include one or more of the following optional features. In some implementations, the first operation includes determining that the first neural network matrix is ​​not... i × i Matrix, where i This is the vector length value used for the hardware circuitry; in response, the first neural network matrix is ​​updated to generate by zero-padding the first neural network matrix before all iterations of the second operation. i × i The matrix; and after all iterations of the second operation, the first neural network matrix is ​​transformed back to its state before the update by removing zeros padded during the update. In some embodiments, the first operation further includes obtaining data indicating that one or more values ​​of the first neural network matrix are zero values; and updating the first neural network matrix includes preventing the matrix computation unit from performing any operation on a set of values ​​including at least one of the one or more zero values ​​of the first neural network matrix. In some embodiments, swapping the upper right quadrant of the current submatrix and the lower left quadrant of each current submatrix includes: multiplying each row of the first neural network matrix by one or more partial identity matrices to generate one or more vectors, each of the one or more vectors including a portion of the corresponding row in which the elements of the upper right and lower left quadrants of each corresponding current submatrix are swapped; for each row of the first neural network matrix, combining the vectors corresponding to the portions of each corresponding row of the first neural network matrix in which the elements of the upper right and lower left quadrants of each corresponding current submatrix are swapped; and generating an updated first neural network matrix by combining each corresponding row. In some embodiments, the matrix computation unit performs matrix multiplication as a series of vector multiplication operations. In some embodiments, the second operation further includes generating an initial current subdivision of the first neural network matrix, wherein the initial current subdivision includes an initial submatrix as the first neural network matrix. In some embodiments, the first operation further includes sending instructions to hardware circuitry.

[0009] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Even where the hardware circuitry cannot directly perform matrix transpose operations, transpose operations on matrices can be performed in hardware using dedicated hardware circuitry. By using dedicated hardware circuitry to perform transpose operations, even if the dedicated hardware circuitry does not directly support such processing, neural network operations or other operations specifying the transpose operation can be performed without transmitting data back to the host computer, i.e., without performing at least a portion of the computation outside the chip. This allows transpose operations on matrices to be performed without modifying the hardware architecture of the dedicated hardware circuitry. In other words, processing latency caused by performing a portion of the processing outside the chip, in software, or both is avoided.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and description. Other features, aspects, and advantages of the subject matter will become apparent from the description and drawings. Attached Figure Description

[0011] Figure 1 An example neural network processing system is shown.

[0012] Figure 2 This is a flowchart of an example method for performing computations on a given layer of a neural network.

[0013] Figure 3 An example neural network processing system is shown.

[0014] Figure 4 An example architecture including matrix computation units is shown.

[0015] Figure 5 This shows an example architecture of a cell inside a pulsating array.

[0016] Figure 6 An example architecture for a vector computation unit is shown.

[0017] Figure 7 This is a flowchart of an example process for implementing a neural network that specifies a transpose operation on a matrix.

[0018] Figure 8 This is a flowchart of an example process for exchanging the upper right and lower left quadrants of a submatrix using matrix computation units.

[0019] Figure 9 This is an example of a computation used to perform a transpose operation on a neural network matrix.

[0020] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0021] Neural networks with multiple layers can be used to compute inference. For example, given an input, a neural network can compute inference about that input. The neural network computes this inference by processing the input through each layer of the neural network. Each layer receives an input and processes it according to the weight set used by that layer to generate an output.

[0022] Therefore, in order to compute inference from the received input, the neural network receives the input and processes it through each of the neural network layers to generate inference, where the output from one neural network layer is provided as input to the next neural network layer. The data input to a neural network layer (e.g., the input of the neural network or the output of the layer sequentially below it) can be referred to as the activation input to that layer.

[0023] In some implementations, the layers of the neural network are arranged sequentially. In other implementations, the layers are arranged as a directed graph. That is, any particular layer may receive multiple inputs, multiple outputs, or both. The layers of the neural network may also be arranged such that the output of a layer can be sent back as the input of the previous layer.

[0024] Figure 1 An example neural network processing system 100 is shown. The neural network processing system 100 is an example of a system implemented in one or more locations of a computer, in which the systems, components and technologies described below are implemented.

[0025] The neural network processing system 100 is a system that uses dedicated hardware circuitry 110 to perform neural network calculations. Hardware circuitry 110 is an integrated circuit for performing neural network calculations and includes a matrix calculation unit 120 that performs vector-matrix multiplication in hardware. See below for reference. Figure 3 A more detailed description is given of the example dedicated hardware circuit 120.

[0026] Specifically, the neural network processing system 100 receives a request to implement a neural network on a dedicated hardware circuit 110, implements the neural network on the dedicated hardware circuit 110, and, once the given neural network is implemented, uses the dedicated integrated circuit 110 to process the input of the neural network to generate neural network inference.

[0027] In other words, the neural network processing system 100 can receive a request to specify a neural network architecture for a neural network used to process inputs. This neural network architecture defines the number and configuration of layers in the neural network, as well as the values ​​of the parameters for each layer, which has parameters.

[0028] To implement a neural network on an application-specific integrated circuit 110, a neural network processing system 100 includes a neural network implementation engine 150 implemented as one or more computer programs on one or more computers at one or more physical locations.

[0029] The neural network implementation engine 150 generates instructions that, when executed by the dedicated hardware circuit 110, cause the hardware circuit 110 to perform operations specified by the neural network to generate a neural network output from the received neural network input.

[0030] Once the instructions have been generated by the neural network implementation engine 150 and provided to the hardware circuit 110, the neural network processing system 100 can receive neural network inputs and use the neural network to process the neural network inputs by causing the hardware circuit 110 to execute the generated instructions.

[0031] Some neural networks specify transpose operations on their neural network matrices (e.g., neural network matrices that include the weight values ​​used for the layers of the neural network). For example, some neural networks may specify transpose operations on matrices whose first column is denser than their subsequent columns (i.e., has more meaningful values) to speed up the processing of meaningful values ​​for such matrices. Some neural network training algorithms may require transposing the neural network matrix (e.g., during backpropagation). Some neural networks may require matrix transpose as part of the transformation from a convolutional layer to a fully connected layer, or vice versa.

[0032] The main hardware unit performing matrix operations on hardware circuit 110 is matrix calculation unit 120, which cannot directly perform matrix transpose operations. Therefore, integrated circuits cannot directly perform transpose operations on matrices. To implement a neural network that specifies transpose operations on matrices, neural network implementation engine 150 generates instructions that, when executed by dedicated hardware circuit 110 during the processing of neural network inputs by the neural network, cause hardware circuit 110 to use matrix multiplication unit 120 and vector calculation unit 140 to perform matrix transpose operations on the matrices. See below for reference. Figures 6 to 9 These instructions and other operations are described in more detail.

[0033] Figure 2 This is a flowchart of an example process 200 for performing computations on a given layer of a neural network using dedicated hardware circuitry. For convenience, method 200 will be described with respect to a system having one or more circuits that execute method 200. Method 200 can be executed for each layer of the neural network to compute inference from received inputs.

[0034] The system receives a weight input set (step 202) and an activation input set (step 204) for a given layer. The weight input set and activation input set can be received from a dedicated hardware circuit's dynamic memory and a unified buffer, respectively. In some embodiments, both the weight input set and activation input set can be received from the unified buffer.

[0035] The system uses a matrix multiplication unit of dedicated hardware circuitry to generate an accumulated value from the weight inputs and activation inputs (step 206). In some implementations, the accumulated value is the dot product of the weight input set and the activation input set. That is, for a weight set that is a subset of all weights in the layer, the system multiplies each weight input by each activation input and adds these products together to form the accumulated value. The system can then compute the dot product of other weight sets with other activation input sets.

[0036] The system can use a vector computation unit with dedicated hardware circuitry to output from the accumulated value generation layer (step 208). In some implementations, the vector computation unit applies an activation function to the accumulated value, which will be referenced below. Figure 5 This will be described further. The output of a layer can be stored in a uniform buffer to be used as input to subsequent layers in the neural network or to determine inference. The system completes processing the neural network when the received input has been processed through each layer to generate inference for the received input.

[0037] Figure 3 An example dedicated hardware circuit 300 for performing neural network computations is shown. System 300 includes a host interface 302. Host interface 302 can receive instructions including parameters for neural network computations. Parameters may include one or more of the following: how many layers should be processed, the corresponding set of weight inputs for each layer of the model, the initial set of activation inputs (i.e., the inputs to the neural network that will compute inference), the corresponding input and output sizes for each layer, the stride value for the neural network computation, and the type of layer to be processed, such as a convolutional layer or a fully connected layer.

[0038] The host interface 302 can send instructions to the sequencer 306, which translates the instructions into low-level control signals for the control circuitry to perform neural network computations. In some embodiments, the control signals regulate the data flow in the circuitry, such as how the weight input set and activation input set flow through the circuitry. The sequencer 306 can send control signals to the unification buffer 308, the matrix computation unit 312, and the vector computation unit 314. In some embodiments, the sequencer 306 also sends control signals to the direct memory access engine 304 and the dynamic memory 310. In some embodiments, the sequencer 306 is a processor that generates the control signals. The sequencer 306 can use the timing of the control signals to send them to each component of the circuitry 300 at the appropriate time. In some other embodiments, the host interface 302 receives control signals from an external processor.

[0039] The host interface 302 can send the weight input set and the initial activation input set to the direct memory access engine 304. The direct memory access engine 304 can store the activation input set at a unified buffer 308. In some embodiments, direct memory access stores the weight set in dynamic memory 310, which may be a memory cell. In some embodiments, dynamic memory 310 is located outside the circuitry.

[0040] The unified buffer 308 is a memory buffer. It can be used to store the active input set from the direct memory access engine 304 and the output from the vector computation unit 314. (See below for reference.) Figure 6 The vector computation unit 314 is described in more detail. The direct memory access engine 304 can also read the output of the vector computation unit 314 from the unified buffer 308.

[0041] Dynamic memory 310 and unified buffer 308 can send the weight input set and activation input set to matrix computation unit 312, respectively. In some embodiments, matrix computation unit 312 is a two-dimensional systolic array. Matrix computation unit 312 can also be a one-dimensional systolic array or other circuitry capable of performing mathematical operations (e.g., multiplication and addition). In some embodiments, matrix computation unit 312 is a general-purpose matrix processor. Dedicated hardware circuitry 300 can use matrix computation unit 312 to perform matrix transpose operations. See below for reference. Figures 8 to 9 A more detailed description is given of using matrix computation unit 312 to perform matrix transpose operations.

[0042] Matrix computation unit 312 can process weight inputs and activation inputs and provide an output vector to vector computation unit 314. In some embodiments, matrix computation unit 312 sends the output vector to a unified buffer 308, which in turn sends the output vector to vector computation unit 314. Vector computation unit 314 can process the output vector and store the processed output vector in the unified buffer 308. The processed output vector can be used as an activation input to matrix computation unit 312, for example, for use in subsequent layers of a neural network. References will be made below respectively. Figure 3 and Figure 6 The matrix calculation unit 312 and the vector calculation unit 314 are described in more detail.

[0043] Figure 4 An example architecture 400 including a matrix computation unit is shown. The matrix computation unit is a two-dimensional systolic array 406. The array 406 includes a plurality of cells 404. In some embodiments, a first dimension 420 of the systolic array 406 corresponds to a column of a cell, and a second dimension 422 of the systolic array 406 corresponds to a row of a cell. The systolic array may have more rows than columns, more columns than columns, or the same number of columns and rows.

[0044] In the example shown, value loader 402 sends activation inputs to the rows of array 406, and weight prefetcher interface 408 sends weight inputs to the columns of array 406. However, in some other embodiments, activation inputs are sent to the columns of array 406 and weight inputs are sent to the rows of array 406.

[0045] Value loader 402 can retrieve values ​​from a uniform buffer, such as Figure 3 A unified buffer 308 receives the activation input. Each value loader can send the corresponding activation input to a different leftmost cell of array 406. For example, value loader 412 can send the activation input to cell 414.

[0046] The weight prefetcher interface 408 can receive data from memory cells, such as... Figure 3 The weights are input to the dynamic memory 310. The weight prefetcher interface 408 can send the corresponding weights to different topmost cells of the array 406. For example, the weight prefetcher interface 408 can send weights to cells 414 and 416. The weight prefetcher interface 408 is further capable of receiving multiple weights from memory cells, such as the dynamic memory 310, and sending the multiple weights in parallel to different topmost cells of the array 406. For example, the weight prefetcher interface 408 can send different weights to cells 414 and 416 simultaneously.

[0047] In some implementations, the host interface, for example Figure 3 The host interface 302 shifts the activation inputs across array 406 along one dimension, such as to the right, and simultaneously shifts the weight inputs across array 406 along another dimension, such as downwards. For example, within one clock cycle, the activation input at cell 414 can be shifted to the activation register in cell 416, to the right of cell 414. Similarly, the weight input at cell 416 can be shifted to the weight register in cell 418, below cell 414.

[0048] In each clock cycle, each cell can process a given weight input, a given activation input, and the accumulated output from neighboring cells to generate a cumulative output. The cumulative output can also be propagated to neighboring cells along the same dimension as the given weight input. Alternatively, each cell can process a given weight input and a given activation input to generate an output without processing the accumulated output from neighboring cells. The output can be propagated to neighboring cells along the same dimension as the given weight input and output without being accumulated. See below for reference. Figure 5 Further description of a single cell.

[0049] The accumulated outputs can be passed along the same columns as the weight inputs, for example, towards the bottom of the columns in array 406. In some implementations, at the bottom of each column, array 406 may include accumulator units 410, which store and accumulate each accumulated output from each column as computation is performed using a layer with more activation inputs than rows. In some implementations, each accumulator unit stores multiple parallel accumulations. Accumulator unit 410 can accumulate the output of each accumulation to generate a final accumulated value. The final accumulated value can be passed to a vector computation unit, such as... Figure 6 The vector computation unit. In some other embodiments, when using a layer processing layer with fewer active inputs than rows, the accumulator unit 410 passes the accumulated value to the vector computation unit without performing any accumulation.

[0050] Figure 5 Examples are shown Figure 4 Example architecture 700 of cells within a pulsating array 406.

[0051] The cell may include an activation register 506 storing activation inputs. The activation register may receive activation inputs from the left neighboring cell (i.e., the cell to the left of a given cell) or from a uniform buffer, depending on the cell's position within the systolic array. The cell may include a weight register 502 storing weight inputs. Weight inputs may be passed from the upper neighboring cell or from a weight prefetcher interface, depending on the cell's position within the systolic array. The cell may also include a register summation 504. Register summation 504 may store accumulated values ​​from the upper neighboring cell. A multiplication circuit 508 may be used to multiply the weight input from the weight register 502 with the activation input from the activation register 506. Multiplication circuit 508 may output the product to a summation circuit 510.

[0052] Summing circuit 510 can sum the product and the accumulated value from sum register 504 in register 504 to generate a new accumulated value. Summing circuit 510 can then send the new accumulated value to another sum register in the lower adjacent cell. The new accumulated value can be used as a sum operand in the lower adjacent cell. Summing circuit 510 can also receive the value from sum register 504 in register 504 and send the value from sum register 504 in register 504 to the lower adjacent cell without summing the value from sum register 504 in register 504 with the product from multiplication circuit 508.

[0053] The cell can also shift the weight input and activation input to adjacent cells for processing. For example, weight path register 512 can send the weight input to another weight register in the lower adjacent cell. Activation register 506 can send the activation input to another activation register in the right adjacent cell. Both the weight input and activation input can thus be reused by other cells in the array in subsequent clock cycles.

[0054] In some implementations, the cell also includes a control register. The control register can store control signals that determine whether the cell should shift a weighted input or an activation input to an adjacent cell. In some implementations, shifting a weighted input or an activation input requires one or more clock cycles. The control signals can also determine whether the activation input or weighted input is passed to multiplication circuit 508, or whether multiplication circuit 508 operates on both activation and weighted inputs. The control signals can also be passed to one or more adjacent cells, for example, using wires.

[0055] In some implementations, weights are pre-shifted into a weight path register 512. The weight path register 512 may receive weight inputs, for example, from an upper adjacent cell, and transfer the weight inputs to a weight register 502 based on control signals. The weight register 502 may statically store the weight inputs such that as active inputs are transferred to the cell, for example, via an active register 506, over multiple clock cycles, the weight inputs remain within the cell and are not transferred to adjacent cells. Therefore, the weight inputs may be applied to multiple active inputs, for example, using a multiplication circuit 508, and each accumulated value may be transferred to an adjacent cell.

[0056] Figure 6 An example architecture 700 of the vector computation unit 602 is shown. The vector computation unit 602 can be derived from, for example, a matrix computation unit referenced from... Figure 3 The matrix calculation unit 312 described or Figure 4 The accumulator 410 of the matrix calculation unit receives the vector of accumulated values.

[0057] The vector calculation unit 602 can process the vector of accumulated values ​​in the activation unit 604. In some embodiments, the activation unit includes circuitry that applies a nonlinear function to each accumulated value to generate an activation value. For example, the nonlinear function could be... tanh (x), where x is the cumulative value.

[0058] Optionally, the vector calculation unit 602 may use a pooling circuit 608 to pool, for example, the value of an activation value. The pooling circuit 608 may apply an aggregation function to one or more values ​​to generate pooled values. In some implementations, the aggregation function is a function that returns the maximum, minimum, or average value of a value or a subset of values.

[0059] The control signal 610 can be, for example, from Figure 1The sequencer 306 transmits the data and can adjust how the vector calculation unit 602 processes the vector of accumulated values. That is, the control signal 610 can adjust whether the activation value is pooled, where the activation value is stored in, for example, the uniform buffer 308, or otherwise adjust the processing of the activation value. The control signal 610 can also specify the activation or pooling function, and other parameters used to process the activation or pooling value, such as a step value.

[0060] The vector computing unit 602 can send values, such as activation values ​​or pooling values, to a unified buffer, for example... Figure 3 A unified buffer 308. In some embodiments, the pooling circuit 608 receives an activation value or a pooling value and stores the activation value or pooling value in the unified buffer.

[0061] Figure 7 This is a flowchart of an example process 700 for implementing a neural network that specifies a transpose operation on a matrix. Typically, process 700 consists of dedicated hardware circuitry (e.g., Figure 1 One or more computer systems execute dedicated hardware circuits 110.

[0062] The system receives a request to implement a neural network on a dedicated hardware circuit (step 702). Specifically, the neural network comprises a plurality of neural network matrices, and a transpose operation is specified on the first neural network matrix of the neural network matrices.

[0063] The system generates instructions that, when executed by dedicated hardware circuitry, cause the dedicated hardware circuitry to transpose the first neural network matrix (step 704). These instructions cause the dedicated hardware circuitry to iteratively transpose the matrix by updating each submatrix of the current subdivision during each iteration. Updating each submatrix of the current subdivision involves swapping the upper right quadrant and the lower left quadrant of the current submatrix using the matrix computation unit in the dedicated hardware circuitry. See below. Figure 8 A more detailed description of updating the submatrix of the current subdivision during each iteration.

[0064] Matrix subdivision involves dividing a matrix into one or more submatrices. In each iteration, instructions cause dedicated hardware circuitry to divide the matrix into one or more (e.g., four) submatrices to generate the current subdivision of the matrix. For example, in the first iteration, instructions cause dedicated hardware circuitry 110 to generate an initial current subdivision that includes only one current submatrix. In other words, the current submatrix of the first iteration includes the entire first neural network matrix as a single submatrix. In each subsequent iteration, instructions cause the dedicated hardware circuitry to generate an updated current subdivision by dividing each subdivision in the current subdivision into one or more (e.g., four) subdivisions.

[0065] In some implementations, the first neural network matrix is ​​2.i 2 i Matrix, where i It is a non-negative integer, and the update matrix involves dividing the first neural network matrix into two parts of size 2 at each iteration. j 2 j The submatrix is ​​swapped with a corresponding submatrix that is diagonally adjacent to the specific submatrix rather than vertically or horizontally. In some implementations, j The value in the first iteration is ( i -1), and decreases in each iteration.

[0066] The iteration continues until the currently subdivided submatrix is ​​a single value within the first neural network matrix. At this point, the iteration terminates because a single value can no longer be subdivided into more submatrices.

[0067] In some implementations, the system performs matrix multiplication as a combination of vector multiplications of vectors with the maximum vector length. The maximum vector length is the longest vector that can be multiplied by a matrix in one operation by a matrix computation unit (i.e., without dividing the vector into multiple inputs to the matrix computation unit). For example, if the matrix computation unit is a one-dimensional or two-dimensional pulsating array, the maximum vector length is equal to the number of columns or rows in the unit.

[0068] In some of those implementations, the system receives information indicating that zero values ​​have been added to the neural network matrix to adjust the matrix's dimensions so that it can be divided into vectors with the maximum vector length. In other words, the neural network matrix has been zero-padded to fit the system's architectural configuration. In response to this information, the system can avoid performing value-by-value multiplication operations, which involve values ​​identified as having been added as a result of zero-padded, since such operations always return zero. As a result, the system can reduce the number of value-by-value multiplication operations required to perform such vector multiplications.

[0069] The system sends instructions to the dedicated hardware circuit (step 706).

[0070] For example, the neural network implementation engine 150 can provide instructions to the dedicated hardware circuit 110, and the dedicated hardware circuit 110 can, for example, in Figure 3 Instructions are received at the host interface 302. The neural network implementation engine 150 may also provide other instructions and / or parameters for neural network computation, which can also be received by the host interface 302.

[0071] Figure 8 This is a flowchart of an example process 800 for updating a currently subdivided submatrix of a neural network matrix using dedicated hardware circuitry. For example, process 800 can be... Figure 1 The dedicated hardware circuit 110 executes based on instructions received from the neural network implementation engine 150. The dedicated hardware circuit updates the currently subdivided submatrix by exchanging the upper right quadrant and the lower left quadrant of the submatrix using the matrix calculation unit in the dedicated hardware circuit 110.

[0072] Dedicated hardware circuit 110 creates a vector for each row of the neural network matrix (802).

[0073] Dedicated hardware circuit 110 obtains a partial identity matrix (804) for each value of the commutative submatrix that circuit 110 attempts to generate. Circuit 110 can use the same partial identity matrix to generate two or more values ​​of the commutative submatrix.

[0074] A partial identity matrix is ​​a matrix that contains only "0" and "1" values. The "1" values ​​in the partial identity matrix are strategically positioned such that when multiplied by a vector that includes the values ​​in the rows of the first neural network matrix, the output of the multiplication retains some values ​​of that vector while disregarding the others (i.e., outputs "0").

[0075] In some implementations, if the vector containing the values ​​of the rows of the neural network matrix has dimensions... d Then, when multiplying by this vector, the result vector will be... j and( j +1) values ​​respectively return the values ​​in the vector i and( i The partial identity matrix of the value +1) is in [ i , j ]and[ i +1 and j The +1 position has a value of 1 and has a value of 0 elsewhere. d d matrix.

[0076] Dedicated hardware circuit 110 multiplies each row of the neural network matrix by one or more partial identity matrices to obtain the values ​​from the rows required to update the neural network matrix, in order to swap the upper right quadrant and lower left quadrant of each submatrix in the current segmentation (806).

[0077] For example, the vector V1 = [AB] can include two values ​​from the first row of the neural network matrix. To extract the first value of the vector, dedicated hardware circuitry multiplies V1 by the following partial identity matrix I1:

[0078]

[0079] V1 The output of I1 is [A 0]. Therefore, the value of A is preserved, while the value of B is invalidated.

[0080] Dedicated hardware circuitry 110 combines vectors containing portions of each row of the updated neural network matrix to generate that row (908). For example, V1 might be the first row of the neural network matrix M1:

[0081]

[0082] The first row of the updated matrix S1 corresponding to matrix M1 will include the first element of V1 and the first element of vector V2 = [CD], which includes the values ​​of the second row of matrix M1.

[0083] In other words, the two vectors containing the first row of the update matrix S1 are V1. The output of I1 and V2 The output of I1. Dedicated hardware circuit 110 can combine these two vectors to generate the first row of the commutative submatrix S1.

[0084] Dedicated hardware circuit 110 combines each row of the neural network matrix to generate an updated neural network matrix (810).

[0085] Because swapping the upper right and lower left quadrants of each submatrix can be performed using a matrix multiplication unit (e.g., using a series of matrix-vector multiplications and additions, as further described below), the dedicated hardware circuit 110 can perform transpose operations on a matrix without having the capability for direct matrix transpose operations. Thus, the neural network implementation engine 150 can use the hardware circuit 110 to handle incompatible layers of transpose of a specified matrix.

[0086] Figure 9 This is an example of a computation used to perform a transpose operation on a neural network matrix. It can be used... Figure 7 The process and Figure 3 The dedicated hardware circuit 300 is used to execute Figure 9 Examples.

[0087] exist Figure 9 In part (a), the dedicated hardware circuitry forms the current subdivision of the 4×4 matrix by creating a submatrix that includes the entire matrix. This circuitry creates a vector containing the values ​​of each row of the neural network matrix. For example, the circuitry creates a vector Input[0] containing the values ​​of the first row of the matrix.

[0088] Figure 9Part (b) depicts four partial identity matrices. Each partial identity matrix has a structure defined by the positions of the "1" values ​​within the matrix. The structure of each partial identity matrix is ​​strategically designed to extract certain values ​​from the vectors shown in part (a) while invalidating other values ​​in these vectors. For example, a dedicated hardware circuit uses partial identity matrix W1 to extract the first and second values ​​of a vector.

[0089] exist Figure 9 In part (c), the dedicated hardware circuitry performs four sets of computations using the vectors described in part (a) and the partial identity matrix described in part (b). The circuitry uses each set of computations to generate a row of updated neural network matrices that includes the elements of the neural network matrix depicted in part (a), but with the upper right and lower left quadrants of each submatrix swapped. For example, the circuitry uses the first set of computations to generate [ABIJ], which is the first row of the neural network submatrix with the upper right and lower left quadrants swapped for each submatrix.

[0090] Figure 9 Part (d) depicts updating the output of the neural network matrix depicted in part (a) by swapping the upper right and lower left quadrants of each submatrix in the neural network matrix.

[0091] In section (e), a dedicated hardware circuit divides the updated neural network matrix depicted in section (d) into rows.

[0092] Figure 9 Part (f) depicts four partial identity matrices. The structure of each partial identity matrix described in part (f) is strategically designed to extract certain values ​​from the vectors shown in part (d) while invalidating the other values ​​in those vectors.

[0093] In section (g), the dedicated hardware circuitry performs four sets of computations using the vectors described in section (e) and the partial identity matrix described in section (f). When performed on the neural network matrix depicted in section (e), the computation results in an update to the neural network matrix depicted in section (e) to swap the upper right and lower left quadrants of each submatrix in a new subdivision of the neural network matrix into four submatrices. The new matrix shown in section (h) is the transpose of the matrix shown in section (a).

[0094] exist Figure 9 The operations performed in parts (d)-(g) are Figure 9 The operations performed in parts (a)-(c) are repeated. However, after part (g), the new submatrix formed by the submatrices described in part (e) is a single value that cannot be further subdivided into quadrants. Therefore, these operations will no longer be repeated.

[0095] Embodiments of the subject matter and functional operation described in this specification may be implemented as digital electronic circuits, tangibly implemented computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by or control of a data processing apparatus. Optionally or additionally, the program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of these.

[0096] The term "data processing apparatus" encompasses a variety of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program involved, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof, or combinations of one or more of these.

[0097] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordination files, such as a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on one or more computers located at a site or distributed across multiple sites and interconnected via a communication network.

[0098] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)), and the apparatus can also be implemented as special-purpose logic circuitry (e.g., FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)).

[0099] For example, a computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, to receive data from or transfer data to, or both. However, a computer does not necessarily have to have such a device. Moreover, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0100] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as semiconductor storage devices like EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0101] For transmission for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer with a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user sends input to the computer. Other types of devices can also be used for transmission for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0102] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components (e.g., an application server), or frontend components, such as a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0103] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The relationship between clients and servers is generated by computer programs running on individual computers, and they have a client-server relationship with each other.

[0104] While this specification contains numerous specific implementation details, these details should not be construed as limiting any invention or potentially claimed scope, but rather as descriptions of features that may be specific to particular embodiments of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed to be so, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0105] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0106] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions referenced in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for transposing a neural network matrix on a hardware circuit with a matrix computation unit, comprising: In the current segmentation, the intermediate neural network matrix is ​​divided into multiple sub-matrices; Create multiple vectors, each of which corresponds to a row of the intermediate neural network matrix and includes the value of each row of the intermediate neural network matrix; For the plurality of vectors, a plurality of partial identity matrices are obtained, which are used to extract a portion of the value from each of the plurality of vectors while invalidating the remaining portion of the value; Each of the plurality of vectors is multiplied by one or more of the plurality of partial identity matrices to generate rows of an updated neural network matrix, wherein the updated neural network matrix includes elements of the neural network matrix, but wherein the upper right quadrant and lower left quadrant of the plurality of submatrices in the current subdivision are swapped. The intermediate neural network matrix is ​​updated by combining the generated rows; and The intermediate neural network matrix is ​​generated as the transposed neural network matrix.

2. The method according to claim 1, further comprising: A request is received to perform computations of the neural network on the hardware circuitry, the request specifying a transpose operation to be performed on the neural network matrix.

3. The method according to claim 2, further comprising: determining that the neural network matrix is not i x i matrix, where i is a vector length value of the hardware circuit; In response to the determination, updating the neural network matrix by zero padding the neural network matrix to generate i x i matrix; and By removing the zeros filled during the transpose, the transposed neural network matrix is ​​transformed back to the state of the transposed neural network matrix before the transpose.

4. The method according to claim 3, further comprising: Obtain data indicating that one or more values ​​of the neural network matrix are zero; and The hardware circuitry is prevented from performing any operation on the set of values, which includes at least one of the one or more zero values ​​of the neural network matrix.

5. The method of claim 1, wherein, The hardware circuitry performs matrix multiplication as a series of vector multiplication operations based on the maximum vector length of at least one of the plurality of vectors.

6. The method of claim 2, further comprising transmitting the request to the hardware circuitry.

7. A system comprising a non-transitory machine-readable storage device storing instructions for transposing a neural network matrix using hardware circuitry having a matrix computation unit, the instructions being executable by a processing device to cause the execution of an operation comprising: A method for transposing a neural network matrix on a hardware circuit with a matrix computation unit, comprising: In the current segmentation, the intermediate neural network matrix is ​​divided into multiple sub-matrices; Create multiple vectors, each of which corresponds to a row of the intermediate neural network matrix and includes the value of each row of the intermediate neural network matrix; For the plurality of vectors, a plurality of partial identity matrices are obtained, which are used to extract a portion of the value from each of the plurality of vectors while invalidating the remaining portion of the value; Each of the plurality of vectors is multiplied by one or more of the plurality of partial identity matrices to generate rows of an updated neural network matrix, wherein the updated neural network matrix includes elements of the neural network matrix, but wherein the upper right quadrant and lower left quadrant of the plurality of submatrices in the current subdivision are swapped. The intermediate neural network matrix is ​​updated by combining the generated rows; and The intermediate neural network matrix is ​​generated as the transposed neural network matrix.

8. The system according to claim 7, wherein, The operation also includes: A request is received to perform computations of the neural network on the hardware circuitry, the request specifying a transpose operation to be performed on the neural network matrix.

9. The system according to claim 8, wherein, The operation also includes: Determine that the neural network matrix is ​​not i x i Matrix, where i It is the vector length value of the hardware circuit; In response to the determination, the neural network matrix is ​​updated by zero-padding to generate... i x i Matrix; and By removing the zeros filled during the transpose, the transposed neural network matrix is ​​transformed back to the state of the transposed neural network matrix before the transpose.

10. The system according to claim 9, wherein, The operation also includes: Obtain data indicating that one or more values ​​of the neural network matrix are zero; and The hardware circuitry is prevented from performing any operation on the set of values, which includes at least one of the one or more zero values ​​of the neural network matrix.

11. The system according to claim 7, wherein, The operation also includes: Matrix multiplication is performed as a series of vector multiplication operations based on the maximum vector length of at least one of the plurality of vectors.

12. The system according to claim 8, wherein, The operation also includes: The request is transmitted to the hardware circuit.

13. A non-transitory computer-readable storage medium encoded with instructions for transposing a neural network matrix using hardware circuitry having a matrix computation unit, the instructions being executable by a processing device to perform an operation comprising: In the current segmentation, the intermediate neural network matrix is ​​divided into multiple sub-matrices; Create multiple vectors, each of which corresponds to a row of the intermediate neural network matrix and includes the value of each row of the intermediate neural network matrix; For the plurality of vectors, a plurality of partial identity matrices are obtained, which are used to extract a portion of the value from each of the plurality of vectors while invalidating the remaining portion of the value; Each of the plurality of vectors is multiplied by one or more of the plurality of partial identity matrices to generate rows of an updated neural network matrix, wherein the updated neural network matrix includes elements of the neural network matrix, but wherein the upper right quadrant and lower left quadrant of the plurality of submatrices in the current subdivision are swapped. The intermediate neural network matrix is ​​updated by combining the generated rows; and The intermediate neural network matrix is ​​generated as the transposed neural network matrix.

14. The non-transitory computer-readable storage medium according to claim 13, wherein, The operation also includes: A request is received to perform computations of the neural network on the hardware circuitry, the request specifying a transpose operation to be performed on the neural network matrix.

15. The non-transitory computer-readable storage medium according to claim 14, wherein, The operation also includes: Determine that the neural network matrix is ​​not i x i Matrix, where i It is the vector length value of the hardware circuit; In response to the determination, the neural network matrix is ​​updated by zero-padding to generate... i x i Matrix; and By removing the zeros filled during the transpose, the transposed neural network matrix is ​​transformed back to the state of the transposed neural network matrix before the transpose.

16. The non-transitory computer-readable storage medium according to claim 15, wherein, The operation also includes: Obtain data indicating that one or more values ​​of the neural network matrix are zero; and The hardware circuitry is prevented from performing any operation on the set of values, which includes at least one of the one or more zero values ​​of the neural network matrix.

17. The non-transitory computer-readable storage medium according to claim 13, wherein, The operation also includes: Matrix multiplication is performed as a series of vector multiplication operations based on the maximum vector length of at least one of the plurality of vectors.

18. The non-transitory computer-readable storage medium according to claim 14, wherein, The operation also includes: The request is transmitted to the hardware circuit.

Images