Performing matrix multiplication in hardware

Abstract

Methods, systems, and apparatuses are described that perform matrix multiplication using hardware circuits. An example method begins by obtaining input activation values and weight input values in a first floating point format. The input activation values are multiplied with the weight input values to generate product values in a second floating point format having higher precision than the first floating point format. A partial sum value is obtained in a third floating point format having higher precision than the first floating point format. The partial sum value and the product values are combined to generate an updated partial sum value having the third floating point format.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201880019014.4, filed on May 17, 2018. Technical Field

[0003] This specification relates to performing neural network computations in hardware. Background Technology

[0004] A neural network is a machine learning model that uses one or more layers to generate an output, such as a classification, in response to a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] In general, this specification describes a dedicated hardware circuit for computing neural network inference.

[0006] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method for performing matrix multiplication using hardware circuitry, the method comprising the following actions: obtaining input activation values ​​and weight input values, both having a first floating-point format, by a matrix computation unit of the hardware circuitry; multiplying the input activation values ​​and weight input values ​​by a multiplication circuit of the matrix computation unit to generate a product value having a second floating-point format, the second floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the first floating-point format; obtaining a partial sum value in a third floating-point format by the matrix computation unit, the third floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the first floating-point format; and combining at least the partial sum value and the product value by a summing circuit of the hardware circuitry to generate an updated partial sum value having the third floating-point format.

[0007] Embodiments in this regard may include one or more of the following optional features. The precision of the floating-point format may be determined based on the count of available bits of the significant digits in the floating-point format, and the dynamic range of the floating-point format may be determined based on the count of available bits of the exponent in the floating-point format. The second floating-point format has the same dynamic range as the first floating-point format, and the third floating-point format has the same dynamic range as the first floating-point format. The third floating-point format has higher precision than the second floating-point format.

[0008] The hardware circuitry can be configured to perform computations on a neural network with multiple layers, and the input activation values ​​and weight input values ​​can be associated with one of the multiple layers.

[0009] The method may include the following actions: obtaining the original activation value and original weight value of a first matrix computation unit having a third floating-point format; converting the original activation value to a first floating-point format to generate an input activation value; and converting the original weight value to a first floating-point format to generate a weight input value. The method may further include the following actions: receiving a request to process the original activation value with enhanced precision; generating an activation enhanced precision value for the input value, which is a measure of the difference between the activation input value and the original activation value; and generating a weight enhanced precision value for the weight input value, which is a measure of the difference between the weight input value and the original weight value. Multiplying the activation input value and the weight input value to generate a product value may include the following actions: multiplying the input value by the weight input value, multiplying the input value by the weight enhanced precision value, multiplying the weight input value by the activation enhanced precision value, and multiplying the activation enhanced precision value by the weight enhanced precision value by the multiplication circuit of the matrix computation unit, and combining the products of the multiplications to generate a product value.

[0010] Other embodiments in this regard include corresponding computer systems, apparatuses, and computer programs, all configured to perform actions of the method, recorded on one or more computer storage devices. A system of one or more computers may be configured to perform a specific operation or action by means of software, firmware, hardware, or any combination thereof installed on the system, which in operation may cause the system to perform actions. One or more computer programs may be configured to perform a specific operation or action by means of instructions that, when executed by a data processing apparatus, cause the apparatus to perform actions.

[0011] Another innovative aspect of the subject matter described in this specification can be embodied in the hardware circuitry comprising a matrix computation unit configured to perform a first set of operations, the first set of operations including: acquiring an activation input value and a weight input value, both having a first floating-point format; storing the weight input value in a weight register configured to store a value having the first floating-point format; multiplying the activation input value and the weight input value using a multiplication circuit of the hardware circuitry to generate a product value having a second floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the dynamic range of the first floating-point format; acquiring a partial sum value in a third floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the dynamic range of the first floating-point format; storing the partial sum value in a register configured to store a value having the third floating-point format; and combining the partial sum value with the product value using a summing circuit of the matrix computation unit to generate an updated partial sum value having the third floating-point format.

[0012] Embodiments in this regard may include one or more of the following optional features. A first set of operations may include storing an activation input value in an activation register, the activation register being configured to store a value having a first floating-point format. A first set of operations may include storing a weight input value in a weight register, the weight register being configured to store a value having a first floating-point format. A first set of operations may include storing a partial sum value in a register, the sum being configured to store a value having a third floating-point format. The hardware circuitry may include external summing circuitry outside the matrix computation unit. The first set of operations may include receiving a request to process the original activation value with enhanced precision; generating an activation enhanced precision value for the input value, the activation enhanced precision value being a measure of the difference between the activation input value and the original activation value; and generating a weight enhanced precision value for the weight input value, the weight enhanced precision value being a measure of the difference between the weight input value and the original weight value. Multiplying the activation input value by the weight input value to generate a product value may include the following actions: multiplying the input value by the weight input value, multiplying the input value by the weight enhanced precision value, multiplying the weight input value by the activation enhanced precision value, and multiplying the activation enhanced precision value by the weight enhanced precision value by the multiplication circuitry of the matrix computation unit. The external summing circuit can be configured to perform a second set of operations, including combining the products of multiplications to generate a product value.

[0013] Other embodiments in this regard include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform actions of a first set of operations and a second set of operations. A system of one or more computers may be configured to perform specific operations or actions by means of software, firmware, hardware, or any combination thereof installed on the system, which in operation may cause the system to perform actions. One or more computer programs may be configured to perform specific operations or actions by means of instructions comprising which, when executed by a data processing apparatus, cause the apparatus to perform actions.

[0014] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages: The hardware circuitry can perform matrix multiplication with reduced overflow and / or precision loss. The hardware circuitry can perform matrix multiplication with enhanced precision exceeding that provided by the floating-point format of the input registers in the hardware circuitry. The hardware circuitry can perform matrix multiplication with reduced overflow on input matrices whose values ​​are stored using the IEEE single-precision floating-point format, even if the hardware circuitry stores the input matrix values ​​in a 16-bit floating-point format.

[0015] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0016] Figure 1A A high-level diagram of an example dedicated hardware chip used for training a neural network is shown.

[0017] Figure 1B A high-level example of a computing core is shown.

[0018] Figure 1C An example neural network processing system is shown.

[0019] Figure 2 The diagram illustrates an example architecture that includes matrix multiplication units. The matrix multiplication units are two-dimensional pulsating arrays.

[0020] Figure 3 The figure illustrates an example architecture of multiple individual units within a pulsating array.

[0021] Figure 4 An example architecture of a single entity in a vector computation unit is shown.

[0022] Figure 5 An example architecture for a vector computation unit is shown.

[0023] Figure 6 An example format for floating-point values ​​is shown.

[0024] Figure 7 An example architecture of the multiplication circuit for a single matrix computation unit is shown.

[0025] Figure 8 This is a flowchart of an example process for performing matrix multiplication using a matrix computation unit.

[0026] Figure 9 This is a flowchart of an example process for performing the activation input value multiplied by the weight input value.

[0027] The reference numerals and labels in the accompanying drawings indicate the same elements. Detailed Implementation

[0028] Neural networks with multiple layers can be used to perform computations. For example, given an input, a neural network can compute an inference for that input. The neural network computes this inference by processing the input through each layer in the network. Each layer receives the input and processes it according to the set of weights in that layer to generate an output.

[0029] Therefore, in order to compute inferences based on the received inputs, the neural network receives the inputs and processes them through each layer in the neural network to generate inferences, where the output from one neural network layer is provided as input to the next neural network layer. The data inputs to the neural network layers (e.g., the inputs to the neural network or the outputs of the layers below the layers in sequence) can be referred to as the activation inputs to the layers.

[0030] In some implementations, the layers in a neural network are arranged sequentially. In other implementations, the layers are arranged as a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. Layers in a neural network can also be arranged such that the output of a layer can be sent back to previous layers as input.

[0031] The neural network can also be trained to determine the trained values ​​of the weights of the layers within the network. Typically, during training, the neural network processes the inputs and adjusts the weights based on the outputs generated by the neural network for those inputs.

[0032] This specification describes a dedicated hardware circuit for performing neural network computations, i.e., inference or training operations, including matrix multiplication operations performed by neural network layers.

[0033] Figure 1AA high-level diagram of an example dedicated hardware chip used for training neural networks is shown. As shown, a single dedicated hardware chip comprises two independent processors (102a, 102b). Each processor (102a, 102b) contains two distinct cores: (1) a computational core, namely a Very Long Instruction Word (VLIW) machine (103a, 103b), and (2) a sparse computational core, namely an embedded layer accelerator (105a, 105b).

[0034] Each computing core (103a, 103b) is optimized for dense linear algebra problems. Each computing core executes its own stream of very long instruction words.

[0035] The example sparse computation cores (105a, 105b) map ultrasparse, high-dimensional data to dense, low-dimensional data, allowing the remaining layers to process the densely packed input data. For example, the sparse computation core can perform computations on any embedded layer in a neural network being trained.

[0036] To perform this sparse-to-dense mapping, the sparse computing core uses a pre-built lookup table, or embedding table. For example, given a series of query terms as user input, each query term is converted into a hash identifier or a vector that is one-hot encoded. Using the identifier as a table index, the embedding table returns the corresponding dense vector, which can be an input activation vector for the next layer. The sparse computing core can also perform reduction operations across search query terms to create a dense activation vector. Because embedding tables can be large and unsuitable for the limited capacity of high-bandwidth memory in a dedicated hardware chip, the sparse computing core performs efficient, sparse, distributed lookups. Further details about the functionality of the sparse computing core can be found in U.S. Patent Application No. 15 / 016,486, filed February 5, 2016, entitled MATRIX PROCESSING APPARATUS.

[0037] Figure 1B A high-level example of a computing core (101) is shown. A computing core can be a machine that controls several computing units in parallel, i.e., a VLIW machine. Each computing core (101) includes: scalar memory (104), vector memory (108), scalar processor (107), vector register (106), and extended vector units (i.e., matrix multiplication unit (MXU) (113), transpose unit (XU) (114), and reduction and permutation unit (RPU) (116)).

[0038] The example scalar processor executes VLIW instructions to fetch / execute loops and control the computational core. After fetching and decoding the instruction bundle, the scalar processor itself executes the instructions residing in the scalar slots of the bundle using only a plurality of multi-bit registers (i.e., 32 32-bit registers of the scalar processor (107) and the scalar memory (104)). The scalar instruction set includes normal arithmetic operations, such as those used in address calculations, load / store instructions, and branch instructions. The remaining instruction slots are encoded for instructions used by the vector processor or other extended vector units (113, 114, 116). The decoded vector instructions are forwarded to the vector processor.

[0039] The scalar processor (107) can forward the values ​​of up to three scalar registers along with vector instructions to other processors and units for execution. The scalar processor can also retrieve computation results directly from the vector processor. However, in some implementations, the example chip has a low-bandwidth communication path from the vector processor to the scalar processor.

[0040] The vector instruction scheduler sits between the scalar processor and the vector processor. This scheduler receives decoded instructions from the non-scalar VLIW slot and broadcasts these instructions to the vector processors. (See reference) Figure 1C A detailed description of the vector processor is provided.

[0041] An example scalar processor (107) accesses a small, fast private scalar memory (104), which is backed up by a larger but slower high-bandwidth memory (HBM) (110). Similarly, an example scalar processor accesses a small, fast private vector memory (108), which is also backed up by HBM (110). Word-granular access occurs between the scalar processor (107) and the scalar memory (104), or between the vector processor and the vector memory (108). The granularity of loading and storing between the vector processor and the vector memory is a vector of 128 32-bit words. Direct memory access occurs between the scalar memory (104) and the HBM (110), and between the vector memory (108) and the HBM (110). In some implementations, memory transfers from the HBM (110) to the processor (107) can only be accomplished via either the scalar or vector memory. Alternatively, there may be no direct memory transfers between the scalar memory and the vector memory.

[0042] Instructions can specify operations for the extended vector unit. There are two dimensions, 128 by 8 vector units. Each vector unit can send a register value along with each executed vector unit instruction as an input operand to the extended vector unit. Each extended vector unit receives the input operand, executes the corresponding operation, and returns the result to the vector processor (306). The following will refer to... Figure 4The extended vector unit is described.

[0043] Figure 1C An example application-specific integrated circuit 100 for performing neural network computations is shown. As shown, the chip includes two computational cores (103a, 103b) and two sparse computational cores (152a, 152b).

[0044] The chip has a shared region that includes a host interface (150) to a host computer, four stacks of high-bandwidth memory (156a-156d) along the bottom, and an inter-chip interconnect (148) connecting the interface and memory, as well as data from other chips. Two stacks of high-bandwidth memory (156a-b, 156c-d) are associated with each computing core (103a, 103b).

[0045] The chip stores data in high-bandwidth memory (156c-d), reads data into and out of vector memory (108), and processes the data. The computing core (103b) itself includes vector memory (108) as on-chip S-RAM divided into two dimensions. The vector memory has an address space in which addresses hold floating-point numbers, i.e., each is a 32-bit 128-digit number. The computing core (103b) also includes computation units for calculating values ​​and scalar units for controlling the computation units.

[0046] The vector processor consists of a two-dimensional array of 128×8 vector processing units, all of which execute the same instructions in Single Instruction, Multiple Data (SIMD) mode. The vector processor has channels and sub-channels, i.e., 128 channels and 8 sub-channels. Within a channel, vector units communicate with each other by loading and storing instructions. Each vector unit can access one 4-byte value at a time. Vector units not belonging to the same channel cannot communicate directly. These vector units must use the reduction / permutation units described below.

[0047] The computation unit includes vector registers (32 vector registers) in a vector processing unit (106) that can be used for floating-point and integer operations. The computation unit includes two arithmetic logic units (ALUs) (126c-d) to perform computations. One ALU (126c) performs floating-point addition and the other ALU (126d) performs floating-point multiplication. Both ALUs (126c-d) can perform various other operations, such as shifting, masking, and comparison. For example, the computation core (103b) might want to add vector register V1 and the second vector register V2 and put the result into a third vector register V3. To perform the addition, the computation core (103b) performs the operation multiple times in one clock cycle, i.e., 1024 times. Using these registers as operands, each of the vector units can simultaneously execute two ALU instructions, as well as one load and one store instruction, in each clock cycle. The base address used for the load or store instruction can be computed in the scalar processor and forwarded to the vector processor. Each vector unit in each subchannel can use various methods, such as stepping and address registers with specific indexes, to calculate its own offset address.

[0048] The computation unit also includes an extended unary pipeline (EUP) (116) that performs operations such as square root and reciprocal. Since these operations receive one operand at a time, the computation core (103b) takes three clock cycles to execute them. Because EUP processing takes more than one clock cycle, a first-in-first-out (FIFO) data store is used to store the results. After the operation is complete, the results are stored in a FIFO. The computation core can later use separate instructions to pull data from the FIFO and place it into a vector register. A random number generator (120) allows the computation core (103b) to generate random numbers per cycle, i.e., 128 random numbers per cycle.

[0049] As described above, each processor has three extended vector units: a matrix multiplication unit (113) that performs matrix multiplication operations; a cross-channel unit (XLU) that includes a transpose unit (XU) (114) that performs the transpose operation of a matrix (i.e., a 128x128 matrix); and a reduction and permutation unit (in... Figure 1C The elements are shown as separate units, approximately simple element 115 and permutation element 116.

[0050] The matrix multiplication unit performs matrix multiplication between two matrices. Since the computation core needs to load a set of numbers as the matrices to be multiplied, the matrix multiplication unit (113) receives data. As shown, the data comes from vector registers (106). Each vector register contains 128 × 8 numbers, or 32 bits. However, as the data is sent to the matrix multiplication unit (113), floating-point conversions may occur to change the numbers to a smaller bit size, i.e., from 32 bits to 16 bits. The serializer (130) ensures when the numbers are read from the vector registers; the two-dimensional array, i.e., the 128 × 8 matrix, is read as a set of 128 numbers sent to the matrix multiplication unit (113) in each of the next eight clock cycles. After the matrix multiplication has completed its computation, the results are parallelized (132a, 132b), meaning the resulting matrix is ​​held for several clock cycles. For example, for a 128×8 array, the 128 numbers are held in each of the 8 clock cycles and then pushed to a FIFO (transpose result FIFO (TRF) 134 or multiplication result FIFO (MRF) 136) so that the two-dimensional array of 128×8 numbers can be captured and stored in a vector register (106) in one clock cycle.

[0051] During a period of 128 cycles, the weights are shifted into the matrix multiplication unit (113) as numbers to be multiplied with the matrix. Once the matrix and weights are loaded, the computation core (103b) can send a set of numbers, i.e., 128 × 8 numbers, to the matrix multiplication unit (113). Each row of the set can be multiplied by the matrix to produce multiple results, i.e., 128 results per clock cycle. While the computation core performs matrix multiplication, it also shifts a new set of numbers in the background to the next matrix that the computation core will multiply, so that the next matrix is ​​available when the computation of the previous matrix is ​​complete. The matrix multiplication unit (113) can process the weight inputs and activation inputs and provide the output vector to the vector register 106. The vector processing unit can process the output vector and store the processed output vector in the vector memory. For example, the vector processing unit can apply a nonlinear function to the output of the matrix multiplication unit to generate activation values. In some implementations, the vector computation unit 114 generates normalized values, merged values, or both. The processed output vector can be used as the activation input to matrix multiplication unit 112, for example, in subsequent layers of the neural network.

[0052] The transpose unit transposes the matrix. The transpose unit (114) receives numbers and transposes them, such that the numbers in the channel are transposed using numbers from another dimension. In some implementations, the vector processor includes 128×8 vector units. Therefore, to transpose a 128×128 matrix, a full matrix transpose requires 16 separate transpose instructions. Once transposed, the transposed matrix is ​​available. However, explicit instructions are needed to move the transposed matrix into the vector register file.

[0053] The reduction / permutation units (or units 115, 116) address the problem of cross-channel communication by supporting various operations such as permutation, channel rotation, rotational permutation, channel reduction, permuted channel reduction, and segmented permuted channel reduction. As shown, these computations are separate; however, the computation core can use one computation or another, or a computation linked to another. The reduction simplifier (115) reduces each row of numbers and feeds the numbers to the permutation unit (116). The permutation unit modifies the data between different channels. The transpose unit, reduction simplifier, permutation unit, and matrix multiplication unit all take more than one clock cycle to complete. Therefore, each unit has an associated FIFO, allowing the results of computations to be pushed into the FIFO and later, individual instructions to pull data from the FIFO and push it into a vector register. By using the FIFO, the computation core does not need to retain multiple vector registers for long operation durations. As shown, each unit retrieves data from the vector register (106).

[0054] The computational core uses scalar units to control the computational units. The scalar units have two main functions: (1) performing loop counting and addressing; and (2) generating direct memory address (DMA) requests, causing the DMA controller to move data in the background between high-bandwidth memory (156c-d) and vector memory (108) and then to inter-chip connections (148) to other chips in the example system. The scalar units contain an instruction memory (104), an instruction decoder and issuer (102), a scalar processing unit (107) containing a scalar register (i.e., 32 bits), scalar memory (104), and two ALUs (126a, b) for performing two operations per clock cycle. The scalar units can feed operands and immediate values ​​to vector operations. Each instruction can be sent from the instruction decoder and issuer (102) as an instruction bundle containing instructions to be executed on the vector register (106). Each instruction bundle is a Very Long Instruction Word (VLIW), where each instruction is a multi-bit wide instruction divided into multiple instruction fields.

[0055] Figure 2The diagram illustrates an example architecture 200 including matrix multiplication units (MXUs) 201a and 201b. Each MXU is a two-dimensional systolic array. The arrays are wired together to perform matrix multiplication operations. The MXU multiplies a vector of 128 elements with a preloaded 128x128 matrix, with one multiplication per clock cycle having a constant throughput.

[0056] Each MXU can have 128 rows and 128 columns. MXUs can be divided into identical blocks called tiles. For example, an MXU can be divided into 32 tiles, each containing 32 rows by 16 columns. Each tile can be further divided into multiply-accumulate subcells. Each cell takes a vector data input operand, multiplies the operand by stored weights to obtain the result, and adds the result to a partial sum to produce a new partial sum. In some implementations, subcells can be grouped into larger multicells, such as a 2×2 array of multiply-accumulate subcells or a 4×4 array of multiply-accumulate subcells called sedecim cells. Data can be moved across the systolic array at a rate of one per clock cycle from one multiply-accumulate subcell to the next.

[0057] Before initiating a series of vector-matrix multiplications, the matrices need to be preloaded into the MXU. This matrix data is called the "weights." The weight matrix is ​​transferred to the MXU via the source bus from the bus connected to the MXU and shifted into the weight shift register. The contents of the weight shift register are then loaded into the weight matrix register, allowing the matrix multiplication to begin.

[0058] like Figure 2 As shown, each MXU (e.g., 113a and 113b) is connected to three buses: a first source bus (230a, 240b) for untransposed weights, a second source bus (220a, 220b) for transposed weights, and a left-hand bus (210a, 210b) for vector data to be multiplied with the matrix stored in the MXU. The MXUs are connected to the buses via wires attached to the edges of the MXUs. Each transposed unit (XU) (e.g., 114a and 114b) is also connected to the first and second source buses.

[0059] The first and second source buses are multi-purpose buses containing data sent from the vector processing unit to be consumed by the XU or MXU. Data processing occurs in the vector processing data path, which includes vector register 206, serialization processing unit 202, and selection unit 204. There are several methods by which the vector processing unit can send weights on the bus. Weights can be sent normally, "hi", or "low". Eight 32-bit floating-point numbers for each channel (one 32-bit floating-point number for each sub-channel) are rounded to bfloats, 16-bit floating-point numbers. These values ​​are packed into four pairs and sent to the MXU every other cycle in an eight-cycle process. The difference between normal, "hi", and "low" is how the vector processing unit performs the floating-point 32-to-bfloat conversion. Weights can be packed, meaning that each of the eight 32-bit values ​​for each channel contains a packed pair of bfloats. The source bus is used in each of the eight consecutive cycles, sending sixteen values ​​instead of eight values ​​to the MXU. During odd-numbered cycles, the lower 16 bits of each subchannel are sent to the MXU, and during even-numbered cycles, the higher 16 bits of each subchannel are sent to the MXU. Alternatively, weights may be sent byte-by-byte. Each 32-bit operand contains a set of four packed 8-bit signed two's complement integers. Each byte is converted to a modified signed value. These values ​​are sent to the MXU via the source bus over eight consecutive cycles.

[0060] Weights can be sent as non-transpose or transpose instructions using either the first or second source bus and shifted into the weight shift register. As described below, when triggered by a load operation, the contents of the weight shift register are loaded into the weight matrix. The load path from the weight shift register to the weight matrix register also uses byte-mode data to perform the conversion from the modified signed value to bfloat. The load control bus indicates whether this conversion should be performed.

[0061] Depending on the instruction being executed, a 32-bit value from the source bus may contain either a packed pair of 16-bit floating-point values, where the values ​​in bits [15:0] represent the earlier value (in time), or a packed set of four 8-bit integers in a modified signed-value format, where the values ​​in bits [7:0] represent the earliest value (in time) followed by the others. When the MXU receives data from the bus, the data values ​​are evenly distributed across the MXU, with value 0 on the left and value 127 on the right.

[0062] The LHS data bus transmits 128 16-bit floating-point numbers to be multiplied by a matrix stored in the connected MXU in a specific format, such as bfloat. The data on the LHS data bus originates from the vector processing unit and is passed through transpose units, such as 114a and 114b. When the LHS inputs arrive at the MXU, these values ​​are evenly distributed across the MXU, with values ​​0 on the left and 127 on the right.

[0063] The results of matrix multiplication are uniformly distributed across MXU and sent from MXU to the matrix result FIFO (MRF), e.g., 136a and 136b. The results from XU are sent to the corresponding transpose result FIFO (TRF), e.g., 134a and 134b.

[0064] Figure 3 The diagram illustrates an example architecture of multiple units within a matrix multiplication unit. As described above, the matrix multiplication unit is a two-dimensional systolic array. This array comprises multiple multiply-add sub-units that can be grouped into multiple units. In some implementations, the first dimension of the systolic array corresponds to the columns of a unit, and the second dimension of the systolic array corresponds to the rows of a unit. The systolic array can have more rows than columns, more columns than rows, or an equal number of columns and rows. This specification describes certain processing for columns or vertically. However, different designs can be implemented for rows or horizontally.

[0065] In the illustrated example, data registers 315a and 315b on the left-hand side send vector data inputs to the rows of the array. Weight shift chains 301A and 301B send weight input values ​​to the columns of the array, and weight shift chains 302a and 302b send weight input values ​​to the rows of the array. Shift chains are wired paths along which values ​​can be passed, for example, from the source bus and to each of the multiple registers within the matrix multiplication unit.

[0066] Each weight shift register 305 is designed to shift the weight content value from the source bus along the chain of weight shift registers 305. After all data has been shifted in, a parallel copy operation ensures that all data is copied from the weight shift register 305 to the corresponding weight matrix register 325. While the data is in the weight matrix register 325, the data is used for multiplications of any number of cycles. During this time, more weights can (and typically) be shifted into the weight register 305 in the background to prepare for the next set of multiplications.

[0067] Left-hand data registers 315a and 315b can receive vector data inputs. Each left-hand data register holds one LHS data item per clock cycle. Each vector data input received by multiple units can flow freely in the corresponding left-hand registers of the multiple units, such as left-hand data registers 315a and 315b. The left-hand data registers store vector data inputs, depending on the position of the multiple units within the array, which can be provided by a vector register or by an adjacent multiple unit located to the left of a given multiple unit. For example, if multiple unit 300 is located at the leftmost position within the systolic array of matrix multiplication units, the vector data input is provided by a vector register. The vector register can provide multiple different vector data inputs to multiple unit 300, where each received vector data input can then be stored in a different one of the left-hand data registers 315. Each row receives one value per clock cycle, regardless of the number of rows grouped into multiple units.

[0068] Each left-handed register can be coupled to a single unit along the first dimension of the multi-unit array. The connection from the left-handed register to that single unit is... Figure 3 The dashed lines in the diagram indicate this. For example, the left-hand data register 315a (left-side data register) in the multi-unit array is coupled to units 350a and 350c in the first row. Similarly, the left-hand data register 315b (second left-hand register) in the multi-unit array is coupled to units 350b and 350d in the second row. Each left-hand register 315 transfers the stored vector data to the unit 350 coupled to which the left-hand register is located. Therefore, for a given number of units extending along a first dimension (e.g., along a given row or along a given column), vector data input can be transferred to all units in the multi-unit array, not just a single unit, thereby allowing the activation input to spread rapidly throughout the unit array and improving the operational efficiency of the multi-unit array.

[0069] Multiple vector data inputs can also be sent to an adjacent left-hand register, allowing the multiple vector data inputs to be used in another multi-cell of the array. This process allows shifted vector inputs to be used in another specific multi-cell of the array.

[0070] Each of the multiple units 300, 350, contains stored weight values. Weights are loaded into the units of the systolic array before the matrix multiplication process begins. Dedicated chains and weight shift registers are provided for weight shifting, allowing new weights to be shifted concurrently with the execution of previous matrix multiplication operations. Weight inputs can be loaded into the multiple units in a manner that reduces the latency of the overall matrix multiplication operation.

[0071] As described above, weight shift chains 301 and 302 can receive weight inputs from the source bus. The shift chains can send multiple corresponding weight inputs to the weight register 325 associated with the multi-unit 300.

[0072] In some implementations, the weight shift register shifts vector data inputs along one dimension (e.g., to the right) across the entire array, while simultaneously shifting weight inputs along one or two dimensions (e.g., to the right or down) across the entire array. For example, within one clock cycle, each vector data input of multiple active inputs at multi-unit 300 can be shifted to the corresponding left-hand data register in the next multi-unit in the same row. Each clock cycle, both horizontal data (left-hand data) and vertical data (partial sum) shift one multi-unit per clock cycle. Weights are shifted only when indicated by the system and, depending on the implementation and the instructions executed, can be shifted by 1, 2, or 4 rows (or columns).

[0073] Multiplexer 330 selects weights from the weight shift register 305 of the first shift chain 301 or the second shift chain 302 and forwards the selected inputs to a single row into the weight matrix register 325. Although multiplexer 330 is shown outside the boundary of unit 350, in some implementations, multiplexer 330 exists inside unit 350.

[0074] During a clock cycle, each multi-unit can process multiple given weight inputs and multiple given activation inputs to generate multiple accumulated outputs. Typically, processing involves multiplication operations to multiply the activation inputs with stored weights. The accumulated outputs can also be propagated down the same dimension as the given weight inputs to adjacent multi-units. In some implementations, during a given clock cycle, weights are shifted more than one multi-unit to transition from one convolution computation to another.

[0075] The accumulated output can be passed along the same column as the weight input, for example, towards the bottom of the columns in the array. In some implementations, partial sum registers 310a, 311A ​​pass partial sum values ​​from previous multi-units to multi-units. The array may include partial sum registers 310b, 311b storing the accumulated output from each column of the multi-units. For each column of the multi-unit, the product generated by the sub-units in the column is combined with the incoming partial sum from the aforementioned multi-unit, and then sent as the next partial sum. For some multi-units, such as those in the bottom column of a pulsating array, the accumulated output may include a final accumulated value that can be sent to the vector computation unit. In some implementations, the final accumulated value is sent directly from the bottom multi-unit of the array to the vector computation unit, while in other implementations, the final accumulated value is first stored in memory or processed by different components before being sent to the vector computation unit.

[0076] In some implementations, at each clock cycle, each unit multiplies the given weighted input with the left-hand data input to generate a product value. The unit can then combine the product value with a partial sum value received from another unit to generate an updated partial sum value. The unit can then transfer the partial sum value to another unit in the matrix computation unit.

[0077] Figure 4 An example architecture 400 of a single unit within a pulsating array is shown.

[0078] A single cell may include an activation register 406 for storing activation inputs. Depending on the cell's position within the systolic array, the activation register may receive activation inputs from the left-adjacent cell (i.e., the cell to the left of a given cell) or from a uniform buffer. A single cell may include a weight register 402 for storing weight inputs. Depending on the cell's position within the systolic array, weight inputs may be transferred from the top-adjacent cell or from a weight acquirer interface. A single cell may also include a sum in register 404. The sum in register 404 may store the accumulated value from the top-adjacent cell. The activation register 406 and the weight register 402 may be registers configured to store values ​​of a specific size, such as floating-point values ​​with a specific format.

[0079] Multiplication circuit 408 can be used to multiply the weight input from weight register 402 with the activation input from activation register 406. Multiplication circuit 408 can output the product to summing circuit 410. In some implementations, the input and output values ​​of multiplication circuit 408 can have different sizes and / or formats.

[0080] Summing circuit 410 can sum the product and accumulated value from the sums in register 404 to generate a new accumulated value. Summing circuit 410 can then send the new accumulated value to another sum in the register located in the bottom adjacent unit. The new accumulated value can be used as an operand for summation in the bottom adjacent unit. Summing circuit 410 can also accept the value of the sum from register 404 and send the value of the sum from register 404 to the bottom adjacent unit without adding the value of the sum from register 404 to the product from multiplication circuit 408. In some implementations, the input values ​​of summing circuit 410 can have different sizes and / or formats. In some embodiments, some input and output values ​​of summing circuit 410 can have different sizes and / or formats.

[0081] Individual cells can also shift weight and activation inputs to adjacent cells for processing. For example, weight path register 412 can send weight inputs to another weight register in the bottom adjacent cell. Activation register 406 can send activation inputs to another activation register in the right adjacent cell. Therefore, both weight and activation inputs can be reused by other cells in the array in subsequent clock cycles.

[0082] In some implementations, the unit also includes a control register. The control register can store control signals that determine whether the unit should shift the weight input or the activation input to an adjacent unit. In some implementations, shifting the weight input or activation input takes one or more clock cycles. The control signals can also determine whether the activation input or the weight input is passed to multiplication circuit 408, or whether multiplication circuit 408 operates on both the activation input and the weight input. The control signals can also be passed to one or more adjacent units, for example, using wires.

[0083] In some implementations, the weights are pre-shifted into a weight path register 412. The weight path register 412 can, for example, receive weight inputs from the top adjacent cell and transfer the weight inputs to a weight register 402 based on control signals. The weight register 402 can statically store the weight inputs such that while active inputs are transferred to a cell over multiple clock cycles, for example via an active register 406, the weight inputs remain within the cell and are not transferred to adjacent cells. Therefore, the weight inputs can, for example, be applied to multiple active inputs using a multiplication circuit 408, and the corresponding accumulated values ​​can be transferred to adjacent cells.

[0084] Figure 5 An example architecture 500 of a vector computation unit 502 is shown. The vector computation unit 502 can generate data from a matrix computation unit (e.g., reference...). Figure 3 The described matrix calculation unit 312) receives a vector of accumulated values.

[0085] Control signal 510 can be transmitted and can adjust how vector calculation unit 502 processes the vector of accumulated values. That is, control signal 510 can adjust whether the left-handed data value is pooled, whether the left-handed data value is stored in, for example, ve, or can otherwise adjust the processing of the activation value. Control signal 510 can also specify the activation or pooling function and other parameters used to process the activation value or pooled value, such as the step size.

[0086] Vector computation unit 502 can send values, such as left-handed data values ​​or pooled values. In some implementations, pooling circuit 508 receives the activation value or pooled value and stores it in a unified buffer.

[0087] Figure 6Example format 600 for floating-point values ​​is shown. Each of the values ​​processed by the matrix computation unit (e.g., the values ​​stored in the registers of the individual units of the matrix computation unit) can be stored as a floating-point value with a specific format.

[0088] The format 600 is characterized by a sign bit 601, a set of bits called the significand 602, and another set of bits called the exponent 603.

[0089] The sign bit 601 indicates whether the value stored using format 600 is positive or negative. The significant number 602 includes one or more significant digits of the value stored using format 600. Therefore, the size of the significant number 602 of format 600, i.e., the number of bits, represents the maximum possible precision of any value stored using format 600. The exponent 603 represents the power of the range base used to convert the stored value to its normalized form. Therefore, the size of the exponent 603 of format 600 represents the maximum possible dynamic range of any value stored using format 600.

[0090] In some cases, the normalized form used by the system to interpret floating-point numbers contains one or more constant values. For example, the normalized form can always be of form 1.XXXX. 2^XXXX, where the integer part of the first value is always constant, for example, equal to 1. In some such cases, the significant number 602 may include only the non-constant bits of the normalized value and exclude the constant bits. In these cases, the bits that are constant in the normalized form and therefore do not appear in the significant number 602 are called "hidden bits". A computer system that interprets a binary floating-point value with hidden bits will add the hidden bits to the significant number 602 according to the normalized form that forms the value in format 600.

[0091] The way a computer system stores a binary number as a floating-point value depends on the normalized form the system uses to interpret the floating-point number, the size of the significant digits 602, and the exponent 603 of the specific floating-point format 600 used. For example, the floating-point format 600 may include a sign bit 601, 4 significant digits 602, and a 3-bit exponent 603, and the computer system may store the number in the normalized form X.XXX. 2^XXX is used to interpret binary numbers with this format 600, where X is a single binary digit, either "0" or "1". Furthermore, the computer system can assume that the binary value before the decimal point in the first value of the normalized form is always a single hidden bit that does not appear in the significant digit 602. Therefore, such a computer system can interpret the binary number +11.111 using a sign bit 601 (because the binary number is positive) that is "0", an exponent 603 that is "001", and a significant digit 602 that is "1111".

[0092] If a computer system cannot correctly store numbers using the specified format, attempting to store those numbers may cause overflow and lead to unpredictable or undesirable behavior. The example illustrating the computer system described above can store a number exceeding the maximum precision binary number allowed in the valid number 602 of the system's format 600 without overflowing, even though such rounding results in reduced precision.

[0093] On the other hand, if the dynamic range of a binary number exceeds the maximum range allowed in exponent 603 of format 600, the computer system cannot round the dynamic range. Continuing with the example given above, the computer system cannot store and interpret the number 111111111.012 because the normalized form of this number has a dynamic range of 10002, and this dynamic range cannot be represented using the 3-bit example exponent width within the range of values ​​allowed in exponent 603 of format 600.

[0094] As another example, the system can use the bfloat format with an 8-bit exponent and 7 significant digits. To reduce the possibility of overflow, if an operation in the computer system transforms a first floating-point value with a first format to a second floating-point value with a second format in any way, it is important that the dynamic range of the second format is greater than or equal to the dynamic range of the first format. This includes cases where the system converts a first floating-point value to a second floating-point value and cases where the system applies an operation to a first floating-point value to generate a second floating-point value. For example, if the computer system multiplies two values ​​with a first format to generate a result value with a second format, it is important that the dynamic range of the second format is greater than or equal to the dynamic range of the first format to reduce the possibility of overflow. If the two values ​​being multiplied have different formats, it is important that the dynamic range of the second format is greater than or equal to the dynamic range of the format with the larger dynamic range to reduce the possibility of overflow.

[0095] Examples of floating-point format 600 include the IEEE single-precision format, the bfloat format, and the extended bfloat format.

[0096] The IEEE single-precision format is a 32-bit format, comprising 1 sign bit (601), 8 exponent bits (603), and 23 significant bits (602). The bfloat format is a 16-bit format, comprising 1 sign bit (601), 8 exponent bits (603), and 7 significant bits (602). The extended bfloat format is a 20-bit format that includes a 20-bit format, comprising 1 sign bit (601), 8 exponent bits (603), and 11 significant bits (602).

[0097] Importantly, all three formats mentioned above have the same exponent size of 603, and therefore the same dynamic range. However, the single-precision format allows for more precision than the extended bfloat format, and the extended bfloat format allows for more precision than the bfloat format. To reduce the possibility of overflow while improving precision, the matrix computation unit can store the activation input and weight input values ​​in a register that stores the values ​​in the bfloat format, store the product of the input values ​​in a register that stores the values ​​in the extended bfloat format, and store the sum of the product and partial sums in a register that stores the values ​​in the IEEE single-precision format.

[0098] Figure 7 An example architecture 700 for the multiplication circuit used in matrix computation is shown. Figure 7 The matrix computation unit depicted in the diagram multiplies two input values ​​(e.g., activation input value 701 and weight input value 702) to generate a result value, such as product value 705.

[0099] Architecture 700 includes a multiplier 703 and an adder 704. The multiplier 703 multiplies the significant digits and sign bits of two input values ​​to generate the significant digits and sign bits of the result value. The adder 704 adds the exponents of two input values ​​to generate the exponent of the result value. The combination of the significant digits, sign bits, and exponent of the result value generates the result value.

[0100] Figure 8 This is a flowchart of an example process 800 for performing matrix multiplication using a matrix computation unit. Process 800 can be performed by a matrix computation unit (e.g., ... Figure 2 The matrix computation unit 113) performs matrix multiplication according to the architecture of the matrix computation unit.

[0101] The matrix computation unit can execute process 800 multiple times in parallel to compute a vector output as a product of a vector and a matrix (e.g., an input vector containing multiple activation input values ​​and a weight matrix containing multiple activation weight values).

[0102] The matrix calculation unit obtains activation input values ​​and weight input values ​​(802) in first floating-point format. See below for reference. Figure 9 A more detailed description is provided of obtaining the activation input and weight input values.

[0103] The matrix computation unit uses its multiplication circuitry to multiply the activation input value and the weight input value to generate a product value (804) with a second floating-point format. The second floating-point format has higher precision than the first floating-point format and a dynamic range at least as large as that of the first floating-point format. By storing the result of multiplying the activation input value and the weight input value in a format with higher precision than those input values, the matrix computation unit reduces the possibility of losing precision in storing the multiplication result. By storing the result of multiplying the activation input value and the weight input value in a format with a dynamic range at least as large as that of those input values, the matrix computation unit also reduces the possibility of overflow in storing the multiplication result.

[0104] The following is for reference. Figures 8 to 9 The process of multiplying the activation input value with the weight input value is described in more detail.

[0105] In some implementations, the first floating-point format is a 16-bit format with 1 sign bit, 8 exponent bits, and 7 significant bits, the 7 significant bits optionally excluding the hidden bits in the normalized form of the corresponding binary number, and the second floating-point format is a 20-bit format with 1 sign bit, 8 exponent bits, and 11 significant bits, the 11 significant bits optionally excluding the hidden bits in the normalized form of the corresponding binary number.

[0106] The matrix calculation unit obtains partial sum values ​​in a third floating-point format (806). The matrix calculation unit can also obtain partial sum values ​​from individual units within the matrix calculation unit.

[0107] The third floating-point format has higher precision than the first floating-point format and a dynamic range at least as large as that of the first floating-point format. Therefore, the partial sum value has a format that allows for higher precision than the format of the input value and a dynamic range at least as large as that of the format of the input value.

[0108] In some implementations, the third floating-point format has higher precision than the second floating-point format. In other words, the three floating-point formats can be ordered in order of their precision, starting with the format with the highest precision: third floating-point format, second floating-point format, and first floating-point format. In some implementations, the third floating-point format has a dynamic range that is at least as large as that of the second floating-point format.

[0109] In some implementations, the third floating-point format is the IEEE Standard 754 single-precision format or another 32-bit format with 1 sign bit, 8 exponent bits, and 23 significant bits, excluding the hidden bits in the normalized form of the corresponding binary number.

[0110] The matrix computation unit uses its summing circuitry to combine the partial sum and the product value to generate an updated partial sum (808) with a third format. By storing the combined result of the product and partial sum in the same format as the partial sum, the matrix computation unit reduces the possibility of overflow or loss of precision. This is especially true in implementations where the format of the product value (i.e., the second format) has lower precision than the format of the updated partial sum (i.e., the third format). In such implementations, the matrix computation unit reduces the possibility of loss of precision by storing the combined result in a format with higher precision than the format of the product value. Similarly, in implementations where the third format has a dynamic range at least as large as that of the second format, the matrix computation unit reduces the possibility of overflow by storing the combined result in a format with a larger dynamic range than the format of the product value.

[0111] In some implementations, the matrix computation unit will update parts and transfer them to another component of the matrix computation unit, such as a single unit or a multi-unit structure in the matrix computation unit.

[0112] Figure 9 This is a flowchart for performing an example process 900 that multiplies activation input values ​​by weight input values. For convenience, process 900 will be described as being performed by a system of one or more computers located at one or more locations. For example, a neural network processing system appropriately programmed according to this specification (e.g., Figure 1A The neural network processing system 100 can execute process 900.

[0113] The neural network system can execute process 900 multiple times in parallel to compute a vector output that includes a higher-precision part of the product of a vector and a matrix (e.g., an input vector including multiple activation input values ​​and a weight matrix including multiple activation weight values) and a lower-precision part of the product of the vector and the matrix.

[0114] The system obtains the original activation values ​​and original weight values ​​(902). The system can implement an engine from the system's neural network (e.g., Figure 1C The neural network implementation engine (150) obtains the raw values. The raw values ​​can be in any format, such as IEEE single-precision floating-point format.

[0115] The system converts the original activation value to a first format to generate the activation input value (904) and the original weight value to a first format to generate the weight input value (906). The system can then store the number represented by the original activation value as a new value with the new format.

[0116] The system determines whether it has received a request to multiply the original activation value by an activation input value with enhanced precision (908). The system can do this from the system's end-user and / or from the system's neural network implementation engine (e.g., Figure 1C The neural network implementation engine 150 receives this enhanced precision request. The request instructs the system to store the result of multiplying the original activation values ​​by the activation input values ​​with reduced precision loss.

[0117] In some implementations, the system receives enhanced precision requests via software instructions (i.e., VLIW instruction bundles). These instruction bundles may include matrix multiplication instructions with multiple different possible opcodes used to express various options, such as components of enhanced precision matrix multiplication. If the system determines that it has not received an enhanced precision request, it uses the matrix computation unit of the hardware circuitry implementing the system to multiply the activated input value by the original activated value to generate a product value with a second format (910).

[0118] Otherwise, if the system determines that it has received an enhanced precision request, the system generates an activation enhanced precision value (912) as the difference between the original activation value and the activation input value, and a weight enhanced precision value (914) as the difference between the original weight value and the weight input value. The system generates the difference between the activation input value and the original input value by subtracting the activation input value from the original activation value, and generates the difference between the weight input value and the original weight value by subtracting the weight input value from the original weight value. The system can perform the subtraction of the two values ​​using appropriate circuitry outside the matrix computation unit, such as using a summation circuit outside the matrix computation unit by adding the first value to the negative of the second value. Both the activation enhanced precision value and the weight enhanced precision value are values ​​in a first floating-point format.

[0119] The system uses the matrix computation unit to perform a set of multiplications between the activation input value, the weight input value, the activation enhancement precision value, and the weight enhancement precision value (916). In order to use the matrix computation unit to perform a multiplication between two values, the system provides two values ​​to the matrix computation unit so that the matrix computation unit performs the multiplication of the two values.

[0120] In some implementations, the set of multiplications includes: multiplying the activation input value by the weight input value, multiplying the activation input value by the weight augmentation precision value, multiplying the weight input value by the activation augmentation precision value, and multiplying the activation augmentation precision value by the weight augmentation precision value.

[0121] The software instructions determine a subset of possible multiplications to include. In some implementations, the set of multiplications includes only multiplying the activation input value by the weight input value and multiplying the activation enhancement precision value by the weight enhancement precision value. This technique can be used to reduce the number of required multiplications when the system determines that at least some of the individual input values ​​and enhancement precision values ​​have the desired level of accuracy.

[0122] The enhanced precision values ​​indicate at least some of the lower precision portions of the original value that were lost during rounding when the original value was stored as an input value with a first format. By using these enhanced precision values ​​in multiplication, the system can perform multiplication involving the higher precision portions of the original value and thus produce multiplication results with higher precision.

[0123] The system then combines the product of the four multiplications to generate (918) a first value in a first format comprising the lower precision portion of the result of multiplying the original values ​​and a second value in a first format comprising the higher precision portion of the result of multiplying the original data. In some implementations, the system uses external summing circuitry outside the matrix computation unit to perform the summation.

[0124] The embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, including structures disclosed in this specification and their structural equivalents, or combinations thereof, in tangibly embodied computer software or firmware, or in computer hardware. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals generated for encoding information to be transmitted to a suitable receiver device for execution by a data processing device, such as machine-generated electrical, optical, or electromagnetic signals. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof.

[0125] The term "data processing apparatus" includes all types of means, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0126] Computer programs (also referred to as or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. Programs can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program, or in multiple files or multiple coordination files. The portion of the file holding other programs or data is, for example, one or more scripts stored in a markup language document, and the multiple coordination files are, for example, files storing portions of one or more modules, subroutines, or code. Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0127] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, and the devices can be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0128] Computers suitable for executing computer programs include, for example, central processing units (CPUs) that may be based on general-purpose or special-purpose microprocessors, or both, or any other type. Typically, the CPU receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the CPU for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not require such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device, to name just a few, such as a universal serial bus (USB) or flash drive.

[0129] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0130] To transmit interactions with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to transmit interactions with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including auditory, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0131] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, middleware components, or frontend components, or any combination of one or more such backend, middleware, or frontend components. The backend components may be, for example, a data server; the middleware components may be, for example, an application server; and the frontend components may be, for example, a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0132] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0133] Example 1 is a method for performing matrix multiplication using hardware circuitry. The method includes: obtaining an input activation value and a weight input value by a matrix calculation unit of the hardware circuitry, both the input activation value and the weight input value having a first floating-point format; multiplying the input activation value and the weight input value by a multiplication circuit of the matrix calculation unit to generate a product value having a second floating-point format, the second floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the dynamic range of the first floating-point format; obtaining a partial sum value in a third floating-point format by the matrix calculation unit, the third floating-point format having higher precision than the first floating-point format and having a dynamic range at least as large as the dynamic range of the first floating-point format; and combining at least the partial sum value and the product value by a summing circuit of the hardware circuitry to generate an updated partial sum value having the third floating-point format.

[0134] Example 2 is the method described in Example 1, wherein the precision of the floating-point format is determined based on the count of available bits of the significant numbers in the floating-point format, and the dynamic range of the floating-point format is determined based on the count of available bits of the exponent in the floating-point format.

[0135] Example 3 is any one of the methods described in Examples 1-2, wherein the second floating-point format has the same dynamic range as the first floating-point format, and the third floating-point format has the same dynamic range as the first floating-point format.

[0136] Example 4 is the method described in any of Examples 1-3, wherein the third floating-point format has higher precision than the second floating-point format.

[0137] Example 5 is a method described in any of Examples 1-4, wherein the hardware circuit is configured to perform computations on a neural network having multiple layers, and the input activation values ​​and the weight input values ​​are associated with one of the multiple layers.

[0138] Example 6 is a method described in any one of Examples 1-5, further comprising: obtaining the original activation value and original weight value of a first matrix computation unit having the third floating-point format; converting the original activation value into the first floating-point format to generate the input activation value; and converting the original weight value into the first floating-point format to generate the weight input value.

[0139] Example 7 is a method described in any of Examples 1-6, further comprising: receiving a request to process the original activation value with enhanced precision; generating an activation enhancement precision value for the input value, the activation enhancement precision value being a measure of the difference between the activation input value and the original activation value; and generating a weight enhancement precision value for the weight input value, the weight enhancement precision value being a measure of the difference between the weight input value and the original weight value; and wherein multiplying the activation input value and the weight input value to generate the product value comprises: multiplying the input value and the weight input value, multiplying the input value and the weight enhancement precision value, multiplying the weight input value and the activation enhancement precision value, and multiplying the activation enhancement precision value and the weight enhancement precision value by the multiplication circuit of the matrix calculation unit, and combining the products of the multiplications to generate the product value.

[0140] Example 8 is a hardware circuit comprising: a matrix calculation unit configured to perform a first set of operations, the first set of operations including: acquiring an activation input value and a weight input value, both the activation input value and the weight input value having a first floating-point format; storing the weight input value in a weight register, the weight register being configured to store a value having the first floating-point format; and multiplying the activation input value and the weight input value using a multiplication circuit of the hardware circuit to generate a product value having a second floating-point format, the second floating-point format having a higher density than the first floating-point format. The method includes: obtaining a partial sum value in a third floating-point format with higher precision and a dynamic range at least as large as that of the first floating-point format; storing the partial sum value in a register configured to store values ​​in the third floating-point format; and using the summation circuitry of the matrix computation unit to combine the partial sum value with the product value to generate an updated partial sum value in the third floating-point format.

[0141] Example 9 is the hardware circuit described in Example 8, and the first set of operations further includes: storing the activation input value in an activation register, the activation register being configured to store a value having the first floating-point format.

[0142] Example 10 is any of the hardware circuits described in Examples 8-9, wherein the first set of operations further includes: storing the weight input value in a weight register, the weight register being configured to store a value having the first floating-point format.

[0143] Example 11 is any one of the hardware circuits described in Examples 8-10, wherein the first set of operations further includes: storing the partial sum value in a register, wherein the sum in the register is configured to store a value having the third floating-point format.

[0144] Example 12 is any of the hardware circuits described in Examples 8-11, further including an external summing circuit outside the matrix calculation unit, and wherein the first set of operations further includes: receiving a request to process the original activation value with enhanced precision; generating an activation enhancement precision value for the input value, the activation enhancement precision value being a measure of the difference between the activation input value and the original activation value; and generating a weight enhancement precision value for the weight input value, the weight enhancement precision value being a measure of the difference between the weight input value and the original weight value; and wherein multiplying the activation input value and the weight input value to generate the product value includes: multiplying the input value and the weight input value by the multiplication circuit of the matrix calculation unit, multiplying the input value and the weight enhancement precision value, multiplying the weight input value and the activation enhancement precision value, and multiplying the activation enhancement precision value and the weight enhancement precision value, and wherein the external summing circuit is configured to perform a second set of operations, the second set of operations including: combining the products of the multiplications to generate the product value.

[0145] Example 13 is a system comprising one or more computers and one or more storage devices containing storage instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding methods of any one of Examples 1-7.

[0146] Example 14 is a computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method of any one of Examples 1-7.

[0147] Embodiment 15 is a system comprising one or more computers and one or more storage devices containing storage instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding hardware circuitry of any one of Embodiments 8-12.

[0148] Example 16 is a computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding hardware circuitry of any one of Examples 8-12.

[0149] While this specification contains numerous specific implementation details, these should not be construed as limiting any invention or the scope of claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially stated so, in some cases, one or more features in a claimed combination may be removed from the combination, and the claimed combination may be adapted for sub-combinations or variations thereof.

[0150] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged as multiple software products.

[0151] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions detailed in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method for performing matrix multiplication using hardware circuitry, the method comprising: The activation input value and the weight input value are obtained by the matrix calculation unit of the hardware circuit, each having a first floating-point format, wherein the hardware circuit is configured to perform calculations of a neural network with multiple layers, wherein the activation input value and the weight input value are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The multiplication circuit of the matrix calculation unit multiplies the activation input value and the weight input value to generate a product value, which has a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; The matrix calculation unit obtains a partial sum value in a third floating-point format, wherein the third floating-point format differs from both the first and second floating-point formats and has higher precision than the first floating-point format; and The summing circuit of the hardware circuitry combines at least the partial sum with the product value to generate an updated partial sum with the third floating-point format.

2. The method according to claim 1, wherein, The precision of the floating-point format is determined based on the count of available bits of the significant numbers in the floating-point format, and the dynamic range of the floating-point format is determined based on the count of available bits of the exponent in the floating-point format.

3. The method according to claim 1, wherein: The second floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

4. The method according to claim 1, wherein, The second floating-point format has the same dynamic range as the first floating-point format, and the third floating-point format has the same dynamic range as the first floating-point format.

5. The method according to claim 1, wherein, The third floating-point format has higher precision than the second floating-point format.

6. The method according to claim 1, wherein: The third floating-point format has higher precision than the first floating-point format; and The third floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

7. The method of claim 1, further comprising: Obtain the original activation value and original weight value of the matrix calculation unit having the third floating-point format; The original activation value is converted into the first floating-point format to generate the activation input value; and The original weight values ​​are converted to the first floating-point format to generate the weight input values.

8. The method of claim 7, further comprising: Receive a request to process the original activation value with enhanced precision; Generate an activation enhancement precision value for the activation input value, wherein the activation enhancement precision value is a measure of the difference between the activation input value and the original activation value; as well as Generate a weight enhancement precision value for the weight input value, wherein the weight enhancement precision value is a measure of the difference between the weight input value and the original weight value; as well as The process of multiplying the activation input value and the weight input value to generate the product value includes: The multiplication circuit of the matrix calculation unit, Multiply the activation input value by the weight input value. Multiply the activation input value by the weight enhancement precision value. Multiply the weight input value by the activation enhancement precision value, and Multiply the activation enhancement precision value by the weight enhancement precision value, and The products of the multiplications are combined to generate the product value.

9. A hardware circuit, comprising: A matrix calculation unit is configured to perform a first set of operations, the first set of operations including: The activation input value and the weight input value are obtained, both having a first floating-point format, wherein the hardware circuit is configured to perform computation of a neural network having multiple layers, wherein the activation input value and the weight input value are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The weight input value is stored in a weight register, which is configured to store values ​​having the first floating-point format; Using the multiplication circuit of the hardware circuit, the activation input value is multiplied by the weight input value to generate a product value, the product value having a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; Obtain a partial sum value in a third floating-point format, wherein the third floating-point format is different from the first floating-point format and the second floating-point format and has higher precision than the first floating-point format; The partial sum is stored in a register, the sum in which is configured to store a value having the third floating-point format; and The summation circuit of the matrix calculation unit is used to combine the partial sum with the product value to generate an updated partial sum with the third floating-point format.

10. The hardware circuit according to claim 9, wherein: The second floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

11. The hardware circuit according to claim 9, wherein: The third floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

12. The hardware circuit according to claim 9, wherein the first group of operations further includes: The activation input value is stored in an activation register, which is configured to store a value having the first floating-point format.

13. The hardware circuit according to claim 9, wherein the first group of operations further includes: The weight input value is stored in a weight register, which is configured to store values ​​having the first floating-point format.

14. The hardware circuit according to claim 9, wherein the first group of operations further includes: The partial sum is stored in a register, which is configured to store a value having the third floating-point format.

15. The hardware circuit according to claim 9, further comprising an external summation circuit outside the matrix calculation unit, wherein, The first group of operations further includes: Receive a request to process the original activation value with enhanced precision; Generate an activation enhancement precision value for the activation input value, wherein the activation enhancement precision value is a measure of the difference between the activation input value and the original activation value; and Generate a weight enhancement precision value for the weight input value, wherein the weight enhancement precision value is a measure of the difference between the weight input value and the original weight value; and The process of multiplying the activation input value and the weight input value to generate the product value includes: The multiplication circuit of the matrix calculation unit, Multiply the activation input value by the weight input value. Multiply the activation input value by the weight enhancement precision value. Multiply the weight input value by the activation enhancement precision value, and Multiply the activation enhancement precision value by the weight enhancement precision value, and The external summing circuit is configured to perform a second set of operations, which include: The products of the multiplications are combined to generate the product value.

16. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform an operation, the operation comprising: Obtain activation input values ​​and weight input values, each having a first floating-point format, wherein the one or more computers are configured to perform computations on a neural network having multiple layers, wherein the activation input values ​​and weight input values ​​are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The activation input value is multiplied by the weight input value to generate a product value, the product value having a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; Obtain a partial sum value in a third floating-point format, wherein the third floating-point format differs from both the first and second floating-point formats and has higher precision than the first floating-point format; and The partial sum value is combined with the product value to generate an updated partial sum value with the third floating-point format.

17. The system according to claim 16, wherein: The second floating-point format has a dynamic range that is at least as large as that of the first floating-point format; and The third floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

18. A non-transitory computer storage medium encoded with instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform operations, said operations including: Obtain activation input values ​​and weight input values, each having a first floating-point format, wherein the one or more computers are configured to perform computations on a neural network having multiple layers, wherein the activation input values ​​and weight input values ​​are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The activation input value is multiplied by the weight input value to generate a product value, the product value having a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; Obtain a partial sum value in a third floating-point format, wherein the third floating-point format differs from both the first and second floating-point formats and has higher precision than the first floating-point format; and The partial sum value is combined with the product value to generate an updated partial sum value with the third floating-point format.

19. The computer storage medium according to claim 18, wherein: The second floating-point format has a dynamic range that is at least as large as that of the first floating-point format; and The third floating-point format has a dynamic range that is at least as large as that of the first floating-point format.

20. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform an operation, the operation comprising: The activation input value and the weight input value are obtained, both having a first floating-point format, wherein the one or more computers include hardware circuitry and the hardware circuitry is configured to perform computations on a neural network having multiple layers, wherein the activation input value and the weight input value are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The weight input value is stored in a weight register, which is configured to store values ​​having the first floating-point format; The activation input value is multiplied by the weight input value to generate a product value, the product value having a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; Obtain a partial sum value in a third floating-point format, wherein the third floating-point format is different from the first floating-point format and the second floating-point format and has higher precision than the first floating-point format; The partial sum is stored in a register, the sum in which is configured to store a value having the third floating-point format; and The partial sum is combined with the product value to generate an updated partial sum with the third floating-point format.

21. A non-transitory computer storage medium encoded with instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform operations, said operations including: The activation input value and the weight input value are obtained, both having a first floating-point format, wherein the one or more computers include hardware circuitry and the hardware circuitry is configured to perform computations on a neural network having multiple layers, wherein the activation input value and the weight input value are associated with one of the multiple layers; The first floating-point format is a 16-bit format, including: one available bit for the sign, eight available bits for the exponent, and seven available bits for the significant number, to represent the floating point in the first floating-point format; The weight input value is stored in a weight register, which is configured to store values ​​having the first floating-point format; The activation input value is multiplied by the weight input value to generate a product value, the product value having a second floating-point format, wherein the second floating-point format is different from the first floating-point format and has higher precision than the first floating-point format; Obtain a partial sum value in a third floating-point format, wherein the third floating-point format is different from the first floating-point format and the second floating-point format and has higher precision than the first floating-point format; The partial sum is stored in a register, the sum in which is configured to store a value having the third floating-point format; and The partial sum is combined with the product value to generate an updated partial sum with the third floating-point format.

Images