Propagation delay reduction

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for scheduling operations to reduce propagation delays between tiles of an accelerator. One of the methods includes receiving a request to generate a schedule for a first layer of a program to be executed by an accelerator configured to perform matrix operations at least partially in parallel, where the program defines a plurality of layers including the first layer, each layer of the program defining a matrix operation to be performed using a respective matrix of values. A plurality of scheduled initial blocks are allocated according to the initial allocation direction. The distribution direction is switched starting with a particular period such that the blocks processed after the selected particular period are processed along a different second dimension of the first matrix. And then distributing all the remaining undistributed blocks according to the switched distribution direction.

Description

[0001] This application is a divisional application of PCT patent application filed on August 20, 2020, with application number 202080047574.8 and title "Reduction of Propagation Delay". Technical Field

[0002] This manual relates to machine learning accelerators. Background Technology

[0003] A machine learning accelerator is an application-specific integrated circuit (ASIC) designed to perform highly parallel synchronous operations. Parallelism is achieved by integrating many different independent processing elements that can execute concurrently.

[0004] This device is ideal for accelerating inference via neural networks. A neural network is a machine learning model that uses multiple layers of operations to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between the input layer and the output layer. The output of each layer serves as the input to another layer in the network (e.g., the next hidden layer or the output layer).

[0005] Typically, the computational operations required for each layer can be implemented by performing matrix multiplication. Often, one of the matrices is a vector, for example, matrix multiplication by vector multiplication. Machine learning accelerators therefore allow the multiplication and addition of matrix multiplication to be performed with high parallelism.

[0006] However, due to the dependencies between layers in a neural network, there is an inherent latency in these computational mechanisms. This latency arises because the output of one layer becomes the input of the next. Therefore, the layers of a neural network typically must be executed sequentially, rather than in parallel. In other words, the last computational operation of one layer must usually be completed before the first computation of the next layer begins.

[0007] Two types of latency typically occur in machine learning accelerators that use multiple tiles assigned to different corresponding layers. First, computational latency occurs because the chip components wait for input data before they are actually available to perform computations. Second, propagation latency occurs because the output of one layer computed by one tile needs to be propagated to the input of another layer computed by a second tile. Computational latency can be improved by building larger devices with more computing elements. However, propagation latency tends to increase with device size because the distance the data needs to travel between tiles also increases. Summary of the Invention

[0008] This specification describes how the system generates schedules for machine learning accelerators that reduce computational latency between tiles and propagation latency within the machine learning accelerator.

[0009] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. The computational and propagation latency of machine learning accelerators can be reduced by modifying the scheduling of operations. This results in performance improvements without requiring expensive or complex hardware changes. The performance improvements of the scheduling techniques described below also provide computational advantages when only one tile exists, in which case some scheduling can achieve near 100% utilization despite inherent computational dependencies.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1A This demonstrates how changing the schedule can reduce latency between two layers in a neural network.

[0012] Figure 1B The scheduling allocation for a single tile is shown.

[0013] Figure 2 This is a flowchart of an example process for generating a schedule that reduces the latency between tiles of an accelerator.

[0014] Figure 3A It shows the execution of row priority and then switching to column priority.

[0015] Figure 3B This demonstrates how to use row limits to enforce row priority.

[0016] Figure 4 The diagonal scheduling is shown.

[0017] Figure 5 This is a schematic diagram illustrating an example of a dedicated logic circuit.

[0018] Figure 6 An example of a tile used in an ASIC chip is shown.

[0019] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0020] This specification describes techniques for scheduling tile operations to reduce propagation latency between tiles in multi-tile accelerators (e.g., machine learning accelerators).

[0021] In this specification, a tile refers to a device having an array of computing units capable of performing computations on a portion of a matrix. Therefore, a tile refers to any suitable accelerator configured to perform matrix-vector multiplication on a fixed-size block. Each unit may include circuitry that allows the unit to perform mathematical or other computations. In a typical scenario, a tile receives an input vector, multiplies the input vector by a weight matrix using the computing array, and generates an output vector.

[0022] In this specification, scheduling refers to the temporal sequence of portions of a matrix on which operations should be performed. These discrete portions of the matrix are also referred to as blocks. Therefore, scheduling specifies the ordering of blocks for a particular tile.

[0023] Each operation performed by a tile on a different block of a matrix can be called one iteration of the scheduling. If the matrix is ​​completely contained within the computational array of the tile, all matrix operations can be performed without any scheduling. However, when the matrix is ​​larger than the computational array, the system can generate a schedule specifying the order in which the different blocks of the matrix should be processed. For convenience, the scheduling operations in this specification will be referred to as allocation to a specific identifiable clock cycle. However, these clock cycles do not need to correspond to actual hardware clock cycles, and the same technique can be used to allocate computations to time periods comprising multiple hardware clock cycles.

[0024] Figure 1A This demonstrates how changing the schedule can reduce latency between two layers in a neural network. Figure 1A The left-hand side shows a simple and straightforward scheduling where two tiles are used to perform operations on two neural network layers. However, this simple and straightforward scheduling has limitations that can be addressed by using... Figure 1A Enhanced scheduling on the right side reduces latency.

[0025] The first layer 102 has a first weight matrix M1 110. The operation of the first layer 102 includes receiving an input vector V1 115 and multiplying the input vector V1 115 by the first weight matrix M1 110 to generate an output vector V2 117.

[0026] In this example, the first weight matrix 110 is larger than the computation array of the first tile assigned to perform the operations of the first layer 102. The first weight matrix 110 is twice the width and twice the height of the computation array of the first tile. Therefore, the operations of the first layer must be performed in multiple blocks over multiple clock cycles according to a specific schedule.

[0027] exist Figure 1AIn the example, the first scheduler 106 assigns a row-major schedule to the operation of the first layer 102. This means that the first tile assigned to the first layer 102 will operate on two iterations of the upper half of the first matrix 110, and then on two iterations of the lower half of the first matrix 110. Figure 1A In this diagram, the clock cycle allocation is shown on the corresponding matrix blocks. Therefore, for the first matrix 110 according to the first schedule, the first tile will process the upper half of the matrix sequentially on cycle 0 and cycle 1, and process the lower half of the matrix on cycle 2 and cycle 3.

[0028] The output vector 117 of the first layer 102 is then generated by summing the partial results from each iteration. Therefore, the first half of the output vector 117 comprises the summation of partial results from clock cycles 0 and 2. The second half of the output vector 117 comprises the summation of partial results from clock cycles 1 and 3.

[0029] The output vector 117 is then propagated via communication hardware to the second tile, which is assigned to perform matrix operations on the second layer 104 with a second weight matrix M2 120. In this example, it is assumed that the propagation delay of the accelerator is two clock cycles.

[0030] In this diagram, the second layer 104 also has a row priority schedule according to the first schedule 106.

[0031] The first and second tiles, respectively assigned to the first layer 102 and the second layer 104, can perform operations concurrently. However, computation between layers naturally introduces some data dependencies, and propagation delays introduce a delay that affects when operations in the second layer 104 can begin.

[0032] Specifically, the top-left block of the second matrix 120 can only be executed after both period 0 and period 2 have been executed in the first layer 102. Therefore, after period 2 of the first layer has been executed, periods 3 and 4 will be spent propagating the left half of the output vector 117 to compute the second tile of the second layer 104. Thus, the earliest point in time when the result of the second layer can be computed is in period 5.

[0033] For the same reason, the lower left block of the second matrix 120 in the second layer 104 cannot be executed until both cycle 1 and cycle 3 have been executed on the first layer 102 and until the data has been propagated, which causes a propagation delay of two cycles. Because cycle 6 has been assigned to the upper right block, the first scheduler 106 assigns the lower left portion of the second matrix 120 to be processed starting from cycle 7.

[0034] therefore, Figure 1A This illustrates how the first schedule 106 results in a total execution time of 8 cycles.

[0035] The second scheduler 108 adjusts the execution order of the first layer 102. The second scheduler 108 assigns column-major ordering to the first layer 102 instead of row-major ordering.

[0036] In other words, the first layer can first operate on the upper left part of the first matrix 110 in period 0, and then operate on the lower left part of the first matrix 110 in period 1.

[0037] Note that at this point, the operation of the second layer 104 can immediately begin processing the top left block of the second matrix 120. Therefore, after a two-cycle propagation delay in cycles 2 and 3, the top left block of the second matrix 120 can be processed in cycle 4, and the top right block of the second matrix 120 can be processed in cycle 5.

[0038] This rearrangement of the row / column ordering of operations in layer 102 reduces the total execution time of both layers to 7 cycles. In effect, by changing the row / column ordering in layer 102, the system is able to hide the propagation delay of a full cycle between two tiles assigned to operate on layers 1 and 2. While this is a simplified example, the time saving is still 12.5% ​​for a single pass through layers 102 and 104.

[0039] This technique can be generalized and refined into a problem of choosing two values: (1) a specific period M on which the assignment direction switching is performed, and (2) a specific period Ti on which the “lower left block” of the matrix is ​​processed. In this specification, the “lower left block” of the matrix refers to the last block of the matrix that needs to be processed before subsequent layers can begin processing the output generated by that layer. Thus, the “lower left block” can be any corner block of the matrix, or any edge block using the last reached portion of a row or column from the previous layer, depending on the specific arrangement in the schedule.

[0040] For an accelerator with a propagation delay of N cycles between layer n-1 and layer n, and an accelerator with a propagation delay of C cycles between layer n and layer n+1, the system can mitigate the propagation delay by scheduling the lower left block of the matrix of layer n to be processed for at least N cycles from the start of the layer and at least C cycles from the end of the layer.

[0041] Therefore, the enhanced scheduler switches the allocation direction after the selected period M. Typically, M specifies the period at or before a particular period Ti. At period M, the scheduler can switch from allocating blocks in row-priority order to allocating blocks in column-priority order, and vice versa. This is because after period Ti, the tiles continue to receive enough data to generate further output for the next layer. The techniques described below further illustrate how to change the row / column allocation direction of the scheduler to mitigate latency for matrices of arbitrary sizes.

[0042] The same switching in the assignment direction can also reduce latency in machine learning accelerators that have only one tile and little or no propagation latency. For example, suppose the device consists of only a single tile responsible for computing the results of two layers.

[0043] Figure 1B The scheduling allocation for a single tile is shown, which has nine computational elements for processing 4×4 matrices on each of the two layers.

[0044] The first schedule 107 illustrates basic row-major ordering. One potential problem is that some computational elements may be idle because they are waiting for the results of other computations to complete.

[0045] In cycle 0, all nine computational elements were successfully assigned to work on the first two rows of M1 111 and the first element of the third row of M1 111. However, in cycle 1 of the first schedule 107, only seven of the nine computational elements could be assigned work. This is because, when using row-first scheduling, the top-left corner of the second layer cannot be computed before processing the bottom-right corner of the first layer. Therefore, the first result of the second layer 104 cannot be computed until one cycle later.

[0046] Instead, consider using a second schedule 109 that switches the allocation direction. That is, after allocating the first row of matrix 111, the system can switch to column-priority allocation. Therefore, the lower left block of matrix 111 is computed on period 0 instead of period 1. Then, the second-level operations can begin immediately on period 1, since the lower left block has already been processed on period 0.

[0047] As a result, cycle 1 in the second schedule, which features switching in the allocation direction, achieves 100% utilization because some elements of the computation array can begin working on the second-layer operation without waiting for the first-layer operation to complete. The same technique can be used to improve utilization across the layers of a neural network.

[0048] Figure 2 This is a flowchart of an example process for generating a schedule to reduce latency for the accelerator. For convenience, this process will be described as being performed by a system of one or more computers located in one or more locations and appropriately programmed in accordance with this specification.

[0049] The system receives a request (210) to generate a schedule for a first layer having a first matrix. The first layer can be one of multiple layers defined by an input program specifying the operations to be performed on each layer. In a device with multiple tiles, each layer can be assigned to a corresponding tile of the device with multiple tiles. Each layer can have its own matrix. For example, the input program can specify the operations of a neural network architecture.

[0050] The system allocates multiple initial blocks (220) to the schedule based on the initial allocation direction in the first dimension. The allocation direction specifies the first dimension of the matrix, and the scheduling iterations should be performed along the first dimension. For example, the allocation direction can initially specify row-major or column-major order.

[0051] The system selects a period (230) for the lower left block. As mentioned above, Ti represents the period of the lower left block of the matrix to be executed. Also as mentioned above, the selection of Ti and the specific type of scheduling can determine M, which is the period for assigning direction switching.

[0052] Generally, regardless of the choice of Ti, a delay of Ti cycles can be hidden between layer i-1 and layer i, and a delay of Wi x Hi - Ti cycles can be hidden between layer i and layer i+1. In other words, the system can choose Ti to balance the delay of hiding the transition from i-1 to i with the delay of the transition from i to i+1.

[0053] Some matrices may be large enough that propagation delays can be completely hidden. Let Li denote the total end-layer latency at the end of layer i, which includes any termination computations or activation functions as well as propagation delays. To hide all the latency of layer i, the following inequality must hold:

[0054] Wi x Hi ≥ Li-1 + Li,

[0055] Here, Wi is the matrix width in blocks, and Hi is the matrix height in blocks. The block size can be determined by the tile hardware.

[0056] When the conditions are met, the system can select Ti as Li-1.

[0057] In other words, the system can schedule blocks so that the lower left block is executed as soon as possible after the previous layer has completed producing the output required to process that block.

[0058] However, not all matrices are large enough to completely hide the delay between layers. In these cases, scheduling can introduce idle periods to force waiting for results to be ready. If there are Si idle periods after layer i, then the following inequality holds for all valid scheduling of layer i:

[0059] Wi x Hi ≥ max(Li-1 – Si-1, 0) + max(Li – Si, 0)

[0060] If this inequality holds for efficient scheduling, then the system can allocate Ti according to the following formula:

[0061] Ti = max(Li⁻¹ - Si⁻¹, 0)

[0062] When using this arrangement for idle cycles, the system also programmatically selects the number of idle cycles through each layer to minimize the total latency introduced by the idle cycles. To this end, the system can perform an optimization process to select an integer number of idle cycles Sk for each layer k such that the following inequality holds:

[0063] Wi x Hi - max(Li - Si, 0) ≥ 0

[0064] and

[0065] Si-1 ≥ Li-1 + max(Li – Si, 0) - Wi x Hi

[0066] The system switches the allocation direction so that blocks processed after a specific block are processed sequentially along the second dimension (240). The choice of M (switching cycle) depends on the type of scheduling used. See below for reference. Figures 3A-3B A more detailed description of the example of choosing M

[0067] The system allocates all remaining unallocated blocks (250) according to the allocation direction after the switch. In other words, the system can allocate all unscheduled blocks according to the sorting based on the second dimension.

[0068] Figures 3A-4 An example scheduling using the switched allocation direction is shown. Figures 3A-3B In the diagram, numbered arrows indicate blocks of circuits designated for execution in a specific order.

[0069] Figure 3A This illustrates the execution of row-priority and then switching to column-priority. In other words, the system allocates blocks along the top row for processing first, then along the second row for processing next, and so on.

[0070] In this example, period M occurs somewhere in the middle of the fourth row of the block. Therefore, the system switches the allocation direction and begins allocating blocks in column priority order. The system can do this so that the bottom left corner of the scheduling matrix is ​​executed on the selected period Ti. In other words, the system calculates the row priority order until the number of untouched rows equals the difference between the current period and Ti.

[0071] Figure 3A The scheduling shown results in most of the computation being spent in the column-first phase. This tends to deliver the output at a very uniform rate and leaves some idle cycles at the end of each column. This can be advantageous, for example, when the output of each layer requires additional processing (as in the case of LSTM).

[0072] Figure 3B This illustrates the use of row constraints to perform priority ordering. In this example, the row-priority phase processes only a finite number of blocks before moving to the next row. In this example scheduling, the initial row contains more blocks than subsequent rows. In some implementations, the system calculates the row constraint by computing the value N = (Ti / Hi-1), where Hi is the number of blocks in each column of the matrix. The system can then use an upper bound on N for the initial row and a lower bound on N for subsequent rows.

[0073] Therefore, the period of the lower left block Ti in this example is given by two N values ​​and the number of rows in the matrix. In other words, if there are 8 rows in the matrix, floor(N) = 3, and ceiling(N) = 4, then Ti = 5 x 4 + 3 x 3 – (3-1) = 27. The switching period M in this case is given by M = 5x4 + 3x3 = 29.

[0074] Figure 3B The scheduling in the middle eliminates latency when processing the first few columns and reduces memory requirements. However, Figure 3B The scheduling implementation in this context may be more complex.

[0075] Figure 4 The diagram illustrates diagonal scheduling. As shown, during row priority, each row receives a decreasing number of blocks defined by the slope of the diagonal. In this example, the system selects Ti by calculating the number of blocks needed to fill the top-left diagonal, and the system can choose M = Ti.

[0076] Diagonal scheduling is symmetrical between the row-first and column-first phases, but it suffers from the disadvantages of the two scheduling methods mentioned above.

[0077] Figure 5This is a schematic diagram illustrating an example of a special-purpose logic circuit (specifically, an ASIC 500). The ASIC 500 includes multiple synchronous processors, which, for simplicity, will be referred to as tiles. For example, the ASIC 500 includes tile 502, wherein one or more tiles 502 include special-purpose circuitry configured to perform synchronous computations such as multiplication and addition operations. Specifically, each tile 502 may include a computational array of cells, wherein each cell is configured to perform mathematical operations (e.g., see...). Figure 6 The exemplary tile 500 shown and described herein. In some embodiments, the tiles 502 are arranged in a grid pattern, wherein the tiles 502 are arranged along a first dimension 501 (e.g., rows) and along a second dimension 503 (e.g., columns). For example, in Figure 5 In the example shown, tile 502 is divided into four distinct sections (510a, 510b, 510c, 510d), each section containing 288 tiles arranged in a grid of 18 tiles vertically and 16 tiles horizontally. In some embodiments, Figure 5 The ASIC 500 shown can be understood as comprising a single pulsed cell array subdivided / arranged into separate tiles, wherein each tile includes a subset / subarray of cells, local memory, and bus lines (see, for example, [reference needed]). Figure 6 ).

[0078] The ASIC 500 also includes a vector processing unit 504. The vector processing unit 504 includes circuitry configured to receive output from tile 502 and compute a vector computation output value based on the output received from tile 502. For example, in some embodiments, the vector processing unit 504 includes circuitry (e.g., multiplication circuitry, adder circuitry, shifter, and / or memory) configured to perform an accumulation operation on the output received from tile 502. Alternatively or additionally, the vector processing unit 504 includes circuitry configured to apply a nonlinear function to the output of tile 502. Alternatively or additionally, the vector processing unit 504 generates a normalized value, a pooled value, or both. The vector computation output of the vector processing unit can be stored in one or more tiles. For example, the vector computation output can be stored in memory uniquely associated with tile 502. Alternatively or additionally, the vector computation output of the vector processing unit 504 can be transmitted to circuitry outside the ASIC 500, for example, as the output of a computation. In some implementations, the vector processing unit 504 is segmented such that each segment includes circuitry configured to receive output from a set of corresponding tiles 502 and to compute a vector computation output based on the received output. For example, in Figure 5In the example shown, the vector processing unit 504 includes two rows spanning along the first dimension 501, each row comprising 32 segments 506 arranged in 32 columns. Each segment 506 includes circuitry (e.g., multiplication circuitry, adder circuitry, shifter, and / or memory) configured to perform vector computations based on outputs (e.g., cumulative sums) from the corresponding column of the tile 502, as explained herein. Figure 5 As shown, the vector processing unit 504 can be located in the middle of the grid of the tile 502. Other locations for the vector processing unit 504 are also possible.

[0079] The ASIC 500 also includes a communication interface 508 (e.g., interfaces 508a, 508b). The communication interface 508 includes one or more sets of serializer / deserializer (SerDes) interfaces and general purpose input / output (GPIO) interfaces. The SerDes interface is configured to receive instructions for the ASIC 500 (e.g., instructions for operating the controllable bus lines described below) and / or input data, and to output data from the ASIC 500 to external circuitry. For example, the SerDes interface can be configured to transmit instructions and / or input data at 32 Gbps, 56 Gbps, or any suitable data rate through the set of SerDes interfaces included within the communication interface 508. The GPIO interfaces are configured to provide interfaces for debugging and / or booting. For example, when the ASIC 500 is powered on, the ASIC 500 can run a bootloader. If the program fails, an administrator can use the GPIO interfaces to debug the source of the failure.

[0080] ASIC 500 also includes multiple controllable bus lines configured to transfer data between communication interface 508, vector processing unit 504, and multiple tiles 502 (see, for example, see...). Figure 6 The controllable bus routing includes, for example, wiring extending along a first dimension 501 (e.g., rows) and a second dimension 503 (e.g., columns) of the grid. A first subset of the controllable bus routing extending along the first dimension 501 can be configured to, in a first direction (e.g., towards...). Figure 5 Data is transmitted to the right side. A second subset of the controllable bus lines extending along the first dimension 501 can be configured to transmit data in the second direction (e.g., to the right side). Figure 5 Data is transmitted to the left side. A first subset of the controllable bus lines extending along the second dimension 503 can be configured to transmit data upwards (e.g., to the left side). Figure 5 Data is transmitted from the top. A second subset of the controllable bus lines extending along the second dimension 503 can be configured to transmit data in the fourth direction (e.g., to the top). Figure 5 Data is transmitted from the bottom.

[0081] Each controllable bus line includes multiple transmitter elements, such as flip-flops, for transmitting data along the line according to a clock signal. Transmitting data via the controllable bus line may include shifting data from a first transmitter element of the controllable bus line to a second adjacent transmitter element of the controllable bus line at each clock cycle. In some embodiments, data is transmitted via the controllable bus line at the rising or falling edge of a clock cycle. For example, data present on a first transmitter element (e.g., a flip-flop) of the controllable bus line at a first clock cycle may be transmitted to a second transmitter element (e.g., a flip-flop) of the controllable bus line at a second clock cycle. In some embodiments, the transmitter elements may be periodically spaced apart from each other at a fixed distance. For example, in some cases, each controllable bus line includes multiple transmitter elements, with each transmitter element located within or near a corresponding tile 502.

[0082] Each controllable bus line also includes multiple multiplexers and / or demultiplexers. The multiplexers / demultiplexers of the controllable bus line are configured to transmit data between the bus line and components of the ASIC chip 500. For example, the multiplexers / demultiplexers of the controllable bus line can be configured to transmit data to and / or from tile 502, transmit data to and / or from vector processing unit 504, or transmit data to and / or from communication interface 508. Transmitting data between tile 502, vector processing unit 504, and communication interface can include sending control signals to the multiplexers based on the desired data transmission to occur. These control signals can be stored in registers directly coupled to the multiplexers and / or demultiplexers. Then, the value of the control signal can be determined, for example, what data is transmitted from the source (e.g., the memory within tile 502 or vector processing unit 504) to the controllable bus line, or alternatively, what data is transmitted from the controllable bus line to the sink (e.g., the memory within tile 502 or vector processing unit 504).

[0083] The controllable bus lines are configured to be controlled at the local level, such that each tile, vector processing unit, and / or communication interface includes its own set of control elements for manipulating the controllable bus lines passing through that tile, vector processing unit, and / or communication interface. For example, each tile, 1D vector processing unit, and communication interface may include a corresponding set of transmitter elements, multiplexers, and / or demultiplexers for controlling data transmission to and from the tile, 1D vector processing unit, and communication interface.

[0084] To minimize latency associated with the operation of ASIC 500, tiles 502 and vector processing unit 504 can be positioned to reduce the distance data travels between the various components. In certain embodiments, both tiles 502 and communication interface 508 can be divided into multiple portions, with both the tile portions and the communication interface portions arranged such that the maximum distance data travels between the tiles and the communication interface is reduced. For example, in some embodiments, a first set of tiles 502 can be arranged in a first portion on a first side of the communication interface 508, and a second set of tiles 502 can be arranged in a second portion on a second side of the communication interface. As a result, the distance from the communication interface to the farthest tile can be halved compared to a configuration where all tiles 502 are arranged in a single portion on one side of the communication interface.

[0085] Alternatively, the tiles can be arranged in different numbers of sections (such as four sections). For example, in Figure 5 In the example shown, multiple tiles 502 of the ASIC 500 are arranged in multiple sections (510a, 510b, 510c, 510d). Each section includes a similar number of tiles 502 arranged in a grid pattern (e.g., each section may include 256 tiles arranged in 16 rows and 16 columns). The communication interface 508 is also divided into multiple sections: a first communication interface 508a and a second communication interface 508b are arranged on either side of the section of tiles 502. The first communication interface 508a can be coupled to the two tile sections 510a, 510c on the left side of the ASIC chip 500 via a controllable bus line. The second communication interface 508b can be coupled to the two tile sections 510b, 510d on the right side of the ASIC chip 500 via a controllable bus line. As a result, the maximum distance traveled by data to and / or from the communication interface 508 (and therefore the latency associated with data propagation) can be halved compared to an arrangement where only a single communication interface is available. Other coupling arrangements between tile 502 and communication interface 508 may also reduce data latency. The coupling arrangement between tile 502 and communication interface 508 can be programmed by providing control signals to the transmitter elements and multiplexers of the controllable bus line.

[0086] In some implementations, one or more tiles 502 are configured to initiate read and write operations with respect to controllable bus lines and / or other tiles within the ASIC 500 (referred to herein as "control tiles"). Remaining tiles within the ASIC 500 may be configured to perform computations (e.g., computational layer inference) based on input data. In some implementations, control tiles include the same components and configuration as other tiles within the ASIC 500. Control tiles may be added as one or more additional tiles, one or more additional rows, or one or more additional columns to the ASIC 500. For example, for a symmetric grid of tiles 502 (where each tile 502 is configured to perform computations on input data), an additional row or more rows of control tiles may be included to handle read and write operations on tiles 502 used to perform computations on the input data. For example, each section may include 18 rows of tiles, where the last two rows may include control tiles. In some implementations, providing separate control tiles increases the amount of available memory in the other tiles used to perform computations. However, separate tiles dedicated to providing the controls as described herein are not necessary, and in some cases, separate control tiles are not provided. Instead, each tile may store in its local memory instructions for initiating read and write operations for that tile.

[0087] In addition, although Figure 5 Each section shown comprises tiles arranged in 18 rows by 16 columns, but the number of tiles 502 and their arrangement within a section can vary. For example, in some cases, a section may comprise an equal number of rows and columns.

[0088] Furthermore, despite Figure 5 The tile 502 is shown as being divided into four parts, but it can be divided into other different groups. For example, in some implementations, tile 502 is divided into two distinct parts, such as a first part above vector processing unit 504 (e.g., closer to...). Figure 5 The second part (e.g., closer to the top of the page shown) and below the vector processing unit 504 (e.g., closer to the top of the page) and the second part below the vector processing unit 504. Figure 5 (See bottom of page). In this arrangement, each section can contain, for example, 576 tiles arranged as a grid of 18 tiles vertically (along direction 503) and 32 tiles horizontally (along direction 501). Sections can contain other total numbers of tiles and can be arranged in arrays of different sizes. In some cases, the division between sections is described by the hardware features of the ASIC 500. For example, as... Figure 5 As shown, portions 510a, 510b and portions 510c, 510d can be separated by vector processing unit 504.

[0089] Latency can also be reduced by centering the vector processing unit 504 relative to the tile portion. In some embodiments, the first half of the tile 502 is arranged on the first side of the vector processing unit 504, and the second half of the tile 502 is arranged on the second side of the vector processing unit 504.

[0090] For example, in Figure 5 In the illustrated ASIC chip 500, the vector processing unit 504 comprises two sections (e.g., two rows), each section including a plurality of segments 506 matching the number of columns of the tiles 502. Each segment 506 can be positioned and configured to receive output, such as a cumulative sum, from the corresponding column of the tile 502 within the section of the tile. Figure 5 In the example shown, tile portions 510a and 510b located on the first side of vector processing unit 504 (e.g., above vector processing unit 504) are coupled to the top row of segment 506 via a controllable bus line. Tile portions 510c and 510d located on the second side of vector processing unit 504 (e.g., below vector processing unit 504) are coupled to the bottom row of segment 506 via a controllable bus line. Furthermore, each tile 502 in the first half above processing unit 504 can be located at the same distance from vector processing unit 504 as the corresponding tile 502 in the second half below processing unit 504, such that there is no difference in total delay between the two halves. For example, tile 502 in row i of the first part 510a (where variable i corresponds to row position) can be located at the same distance from the distance vector processing unit 504 as tile 502 in row m-1-i of the second part of the tiles (e.g., part 510c) (where m represents the total number of rows in each part, and it is assumed that the rows increase in the same direction in both parts).

[0091] Compared to an arrangement where the vector processing unit 504 is located at the far end (e.g., bottom) of all tiles 502, configuring the tile sections in this way can halve the distance data travels to and / or from the vector processing unit 504 (and therefore the delay associated with data propagation). For example, the delay associated with receiving the accumulated sum from section 510a via a column of tiles 502 can be half the delay associated with receiving the accumulated sum from sections 510a and 510c via a column of tiles 502. The coupling arrangement of the tiles 502 and the vector processing unit 504 can be programmed by providing control signals to the transmitter elements and multiplexers of the controllable bus line.

[0092] During operation of the ASIC chip 500, activation inputs can be shifted between tiles. For example, activation inputs can be shifted along the first dimension 501. Furthermore, the output of computations performed on tile 502 (e.g., the output of computations performed by a computation array within tile 502) can be shifted between tiles along the second dimension 503.

[0093] In some implementations, the controllable bus lines can be physically hardwired to allow data to skip tiles 502, thereby reducing latency associated with the operation of the ASIC chip 500. For example, the output of a computation performed by the first tile 502 can be shifted along the second dimension 503 of the grid to a second tile 502 located at least one tile away from the first tile 502, thus skipping tiles in between. In another example, an active input from the first tile 502 can be shifted along the first dimension 501 of the grid to a second tile 502 located at least one tile away from the first tile 502, thus skipping tiles in between. By skipping at least one tile when shifting active input or output data, the overall data path length can be reduced, allowing data to be transmitted faster (e.g., eliminating the need to store data at skipped tiles using clock cycles) and reducing latency.

[0094] In an example implementation, each tile 502 within each column of portion 510a can be configured to pass output data toward vector processing unit 504 via a controllable bus line along the second dimension 503. Tiles 502 within each column can also be configured to pass data toward vector processing unit 504 by skipping the next adjacent tile (e.g., via a physical hardwired controllable bus line between tiles). That is, the tile 502 at position (i, j) = (0, 0) in the first portion 510a (where variable i corresponds to the row position and variable j corresponds to the column position) can be hardwired to pass output data to the tile 502 at position (i, j) = (2, 0); similarly, the tile 502 at position (i, j) = (2, 0) in the first portion 510a can be hardwired to pass output data to the tile 502 at position (i, j) = (4, 0), and so on. The last tile that is not skipped (e.g., tile 502 at position (i, j) = (16, 0)) passes the output data to vector processing unit 504. For sections with 18 rows of tiles, such as Figure 5 In the example shown, tile skipping ensures that all tiles within a section are skipped up to nine times by the distance vector processing unit 504, thereby improving the performance of the ASIC chip 500 by halving the data path length and the resulting data latency.

[0095] In another example implementation, each tile 502 within each row of segments 510a, 510c and each row of segments 510b, 510d can be configured to pass activation input along the first dimension 501 via a controllable bus line. For example, some tiles within portions 510a, 510b, 510c, 510d can be configured to pass activation input toward the center of grid 500 or toward communication interface 508. Tiles 502 within each row can also be configured to skip adjacent tiles, for example, by hard-wiring controllable bus lines between tiles. For example, tile 502 at position (i, j) = (0, 0) in the first part 510a (where variable i corresponds to the row position and variable j corresponds to the column position) can be configured to pass the activation input to tile 502 at position (i, j) = (0, 2); similarly, tile 502 at position (i, j) = (0, 2) in the first part 510a can be configured to pass the activation input to tile 502 at position (i, j) = (0, 4), and so on. In some cases, the last tile that is not skipped (e.g., tile 502 at position (i, j) = (0, 14) does not pass the activation input to another tile.

[0096] Similarly, skipped tiles can pass activation input in the opposite direction. For example, tile 502 at position (i, j) = (0, 15) in the first part 510a (where variable i corresponds to the row position and variable j corresponds to the column position) can be configured to pass activation input to tile 502 at position (i, j) = (0, 13); similarly, tile 502 at position (i, j) = (0, 13) in the first part 510a can be configured to pass activation input to tile 502 at position (i, j) = (0, 11), and so on. In some cases, the last tile that is not skipped (e.g., tile 502 at position (i, j) = (0, 1)) does not pass activation input to another tile. By skipping tiles, in some implementations, the performance of ASIC chip 500 can be improved by halving the data path length and the resulting data latency.

[0097] As explained herein, in some embodiments, one or more tiles 502 are dedicated to storing control information. That is, tiles 502 dedicated to storing control information do not participate in performing calculations on input data such as weight inputs and activation inputs. The control information may include, for example, control data for configuring a controllable bus line during operation of the ASIC chip 500, such that data can move around the ASIC chip 500. The control data may be provided to the controllable bus line in the form of control signals for controlling the transmitter elements and multiplexers of the controllable bus line. The control data specifies whether a particular transmitter element of the controllable bus line passes data to the next transmitter element of the controllable bus line, such that data is transferred between tiles according to a predetermined schedule. The control data further specifies whether data is transferred from or to a bus line. For example, the control data may include control signals instructing a multiplexer to transfer data from a bus line to memory and / or other circuitry within the tile. In another example, the control data may include control signals instructing a multiplexer to transfer data from memory and / or circuitry within the tile to the bus line. In another example, control data may include control signals that instruct the multiplexer to transmit data between the bus line and communication interface 508 and / or between the bus line and vector processing unit 504. Alternatively, as disclosed herein, dedicated control tiles are not used. Instead, in this case, the control information for that particular tile is stored in the local memory of each tile.

[0098] Figure 6 An example of a tile 600 used in an ASIC chip 500 is shown. Each tile 600 includes a local memory 602 and a computation array 604 coupled to the memory 602. The local memory 602 includes physical memory located in proximity to the computation array 604. The computation array 604 includes a plurality of cells 606. Each cell 606 of the computation array 604 includes circuitry configured to perform computations (e.g., multiplication and accumulation operations) based on data inputs to the cell 606 (such as activation inputs and weight inputs). Each cell can perform computations (e.g., multiplication and accumulation operations) on a clock signal of one cycle. The computation array 604 may have more rows than columns, more columns than rows, or an equal number of columns and rows. For example, in Figure 6 In the example shown, computation array 604 comprises 64 cells arranged in 8 rows and 8 columns. Other computation array sizes are also possible, such as computation arrays with 16 cells, 32 cells, 128 cells, or 256 cells. Each tile can include the same number of cells and / or the same size computation array. The total number of operations that can then be performed in parallel for the ASIC chip depends on the total number of tiles with the same size computation array within the chip. For example, for Figure 5The ASIC chip 500 shown contains approximately 1150 tiles, meaning it can perform approximately 72,000 computations in parallel per cycle. Examples of usable clock speeds include, but are not limited to, 225 MHz, 500 MHz, 750 MHz, 1 GHz, 1.25 GHz, 1.5 GHz, 1.75 GHz, or 2 GHz. Figure 6 As shown, the computational array 604 for each individual tile is a subset of the larger pulsating array of tiles.

[0099] The memory 602 contained in the tile 600 may include, for example, random access memory (RAM), such as SRAM. Each memory 602 may be configured to store and... Figure 5 The total memory associated with the n tiles 502 of the ASIC chip shown is 1 / n. The memory 602 can be provided as a single chip or multiple chips. For example, Figure 6 The memory 602 shown is provided as four single-port SRAMs, each coupled to the compute array 604. Alternatively, the memory 602 can be provided as two single-port SRAMs or eight single-port SRAMs, and other configurations. After error correction coding, the combined capacity of the memory can be, but is not limited to, 16 kB, 32 kB, 64 kB, or 128 kB. In some implementations, by providing the physical memory 602 locally to the compute array, the wiring density of the ASIC 500 can be significantly reduced. In alternative configurations where the memory is concentrated within the ASIC 500, in contrast to the local provision described herein, wiring for every bit of the memory bandwidth may be required. The total number of wiring required to cover each tile of the ASIC 500 would far exceed the available space within the ASIC 500. Conversely, by utilizing dedicated memory for each tile, the total number of wiring required across the area of ​​the ASIC 500 can be significantly reduced.

[0100] Tile 600 also includes controllable bus lines. These controllable bus lines can be categorized into several different groups. For example, the controllable bus lines may include a first group of general-purpose controllable bus lines 610 configured to transmit data between tiles in each main direction. That is, the first group of controllable bus lines 610 may include: bus line 610a, configured to run along the first dimension 501 of the tile grid towards the first direction (in... Figure 6 Data is transmitted in the direction referred to as "East"; bus line 610b is configured to move along the first dimension 101 of the tile grid towards the second direction (in the direction referred to as "East"). Figure 6 Data is transmitted in the second direction (referred to as "west"), where the second direction is opposite to the first direction; bus line 610c is configured to move along the second dimension 103 of the tile grid towards a third direction (in Figure 6Data is transmitted in the north (referred to as "north" in the text); and bus line 610d is configured to move along the second dimension 103 of the tile grid towards the fourth direction (in the north). Figure 6 Data is transmitted in the fourth direction (referred to as "South"), which is opposite to the third direction. The general-purpose bus line 610 can be configured to carry control data, activation input data, data from and / or to the communication interface, data from and / or to the vector processing unit, and data to be stored and / or used by the tile 600 (e.g., weight input). The tile 600 may include one or more control elements 621 (e.g., triggers and multiplexers) for controlling the controllable bus line and thus routing data to and / or from the tile 600 and / or memory 602.

[0101] The controllable bus lines may also include a second set of controllable bus lines, referred to herein as computation array partial summation bus line 620. Computation array partial summation bus line 620 can be configured to carry data output from computations performed by computation array 604. For example, bus line 620 can be configured to carry partial summation data obtained from rows in computation array 604, such as... Figure 6 As shown. In this case, the number of bus lines 620 will match the number of rows in array 604. For example, for an 8×8 computing array, there will be 8 partial sum bus lines 620, each bus line coupled to the output of the corresponding row in computing array 604. The computing array output bus lines 620 can also be configured to couple to another tile within the ASIC chip, for example, as input to the computing array of another tile within the ASIC chip. For example, the array partial sum bus line 620 of tile 600 can be configured to receive input (e.g., partial sum 620a) of the computing array of a second tile located at least one tile away from tile 600. The output of computing array 604 is then added to the partial sum line 620 to produce a new partial sum 620b, which can be output from tile 600. The partial sum 620b can then be passed to another tile, or alternatively, to a vector processing unit. For example, each bus line 620 can be coupled to a corresponding segment of the vector processing unit (such as...). Figure 5 (Section 506 in the middle).

[0102] For reference Figure 5 As explained, a controllable bus line may include circuitry such as a transmitter element (e.g., a trigger) configured to allow data to be transmitted along the bus line. In some implementations, each controllable bus line includes a corresponding transmitter element for each tile. See reference... Figure 5To further explain, the controllable bus line may include circuitry such as a multiplexer configured to allow data to be transmitted between communication interfaces of different tiles, vector processing units, and ASIC chips. The multiplexer can be located wherever there is a data source or data slot. For example, in some implementations, such as... Figure 6 As shown, control circuitry 621 (such as a multiplexer) can be located at the intersections of controllable bus lines (e.g., the intersection of general-purpose bus lines 610a and 610d, the intersection of general-purpose bus lines 610a and 610c, the intersection of general-purpose bus lines 610b and 610d, and / or the intersection of general-purpose bus lines 610b and 610c). The multiplexer at the bus line intersection can be configured to transmit data between the bus lines at the intersection. Accordingly, the direction of data travel on the controllable bus lines can be changed through appropriate operation of the multiplexer. For example, data traveling along the first dimension 101 on general-purpose bus line 610a can be transferred to general-purpose bus line 610d, causing the data to instead travel along the second dimension 103. In some embodiments, the multiplexer can be located near the memory 602 of tile 600, allowing data to be transferred to and / or from memory 602.

[0103] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by or control of the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver for execution by the data processing apparatus.

[0104] The term "data processing device" refers to data processing hardware and encompasses various means, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0105] A computer program (also referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer, or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0106] For a system of one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform those operations or actions. For one or more computer programs, being configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0107] As used herein, "engine" or "software engine" refers to an input / output system implemented in software that provides outputs other than the inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device comprising one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0108] The processes and logical flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. These processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0109] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or contained within special-purpose logic circuitry. Typically, a computer will also include or be operatively coupled to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, for receiving data from, sending data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0110] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and optical CD-ROMs and DVD-ROMs.

[0111] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse, trackball, or pressure-sensitive display or other surface) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and receiving response messages from the user in exchange.

[0112] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components (e.g., as a data server), or middleware components (e.g., an application server), or frontend components (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0113] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from it. Data generated at the user device (e.g., the result of user interaction) can be received from the device at the server.

[0114] In addition to the embodiments described above, the following embodiments are also innovative:

[0115] Example 1 is a method comprising:

[0116] Receive a request to generate a schedule for the first layer of a program to be executed by the accelerator, the accelerator being configured to perform matrix operations at least in partial parallelism, wherein the program definition includes multiple layers of the first layer, each layer of the program defining matrix operations to be performed using a corresponding value matrix;

[0117] Multiple initial blocks are allocated and scheduled according to the initial allocation direction, where the initial allocation direction specifies the first dimension of the first matrix for the first layer, and the multiple initial blocks will be executed along the first dimension;

[0118] Choose a specific period to process the last block of the matrix required before subsequent layers can begin processing;

[0119] Switching the allocation direction so that blocks processed after a selected specific period are processed along different second dimensions of the first matrix; and

[0120] All remaining unallocated blocks are allocated according to the switched allocation direction.

[0121] Example 2 is the method according to Example 1, wherein selecting a specific period includes:

[0122] Calculate the propagation delay of the previous layer; and

[0123] A specific period is allocated based on the propagation delay of the previous layer.

[0124] Example 3 is the method according to any one of Examples 1-2, wherein selecting a specific period includes:

[0125] Calculate the propagation delay of the previous layer;

[0126] Calculate the number of idle cycles in the previous layer; and

[0127] Choose the maximum value between the propagation delay of the previous layer and the number of idle cycles of the previous layer.

[0128] Example 4 is the method according to any one of Examples 1-3, wherein scheduling allocates a plurality of initial blocks in row priority order, and wherein all remaining unallocated blocks are allocated in column priority order.

[0129] Example 5 is the method according to Example 4, and further includes selecting a period for switching the allocation direction, including: selecting a period in which the number of unscheduled rows is equal to the difference between the current period and the selected specific period.

[0130] Example 6 is the method according to Example 4, wherein scheduling allocates multiple initial blocks only along a subset of rows of the matrix.

[0131] Example 7 is the method according to Example 6, wherein multiple initial partial rows and multiple subsequent partial rows are scheduled and allocated, wherein the subsequent partial rows are smaller than the initial partial rows.

[0132] Example 8 is the method according to Example 7, wherein the initial partial rows have a length given by ceiling(N), and the subsequent partial rows have a length given by floor(N), where N is given by dividing the selected period by the block height of the matrix on the previous layer.

[0133] Example 9 is the method according to Example 4, wherein the scheduling allocates initial blocks in row priority order to fill the space defined by the diagonal in the matrix.

[0134] Example 10 is the method according to Example 9, wherein the switching of the allocation direction occurs during a selected specific period.

[0135] Example 11 is the method according to any one of Examples 1-10, wherein the accelerator has multiple tiles, and each layer is calculated by a corresponding tile from the multiple tiles.

[0136] Example 12 is the method according to any of Examples 1-10, wherein the accelerator has a single tile to perform two-layer operations.

[0137] Example 13 is a system comprising: one or more computers and one or more storage devices storing instructions operable when executed by the one or more computers to cause the one or more computers to perform the method according to any one of Examples 1 to 12.

[0138] Example 14 is a computer storage medium encoded with a computer program, the program including instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method described according to any one of Examples 1 to 12.

[0139] Although this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from that combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0140] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all of the shown operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0141] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions listed in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the figures do not necessarily require the specific order or sequence shown to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed using an integrated circuit, the method comprising: The scheduling of the processing order of different blocks of a specified matrix; Based on the allocation direction and the matrix operations performed on the first tile of the circuit, for the first layer of the neural network, blocks of the first matrix are processed on the first clock cycle sequence. The allocation direction is switched based on a specific clock cycle selected from the first sequence to perform the allocation direction switch; as well as In response to switching the allocation direction, on the second tile of the circuit, a second matrix for the second layer of the neural network is processed simultaneously with the first matrix.

2. The method according to claim 1, wherein, The allocation direction specifies the direction in which the blocks of the first matrix are processed, and switching the allocation direction includes: Switch the allocation direction from row priority sorting to column priority sorting.

3. The method according to claim 1, wherein, Processing the second matrix of the second layer simultaneously with the first matrix of the first layer includes: The second matrix of the second layer is processed using the output vector calculated by the matrix operation performed on a first subset of the first matrix at the first tile.

4. The method according to claim 3, wherein, A portion of the second matrix is ​​processed simultaneously with a portion of the matrix operation performed on the first layer at the first tile.

5. The method according to claim 1, wherein, The scheduling corresponds to the timing sequence of a portion of the matrix on which a specific block of the integrated circuit operates.

6. The method according to claim 1, wherein, The scheduling specifies the ordering of multiple blocks of a particular tile of the integrated circuit.

7. The method according to claim 1, wherein, Each time a tile of the integrated circuit operates on a different block of the matrix, it corresponds to at least one iteration of the scheduling.

8. The method according to claim 1, wherein, The scheduled matrix operations are assigned to specific, identifiable clock cycles that can be distinguished from the actual hardware clock.

9. The method according to claim 1, wherein: i) The processing of different blocks corresponds to matrix operations performed at one or more tiles of the integrated circuit; ii) The neural network comprises multiple neural network layers; as well as iii) Each tile is assigned to operate on a specific neural network layer among the multiple neural network layers.

10. A hardware integrated circuit configured to perform the following operations: The scheduling of the processing order of different blocks of a specified matrix; Based on the allocation direction and the matrix operations performed on the first tile of the circuit, for the first layer of the neural network, blocks of the first matrix are processed on the first clock cycle sequence. The allocation direction is switched according to the specific clock cycle selected in the first sequence for performing the allocation direction switch; as well as In response to switching the allocation direction, at the second tile of the circuit, a second matrix for the second layer of the neural network is processed simultaneously with the first matrix.

Images