Sharding for Synchronous Processors

Abstract

To increase the performance of a device having many synchronous tiles by using a sharding technique.SOLUTION: A method according to one aspect of the present invention includes a step of receiving a representation of a dataflow graph including a plurality of nodes that each represent respective matrix operations to be performed by a device having a plurality synchronous tiles. Candidate allocations of respective portions of the dataflow graph to each tile of the plurality of synchronous tiles are evaluated according to one or more resource constraints of the device. One of the candidate allocations is selected based on the evaluation of each candidate allocation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This specification relates to compiler techniques for integrated circuit accelerators. [Background technology]

[0002] A synchronous integrated circuit accelerator is an application-specific integrated circuit (ASIC) designed to perform highly parallel synchronous operations. Parallelism is achieved by integrating many different independent processing elements that can execute simultaneously.

[0003] Such devices are well suited for accelerating inference paths, for example, through neural networks in which each independent processing element performs a different multiplication or addition of layer inputs with weights. A neural network is a machine learning model that uses multiple layers of operation to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between the input and output layers. The output of each layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Summary of the Invention [Means for solving the problem]

[0004] An accelerator can use multiple independent processing tiles. As used herein, a tile refers to any suitable device having an array of computational cells that can perform computations on portions of a matrix at least partially in parallel. Thus, each cell contains circuitry that enables the cell to perform mathematical or other computations. In a typical scenario, a tile receives an input vector and uses the computational array to multiply the input vector by a matrix of weights to generate an output vector.

[0005] As used herein, synchronous tiles mean that the operations performed by the computational arrays do not undergo branched execution, similar to if / else statements in an imperative program. Rather, operations may be either partially or fully scheduled in advance. For example, the operations of some synchronous processors may be scheduled down to the individual cycle level, meaning that for every operation of every processing element, the computational array is assigned to a specific slot in a series of execution cycles.

[0006] This document describes techniques for compiling programs written for systems with multiple synchronous processors. As noted above, for simplicity, such devices are referred to herein as tiles.

[0007] A compiler can exploit cycle-level precision with which operations of a program running across multiple synchronous processors can be scheduled. This enables fine-grained sharding of operations, as well as other performance enhancements. The techniques described herein are particularly useful for scheduling operations in systems that operate multiple tiles in a linear array, where each tile communicates only with its neighbors in the linear array. However, the same techniques can also be used in systems lacking this hardware configuration. [Effects of the Invention]

[0008] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: A compiler can utilize various sharding techniques to increase performance of devices with a large number of compute tiles. Execution of tasks on such devices can conform to device specifications / constraints using sharding techniques. A compiler can use sublayer sharding to increase device utilization. A compiler can also use parallelogram time slicing to reduce or eliminate idle zones where tiles do no work because they are waiting for something to occur. A compiler can also use oversharding to improve load balancing between tiles and address load balancing situations that are unpredictable at compile time. A compiler can also use zigzag sharding to balance memory and computation requirements between tiles.

[0009] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a flowchart of an exemplary method for sharding matrix operations for a device having multiple tiles. [Figure 2A] FIG. 10 is a diagram illustrating an example of sublayer sharding. [Figure 2B] FIG. 10 is a diagram illustrating an example of sublayer sharding. [Figure 3A] FIG. 10 is a diagram illustrating an example of parallelogram time slicing. [Figure 3B] FIG. 10 is a diagram illustrating an example of parallelogram time slicing. [Figure 3C] FIG. 10 is a diagram illustrating an example of parallelogram time slicing. [Figure 3D]FIG. 10 is a diagram illustrating an example of parallelogram time slicing. [Figure 4A] FIG. 10 is a diagram illustrating an example of overshading. [Figure 4B] FIG. 10 is a diagram illustrating an example of overshading. [Figure 4C] FIG. 10 is a diagram illustrating an example of overshading. [Figure 5A] FIG. 10 illustrates an example of another version of oversharding called zigzag sharding. [Figure 5B] FIG. 10 illustrates an example of another version of oversharding called zigzag sharding. [Figure 5C] FIG. 10 illustrates an example of another version of oversharding called zigzag sharding. [Figure 6] FIG. 1 illustrates an example of tiles that can be used to implement processing in a system with multiple synchronized processors. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference numbers and designations in the various drawings indicate like elements.

[0012] 1 is a flowchart of an exemplary method for sharding matrix operations of a device having a large number of tiles. As described above, in a target device, tiles can be arranged in or configured to operate as a linear array, with each tile communicating only with its neighbors. Thus, in response to receiving a dataflow graph, the system can generate assignments between nodes of the graph and tiles that take into account the ordering of the tiles in the linear array. For convenience, this exemplary method will be described as being performed by a system of one or more computers located in one or more locations and appropriately programmed in accordance with this specification.

[0013] The system receives 110 a representation of a dataflow graph that specifies matrix operations to be performed by a device having multiple tiles. In this context, matrix operations should be understood to include matrix operations or vector operations. Each node in the dataflow graph specifies a matrix operation to be performed on a matrix dedicated to that node. For example, each node in the graph can represent a layer of a neural network, with the matrix storing the weights to be used and learned for that layer. Each edge in the graph represents the propagation of data, with the output of one node becoming the input of the next node. The system generates 120 multiple candidate assignments of portions of the dataflow graph to each tile of the multiple tiles. The sharding problem can be formulated as a search problem to find the best assignment of portions of the dataflow graph to tiles on the device from among the multiple candidate assignments. Various techniques involved in generating the candidates to be evaluated are described in more detail below. Which candidate assignment is actually best depends heavily on the dataflow graph and the particular application.

[0014] The system evaluates each candidate allocation according to one or more resource constraints of the device 130. Important resource constraints include memory constraints and bandwidth constraints.

[0015] One exemplary memory constraint for a machine learning accelerator is that all weights for all matrices assigned to a particular tile must fit within static random access memory (SRAM) local to that tile. In other words, each tile may have an integrated amount of resident SRAM, which becomes a memory constraint when sharding. The reasons for this constraint involve speed and bandwidth. First, machine learning accelerators are designed to perform an inference pass through a model as fast as possible. For some applications, dynamic random access memory (DRAM) is not fast enough. Additionally, matrix weights can be large and therefore difficult to move around. Therefore, when all weights for all matrices assigned to a particular tile can fit within the tile's local SRAM, the tile does not need to spend time reading and writing weights during an inference pass or training update.

[0016] Another exemplary memory constraint is a bandwidth constraint, which specifies how quickly data must move between nodes in a graph. If a proposed allocation were to exceed the bandwidth threshold of a particular edge in the graph, the system could determine that the bandwidth constraint has been violated.

[0017] The system selects one of the candidate assignments based on an evaluation of each candidate assignment (140). As one example, the system can perform layer sharding, where different matrices from different layers of the input program are assigned to different respective tiles. Thus, for example, in a neural network with eight layers, its weight matrices can be assigned to each of the eight tiles.

[0018] For candidate assignments that satisfy one or more resource constraints, the system can determine which assignment best optimizes tile utilization on the device. Generally, this involves computing the maximum amount of compute time across all tiles, and then selecting the candidate assignment with the smallest maximum amount of compute time. For devices with synchronous tiles that can be scheduled in advance down to the cycle level, these calculations can be much more accurate than compiler estimates for other types of devices.

[0019] After selecting a candidate assignment, the system can further reduce the dataflow graph program by generating hardware instructions to be executed by the tiles of the device. The system can then load the hardware instructions along with the matrix weights onto the device and execute the hardware instructions on the device to perform the operations represented by the dataflow graph.

[0020] 2A-2B illustrate examples of sublayer sharding. Sublayer sharding refers to partitioning one matrix of a particular node in a dataflow graph and assigning different portions to different tiles. FIG. 2A illustrates one exemplary candidate partition in which elements of matrix 200 are partitioned along a diagonal. FIG. 2B illustrates another exemplary candidate partition in which elements of matrix 200 are partitioned between columns. Alternatively or additionally, matrix 200 can also be partitioned between rows. In some implementations, the system uses sublayer sharding to generate candidate assignments whenever the matrix exceeds a size threshold.

[0021] Generating candidate assignments for sublayer sharding requires adding some additional complexity to the dataflow graph. Specifically, the vector inputs of each layer often must be broadcast to every tile that receives a portion of the sublayer sharding result. In addition, partial sums or partial activations generated by individual tiles must be merged into a final sum or activations at a later stage, either in the same tile or in a different tile. The compiler can take these additional complexities into account when evaluating candidate assignments involving sublayer sharding.

[0022] 3A-3C show examples of parallelogram time slicing. Parallelogram time slicing can be advantageous when tiles perform certain operations in different passes with different execution gradients. The execution gradient indicates how quickly a tile can complete all operations in a sequence. Such applications are common in neural networks, where, for example, a single pass through all tiles can represent one inference path through the network.

[0023] 3A illustrates conflicts resulting from different execution gradients. In FIG. 3A, the horizontal axis represents time. The tiles in FIG. 3A are arranged in a linear array structure, such that each tile can only provide data to and receive data from its two immediate neighbors. In some implementations, the linear array structure is convoluted, such as a ring structure, such that the last tile can communicate with the first tile and vice versa.

[0024] Therefore, tiles first perform Pass A sequentially, starting with Tile 0 and continuing through Tile 7. Pass A has a common property of machine learning models in that each tile cannot begin execution until it receives output from the previous tile. Therefore, Tile 1 must wait for Tile 0 to finish, Tile 2 must wait for Tile 1 to finish, and so on.

[0025] The dashed lines in Figure 3A show how this common pattern forms the parallelogram of Pass A.

[0026] 3A also shows a second pass, Pass B. The operations of Pass B take less time than Pass A, and therefore Pass B has a steeper execution slope than Pass A.

[0027] In this example, the different execution slopes of Pass A and Pass B cause a collision between Tile5 and Tile6. This collision occurs because by the time Tile5 finishes Pass B, Tile6 has not yet finished its portion of Pass A. Therefore, stalls 301 and 302 must be introduced either by the compiler or at runtime, causing Tile5 to wait to provide the result of Pass B until Tile6 has finished the operation of Pass A.

[0028] FIG. 3B shows an alternative strategy for managing different execution slopes, which is to introduce a single stall 303 before Pass B has even started, so that the last operation of Pass B in Tile 7 can execute immediately after the last operation of Pass A.

[0029] Both of these strategies result in idle zones 310 and 320 where tiles are idle and doing no work. Idle zone 310 is smaller than idle zone 320, but by introducing a large number of stalls, it adds compile-time or run-time control complexity not present in the strategy in Figure 3B.

[0030] Figure 3C shows how differences in execution slopes can actually introduce additional idle zones. In Figure 3C, the steeper execution slope of Pass B results in a first idle zone 330. However, if Pass A were to be executed again after Pass B, this would introduce a second idle zone 340.

[0031] Figure 3D shows how parallelogram time slicing is used to manage different execution slopes. To reduce idle zones introduced by different execution slopes, the compiler can generate a schedule that effectively forces all paths to have the same execution slope. Thus, different paths effectively fit together, eliminating idle zones.

[0032] A compiler can use stalls or buffering to force a more gradual execution slope. For example, as shown in FIG. 3D, the execution slope of Pass B is reduced by introducing buffering between different tiles. In other words, after Tile 0 completes Pass B, the result is stored in buffer 360. Tile 0 can then immediately start processing the second iteration of Pass A. Alternatively or additionally, a compiler may insert a stall between the end of Pass B and the start of the second iteration of Pass A. Note that while this solution introduces more stalls than the approach shown in FIG. 3A, the overall idle zone is substantially smaller.

[0033] Enforcing a gradual execution slope introduces additional control complexity that the compiler can take into account when evaluating candidate assignments. Specifically, enforcing a gradual execution slope introduces timing constraints for certain paths. For example, in Figure 3D, the timing constraint is that Pass B in Tile1 cannot start immediately after Pass B in Tile0.

[0034] 4A-4C show examples of oversharding. Oversharding refers to a technique for sharding a matrix more finely than the hardware requires. In other words, sometimes a computation can be performed by a single tile, but the matrix is ​​sharded in some way for other reasons. These other reasons are often related to load balancing.

[0035] 4A shows the execution of a single Pass A with nine layers across eight tiles. Thus, the ninth layer in this example is fully allocated to Tile 0. Tile 0 has the availability and memory capacity to fully perform all operations for the ninth layer, which leaves the remaining tiles completely unoccupied during this time.

[0036] 4B illustrates the oversharding of the ninth layer of Pass A. In other words, even if a single tile could perform all of the operations of the ninth layer, the ninth layer is oversharded to distribute it across all eight of the other tiles. There may or may not be intra-layer data dependencies, so different parts of the ninth layer may execute partially in parallel or fully in parallel.

[0037] Therefore, device load balancing is better than if only one tile was doing the 9th layer, which can be especially advantageous if the device is also doing other computations as well.

[0038] FIG. 4C shows another application of oversharding to address runtime load balancing. In some applications, the runtime load on tiles is not known in advance. One such example is that a device may be used to perform automatic machine transformations using several different models trained for different transformations. For example, as shown on the left side of FIG. 4C, Tile 0 may be assigned to run Model A, Tile 1 may be assigned to run Model B, and so on. However, the actual order in which these models are used is not known at compile time, especially when the device is used in a backend server servicing inbound Internet traffic.

[0039] Therefore, it may be the case that Tile0 handles most of the requests, and the other tiles are only used rarely.

[0040] To achieve some runtime load balancing, the compiler can overshard the models and distribute all models across all chips, as shown on the right side of Figure 4C.

[0041] Then, at run time, the utilization of all tiles can be made substantially equal regardless of the actual distribution of run time load.

[0042] 5A-5C show examples of another version of oversharding, called zigzag sharding.,In many machine learning applications, including neural networks, a common pattern,emerges where the compute complexity of a layer decreases, but,its memory requirements increase.

[0043] Thus, a memory pattern may result in each subsequent tile requiring more and more memory, as shown in Figure 5A, and a computational pattern requiring fewer and fewer computational resources, as shown in Figure 5B.

[0044] The compiler can use zigzag sharding to ensure that all tiles have a better balance between memory allocation and computational allocation. Zigzag sharding essentially pairs different complementary layers together so that they are performed by the same tile. So, for example, zigzag sharding can ensure that the same tile performs the first and eighth layers, the second and seventh layers, the third and sixth layers, and the fourth and fifth layers. This allocation is shown in Figure 5C, where the computational pattern takes on a zigzag shape.

[0045] As shown in Figure 5C, the first layer, Layer 1, is over-sharded and distributed across tiles 0-3 to divide the computational complexity of the first layer. In a similar but complementary manner, the eighth layer is also partitioned and distributed across the same tiles 0-3 to divide the memory requirements of the eighth layer.

[0046] Similarly, the second layer is always assigned to the same tile as the seventh layer, the third layer is always assigned to the same tile as the sixth layer, and the fourth layer is always assigned to the same tile as the fifth layer.

[0047] Each of the example sharding techniques described above in connection with Figures 2-4 may be used either alone or in combination with any one or more of the other sharding techniques.

[0048] 6 shows an example of a tile 600 that may be used to implement processing in a system with multiple synchronized processors. For example, the tile 600 may be used as one tile in a linear tile array or in a system in which the tiles operate as a linear tile array.

[0049] Each tile 600 includes a local memory 602 and a computing array 604 coupled to the memory 602. The local memory 602 includes physical memory positioned proximate to the computing array 604. The computing array 604 includes a number of cells 606. Each cell 606 of the computing array 604 includes circuitry configured to perform a computing calculation (e.g., multiplication and accumulation operations) based on data inputs, such as activation inputs and weight inputs to the cell 606. Each cell can perform a computing calculation (e.g., multiplication and accumulation operations) on a cycle of a clock signal. The computing array 604 may have more rows than columns, more columns than rows, or an equal number of columns and rows. For example, in the example shown in FIG. 6, the computing array 604 includes 64 cells organized in 8 rows and 8 columns. Other computing array sizes are possible, such as computing arrays having 16 cells, 32 cells, 128 cells, or 256 cells, among others. Each tile may contain the same number of cells and / or the same size of a computational array.

[0050] The memory 602 included in the tile 600 may include, for example, a random access memory (RAM), such as an SRAM. Each memory 602 may be configured to store 1 / nth of the total memory associated with the n tiles 502 of a system having multiple tiles. The memory 602 may be implemented as a single chip or multiple chips. For example, the memory 602 shown in FIG. 6 is implemented as four single-port SRAMs, each coupled to a computational array 604. Alternatively, the memory 602 may be implemented as two single-port SRAMs or eight single-port SRAMs, among other configurations. The combined memory capacity may be, for example, but not limited to, 16 kB, 32 kB, 64 kB, or 128 kB after error correction coding. By locating the physical memory 602 local to the computational array, wiring density may be significantly reduced.

[0051] The tile 600 also includes controllable bus lines, which may be organized into a number of different groups. For example, the controllable bus lines may include a first group of general-purpose controllable bus lines 610 configured to transfer data between tiles for each cardinal direction. That is, the first group of controllable bus lines 610 includes bus lines 610a configured to transfer data toward a first orientation along the first dimension 101 of the grid tiles (called “East” in FIG. 6 ), bus lines 610b configured to transfer data toward a second orientation along the first dimension 101 of the grid tiles that is opposite to the first orientation (called “West” in FIG. 6 ), bus lines 610c configured to transfer data toward a third orientation along the second dimension 103 of the grid tiles (called “North” in FIG. 6 ), and bus lines 610d configured to transfer data toward a fourth orientation along the second dimension 103 of the grid tiles that is opposite to the third orientation (called “South” in FIG. 6 ). The general-purpose bus lines 610 may be configured to carry control data, activation input data, data from and / or to the communication interface, data from and / or to the vector processing unit, and data (e.g., weight inputs) to be stored and / or used by the tile 600. The tile 600 may include one or more control elements 621 (e.g., flip-flops and multiplexers) for controlling the controllable bus lines and thus for routing data to and / or from the tile 600 and / or from the memory 602.

[0052] The controllable bus lines may also include a second group of controllable bus lines, referred to herein as computational array partial sum bus lines 620. The computational array partial sum bus lines 620 may be configured to carry data output from computations performed by the computational array 604. For example, the bus lines 620 may be configured to carry partial sum data obtained from rows in the computational array 604, as shown in FIG. 6. In such a case, the number of bus lines 620 would match the number of rows in the array 604. By way of example, for an 8×8 computational array, there would be eight partial sum bus lines 620, each coupled to an output of a corresponding row in the computational array 604. The computational array output bus lines 620 may be further configured to couple to another tile in the system, for example, as an input to the computational array of an adjacent tile in a linear array of tiles. For example, the array partial sum bus line 620 of tile 600 may be configured to accept an input (e.g., partial sum 620a) of a computing array of a second tile located at least one tile away from tile 600. The output of computing array 604 may then be added to partial sum bus line 620 to create a new partial sum 620b, which may be output from tile 600. Partial sum 620b may then be passed to another tile or, alternatively, to a vector processing unit. For example, each bus line 620 may be coupled to a corresponding section of a vector processing unit (such as section 506 in FIG. 5).

[0053] Embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or a combination of one or more of these, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded in a tangible, non-transitory storage medium, for execution by or to control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information to be transmitted to a receiver device suitable for execution by the data processing apparatus.

[0054] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can optionally include code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0055] A computer program, also referred to or written as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., in one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0056] When a system of one or more computers is configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0057] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output distinct from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (SDK), an object, etc. Each engine can be implemented in any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smartphone, or other fixed or portable device, that includes one or more processors and computer-readable media. In addition, two or more of the engines may be implemented in the same computing device or in different computing devices.

[0058] The methods and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The methods and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0059] A computer suitable for running a computer program can be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated within, special-purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from, transfer data to, or both of, one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data. However, a computer need not have such devices. Moreover, a computer may be embedded within another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0060] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0061] To provide for user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse, trackball, or presence-sensitive display or other surface, through which the user can provide input to the computer. Other types of devices can be used to provide for user interaction as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual, auditory, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0062] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0063] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server transmits data, e.g., HTML pages, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., results of user interaction, can be received from the device at the server.

[0064] In addition to the above-mentioned embodiments, the following embodiments are also innovative: Embodiment 1 is receiving a representation of a data flow graph including a plurality of nodes, each representing a respective matrix operation to be performed by a device having a plurality of synchronization tiles; generating a plurality of candidate assignments of respective portions of the dataflow graph to each tile of the plurality of synchronization tiles; evaluating each candidate allocation of the plurality of candidate allocations according to one or more resource constraints of the device; selecting one of the candidate assignments based on an evaluation of each candidate assignment; The method includes:

[0065] Embodiment 2 is the method of embodiment 1, wherein each tile of the plurality of synchronization tiles comprises a computing array of independent processing elements.

[0066] Embodiment 3 is the method of embodiment 1 or 2, wherein generating the plurality of candidate allocations according to one or more resource constraints includes generating the candidate allocations according to an SRAM memory constraint, the SRAM memory constraint specifying that all weights of the associated operations must fit within a local SRAM that is local to the corresponding synchronization tile.

[0067] Example 4 is the method of example 3, wherein each local SRAM is a resident memory integrated in the same synchronous tile in which the operation is to be performed.

[0068] Embodiment 5 is the method according to any one of embodiments 1 to 4, wherein generating the plurality of candidate allocations comprises allocating matrices of different layers to different tiles.

[0069] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the step of generating the plurality of candidate allocations includes a step of performing sublayer sharding on a matrix for a particular node in the program, including a step of allocating different blocks of the matrix to different respective synchronization tiles.

[0070] Embodiment 7 is a method according to embodiment 6, in which the step of assigning different blocks of the matrix to each different synchronization tile includes dividing the matrix along rows or columns, or dividing the matrix diagonally.

[0071] Embodiment 8 is the method of embodiment 7, wherein the operations further include performing sublayer sharding on any matrix that exceeds a size threshold.

[0072] Embodiment 9 is a method according to any one of embodiments 1 to 8, wherein the step of generating the plurality of candidate allocations includes a step of performing parallelogram time slicing so that different paths that initially have different execution slopes have substantially similar execution slopes.

[0073] In a tenth embodiment, the step of performing parallelogram time slicing includes: computing a respective running slope for each of two different paths; determining that the first pass has a smaller execution slope than the second pass; in response, modifying the candidate assignments so that the first pass has a gentler performance slope; 10. The method of embodiment 9, comprising:

[0074]

[0023] Embodiment 11 is the method of embodiment 10, wherein modifying the candidate allocation comprises inserting buffering or stalls into the candidate allocation.

[0075] Embodiment 12 is a method according to any one of embodiments 1 to 11, wherein the step of generating the plurality of candidate assignments includes a step of oversharding such that nodes of the data flow graph that can be entirely performed by one of the tiles are divided among multiple different tiles.

[0076] Embodiment 13 is the method of embodiment 12, wherein the step of oversharding includes the step of assigning one respective layer to each tile and dividing the last layer among all the tiles.

[0077] Embodiment 14 is the method of embodiment 12, wherein the step of oversharding includes dividing a plurality of matrices across all the tiles.

[0078] Embodiment 15 is a method according to any one of embodiments 1 to 14, wherein the step of generating the plurality of candidate allocations includes a step of performing zigzag sharding by pairing together different layers having different memory and computational requirements.

[0079] Embodiment 16 is the method of embodiment 15, wherein pairing different layers together includes pairing a first layer with a last layer so that the same tile performs both the first layer and the last layer.

[0080] Embodiment 17 is the method of embodiment 16, wherein the first layer has the greatest computational requirements and the last layer has the greatest memory requirements.

[0081] Embodiment 18 is a method according to any one of embodiments 1 to 17, wherein the tiles are arranged in a linear array such that each tile provides data to only one or two adjacent tiles.

[0082] Embodiment 19 is a system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform the method described in any one of embodiments 1 to 18.

[0083] Embodiment 20 is a computer storage medium encoded with a computer program, the program including instructions operable, when executed by a data processing device, to cause the data processing device to perform a method according to any one of embodiments 1 to 18.

[0084] While the specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, while features may be described above as functioning in a particular combination, and even initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0085] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0086] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the methods depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0087] 101 First Dimension 103 Second Dimension 200 Matrix 301, 302, 303 stalls 310, 320, 330, 340 Idol Zone 360 buffer 502 tiles 506 classification 600 tiles 602 local memory, physical memory 604 Computer Computational Array 606 cells 610 General Purpose Controllable Bus Line 610a, 610b, 610c, 610d bus lines 620 Computer Array Partial Sum Bus Line 620a, 620b partial sum 621 Control Elements

Claims

1. 1. A method comprising: receiving a representation of a data flow graph illustrating matrix operations to be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit, the resource constraints including local memory resources in each of the plurality of synchronous tiles; generating an assignment that assigns portions of the matrix operations to each of the plurality of synchronization tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the assignment, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; A method comprising:

2. generating the assignments, generating one or more candidate assignments; identifying, from among the one or more candidate allocations, a candidate allocation that satisfies the resource constraint; generating the allocation based on the candidate allocations that satisfy the resource constraints; 2. The method of claim 1, comprising:

3. 3. The method of claim 2, wherein the resource constraint of the hardware integrated circuit is a respective amount of static random access memory (SRAM) local to each of the plurality of synchronous tiles.

4. generating the assignments, generating one or more candidate assignments; determining a candidate allocation from the one or more candidate allocations that optimizes utilization of the plurality of synchronization tiles; generating the allocation based on the candidate allocation that optimizes utilization of the plurality of synchronization tiles; 2. The method of claim 1, comprising:

5. 5. The method of claim 4, wherein the resource constraint of the hardware integrated circuit is a bandwidth constraint that specifies how quickly data must move between nodes of the dataflow graph when performing the matrix operations on the hardware integrated circuit.

6. The matrix operation is performed using a matrix containing weights for neural network layers, and the step of generating the assignments comprises: performing sublayer sharding on the matrix containing weights for neural network layers; generating the allocation based on the sublayer sharding performed on the matrix; 2. The method of claim 1, comprising:

7. performing sub-layer sharding on the matrix containing weights for neural network layers, dividing the matrix into different parts for particular nodes in the data flow graph; 7. The method of claim 6, comprising:

8. generating the assignments, determining an allocation of the different portions to the plurality of synchronization tiles; 8. The method of claim 7, comprising:

9. i) the data flow graph comprises a plurality of nodes; ii) each of the plurality of nodes in the dataflow graph represents a layer of a neural network.

10. determining the minimum compute time comprises: generating a plurality of candidate assignments by assigning different portions of the matrix operations to the plurality of synchronization tiles; for each of the plurality of candidate allocations, calculating a maximum amount of compute time to perform the matrix operation across the plurality of synchronization tiles; selecting a candidate assignment from the plurality of candidate assignments that has a minimum or maximum amount of compute time; 2. The method of claim 1, comprising:

11. 1. A system comprising: a processing device; a non-transitory machine-readable storage device for storing instructions; the instructions, when executed by the processing device, are executable to cause performance of an operation, the operation being: an act of receiving a representation of a data flow graph illustrating matrix operations to be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit, the resource constraints including local memory resources in each of the plurality of synchronous tiles; generating an assignment that assigns portions of the matrix operations to each of the plurality of synchronization tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the allocation, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; Including, the system.

12. the act of generating the assignment comprises: generating one or more candidate assignments; identifying, from among the one or more candidate allocations, a candidate allocation that satisfies the resource constraint; generating the allocation based on the candidate allocations that satisfy the resource constraints; The system of claim 11 , comprising:

13. 13. The system of claim 12, wherein the resource constraint of the hardware integrated circuit is a respective amount of static random access memory (SRAM) local to each of the plurality of synchronous tiles.

14. the act of generating the assignment comprises: generating one or more candidate assignments; determining a candidate allocation from the one or more candidate allocations that optimizes utilization of the plurality of synchronization tiles; generating the allocation based on the candidate allocation that optimizes utilization of the plurality of synchronization tiles; The system of claim 11 , comprising:

15. 15. The system of claim 14, wherein the resource constraint of the hardware integrated circuit is a bandwidth constraint that specifies how quickly data must move between nodes of the data flow graph when performing the matrix operations on the hardware integrated circuit.

16. The matrix operation is performed using a matrix containing weights for neural network layers, and the operation of generating the assignment comprises: performing sublayer sharding on the matrix containing weights for neural network layers; generating the allocation based on the sublayer sharding performed on the matrix; The system of claim 11 , comprising:

17. performing sub-layer sharding on the matrix containing weights for neural network layers, dividing the matrix into different parts for a particular node in the data flow graph; 17. The system of claim 16, comprising:

18. i) the data flow graph comprises a plurality of nodes; ii) each of the plurality of nodes in the dataflow graph represents a layer of a neural network.

19. The act of determining the minimum compute time comprises: generating a plurality of candidate assignments by assigning different portions of the matrix operation to the plurality of synchronization tiles; for each of the plurality of candidate allocations, calculating a maximum amount of compute time to perform the matrix operation across the plurality of synchronization tiles; selecting a candidate assignment from the plurality of candidate assignments that has a minimum or maximum amount of compute time; Including, The system of claim 11.

20. a non-transitory machine-readable storage device for storing instructions, The instructions, when executed by a processing device, are executable to cause performance of operations, the operations including: an act of receiving a representation of a data flow graph illustrating matrix operations to be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit, the resource constraints including local memory resources in each of the plurality of synchronous tiles; generating an assignment that assigns portions of the matrix operations to each of the plurality of synchronization tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the allocation, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; a non-transitory machine-readable storage device, including

Images