Sharding for synchronous processor

JP2024040198A5Active Publication Date: 2024-11-28GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024005431
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-08-22
Filing Date
2024-01-17
Publication Date
2024-11-28
Estimated Expiration
2040-08-20

AI Technical Summary

Technical Problem

Existing compiler techniques for synchronous integrated circuit accelerators face challenges in optimizing performance and load balancing across multiple processing elements, leading to idle zones and inefficient resource utilization.

Method used

Implementing sharding techniques such as sublayer sharding, parallelogram time slicing, oversharding, and zigzag sharding to schedule operations across multiple synchronous processors with cycle-level precision, ensuring efficient resource allocation and load balancing.

Benefits of technology

Enhances device performance by reducing idle zones, improving load balancing, and optimizing memory and computational requirements across tiles, resulting in improved utilization and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To increase the performance of a device having many synchronous tiles by using a sharding technique.SOLUTION: A method according to one aspect of the present invention includes a step of receiving a representation of a dataflow graph including a plurality of nodes that each represent respective matrix operations to be performed by a device having a plurality synchronous tiles. Candidate allocations of respective portions of the dataflow graph to each tile of the plurality of synchronous tiles are evaluated according to one or more resource constraints of the device. One of the candidate allocations is selected based on the evaluation of each candidate allocation.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This specification relates to compiler techniques for integrated circuit accelerators. [Background technology]

[0002] A synchronous integrated circuit accelerator is an application specific integrated circuit (ASIC) designed to perform highly parallel synchronous operations. Parallelism is achieved by integrating many different independent processing elements that can execute simultaneously.

[0003] Such devices are well suited for accelerating inference paths, for example, through neural networks in which each independent processing element performs a different multiplication or addition of layer inputs with weights. A neural network is a machine learning model that uses multiple layers of operation to predict one or more outputs from one or more inputs. A neural network typically includes one or more hidden layers located between an input layer and an output layer. The output of each layer is used as an input to another layer in the network, for example the next hidden layer or the output layer. Summary of the Invention [Means for solving the problem]

[0004] An accelerator may use multiple independent processing tiles. As used herein, a tile refers to any suitable device having an array of computational cells that can perform computations on portions of a matrix at least partially in parallel. Thus, each cell contains circuitry that enables the cell to perform mathematical or other computations. In a typical scenario, a tile receives an input vector and uses a computational array to multiply the input vector by a matrix of weights to generate an output vector.

[0005] As used herein, a tile is synchronous means that the operations performed by the computing array do not have branched execution, similar to if / else statements in an imperative program. Rather, the operations may be either partially or fully scheduled in advance. For example, the operations of some synchronous processors may be scheduled down to the individual cycle level, meaning that for every operation of every processing element, the computing array is assigned to a particular slot in a sequence of execution cycles.

[0006] This specification describes techniques for compiling programs written for systems having multiple synchronous processors. As noted above, for simplicity, such devices are referred to herein as tiles.

[0007] A compiler can exploit cycle-level precision with which operations of a program executing across multiple synchronous processors can be scheduled. This allows fine-grained sharding of operations, as well as other performance enhancements. The techniques described herein are particularly useful for scheduling operations in systems that operate multiple tiles in a linear array, where each tile only communicates with its neighbors in the linear array. However, the same techniques can also be used in systems that lack this hardware configuration. Effect of the Invention

[0008] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: Compilers may utilize various sharding techniques to increase performance of devices with a large number of compute tiles. Execution of tasks on such devices may conform to device specifications / constraints using sharding techniques. Compilers may use sub-layer sharding to increase device utilization. Compilers may also use parallelogram time slicing to reduce or eliminate idle zones where tiles do no work because they are waiting for something to happen. Compilers may also use over-sharding to improve load balancing between tiles and address load balancing situations that are unpredictable at compile time. Compilers may also use zigzag sharding to balance memory and computational requirements between tiles.

[0009] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. [Brief description of the drawings]

[0010] [Figure 1] 1 is a flowchart of an example method for sharding matrix operations for a device having a large number of tiles. [Figure 2A] FIG. 13 is a diagram illustrating an example of sublayer sharding. [Figure 2B] FIG. 13 is a diagram illustrating an example of sublayer sharding. [Figure 3A] FIG. 13 is a diagram illustrating an example of time slicing of a parallelogram. [Figure 3B] FIG. 13 is a diagram illustrating an example of time slicing of a parallelogram. [Figure 3C] FIG. 13 is a diagram illustrating an example of time slicing of a parallelogram. [Figure 3D]FIG. 13 is a diagram illustrating an example of time slicing of a parallelogram. [Figure 4A] FIG. 13 is a diagram illustrating an example of overshading. [Figure 4B] FIG. 13 is a diagram illustrating an example of overshading. [Figure 4C] FIG. 13 is a diagram illustrating an example of overshading. [Figure 5A] FIG. 2 shows an example of another version of oversharding called zigzag sharding. [Figure 5B] FIG. 2 shows an example of another version of oversharding called zigzag sharding. [Figure 5C] FIG. 2 shows an example of another version of oversharding called zigzag sharding. [Figure 6] FIG. 2 illustrates an example of tiles that can be used to implement processing in a system having multiple synchronous processors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Like reference numbers and designations in the various drawings indicate like elements.

[0012] 1 is a flow chart of an exemplary method for sharding matrix operations of a device having a large number of tiles. As described above, in a target device, tiles may be arranged in or configured to operate as a linear array, with each tile only communicating with its neighbors. Thus, in response to receiving a dataflow graph, the system may generate an assignment between nodes of the graph and tiles that takes into account the ordering of the tiles in the linear array. For convenience, the exemplary method is described as being performed by a system of one or more computers located in one or more locations and appropriately programmed in accordance with this specification.

[0013] The system receives 110 a representation of a dataflow graph that specifies matrix operations to be performed by a device having multiple tiles. In this context, matrix operations should be understood to include matrix operations or vector operations. Each node of the dataflow graph specifies a matrix operation to be performed on a matrix dedicated to that node. For example, each node in the graph can represent a layer of a neural network, with the matrix storing the weights to be used and learned for that layer. Each edge of the graph represents a propagation of data, where the output of one node becomes the input of the next node. The system generates 120 multiple candidate assignments of portions of the dataflow graph to each tile of the multiple tiles. The sharding problem can be formulated as a search problem to find the best assignment of portions of the dataflow graph to tiles in the device among multiple candidate assignments. Various techniques involved in generating the candidates to be evaluated are described in more detail below. Which candidate assignment is actually best depends heavily on the dataflow graph and the particular application.

[0014] The system evaluates each candidate allocation according to one or more resource constraints of the device 130. Important resource constraints include memory constraints and bandwidth constraints.

[0015] One exemplary memory constraint for a machine learning accelerator is that all weights of all matrices assigned to a particular tile must fit within a static random access memory (SRAM) that is local to that tile. In other words, each tile may have an integrated amount of resident SRAM, which becomes a memory constraint when sharding. The reason for this constraint has to do with speed and bandwidth. First, machine learning accelerators are designed to perform an inference pass through a model as fast as possible. And for some applications, dynamic random access memory (DRAM) is not fast enough. In addition, the weights of a matrix may be large and therefore difficult to move around. So, when all weights of all matrices assigned to a particular tile can fit within the tile's local SRAM, the tile does not have to spend time reading and writing weights during an inference pass or training update.

[0016] Another exemplary memory constraint is a bandwidth constraint, which specifies how quickly data must move between nodes in the graph. If a proposed allocation were to exceed the bandwidth threshold of a particular edge in the graph, the system can determine that the bandwidth constraint has been violated.

[0017] The system selects one of the candidate assignments based on an evaluation of each candidate assignment (140). As one example, the system can perform layer sharding, where different matrices from different layers of the input program are assigned to different respective tiles. Thus, for example, in a neural network with eight layers, the weight matrices can be assigned to each of the eight tiles.

[0018] For candidate allocations that satisfy one or more resource constraints, the system can determine which allocation best optimizes tile utilization on the device. In general, this involves computing the maximum amount of compute time across all tiles, and then selecting the candidate allocation that has the minimum maximum amount of compute time. For devices with synchronous tiles that can be scheduled in advance down to the cycle level, these calculations can be much more accurate than compiler estimates for other types of devices.

[0019] After the system selects a candidate assignment, it can perform further reduction of the dataflow graph program by generating hardware instructions to be executed by the tiles of the device. The system can then load the hardware instructions along with the matrix weights onto the device and execute the hardware instructions on the device to implement the operations represented by the dataflow graph.

[0020] 2A-2B show examples of sub-layer sharding. Sub-layer sharding refers to partitioning one matrix of a particular node in a dataflow graph and assigning different portions to different tiles. FIG. 2A shows one exemplary candidate partition in which the elements of matrix 200 are partitioned along a diagonal. FIG. 2B shows another exemplary candidate partition in which the elements of matrix 200 are partitioned between columns. Alternatively or additionally, matrix 200 can also be partitioned between rows. In some implementations, the system uses sub-layer sharding to generate candidate assignments whenever the matrix exceeds a size threshold.

[0021] Generating candidate assignments for sublayer sharding requires adding some additional complexity to the dataflow graph. Specifically, the vector inputs of each layer often must be broadcast to every tile that receives a portion of the sublayer sharding results. In addition, partial sums or partial activations generated by individual tiles must be merged into a final sum or activations at a later stage, either in the same tile or in different tiles. The compiler can take these additional complexities into account when evaluating candidate assignments involving sublayer sharding.

[0022] 3A-3C show an example of parallelogram time slicing. Parallelogram time slicing can be advantageous when tiles perform certain operations in different passes with different execution gradients. The execution gradient indicates how quickly a tile can complete all operations in a sequence. Such applications are common in neural networks, for example, where a single pass through all tiles can represent one inference pass through the network.

[0023] FIG. 3A illustrates conflicts resulting from different execution gradients. In FIG. 3A, the horizontal axis represents time. The tiles in FIG. 3A are arranged in a linear array structure, such that each tile can only provide data to its two immediate neighbors and can only receive data from its two immediate neighbors. In some implementations, the linear array structure is convoluted, e.g., a ring structure, such that the last tile can communicate with the first tile and vice versa.

[0024] Thus, tiles first perform Pass A sequentially starting with Tile 0 to Tile 7. Pass A has a common property of machine learning models that each tile cannot start executing until it has received output from the previous tile, so Tile 1 must wait for Tile 0 to finish, Tile 2 must wait for Tile 1 to finish, and so on.

[0025] The dashed lines in Figure 3A show how this common pattern forms the parallelogram of Pass A.

[0026] 3A also shows a second pass, Pass B. The operations of Pass B take less time than Pass A, and therefore Pass B has a steeper execution slope than Pass A.

[0027] In this example, a collision occurs between Tile5 and Tile6 due to the different execution slopes of Pass A and Pass B. This collision occurs because by the time Tile5 finishes Pass B, Tile6 has not yet finished its part of Pass A. Therefore, stalls 301 and 302 must be introduced either by the compiler or at run time, so that Tile5 waits to provide the result of Pass B until Tile6 finishes the operation of Pass A.

[0028] FIG. 3B shows an alternative approach for managing different execution slopes, which is to introduce a single stall 303 before Pass B has even started, thereby allowing the last operation of Pass B in Tile 7 to execute immediately after the last operation of Pass A.

[0029] Both of these approaches result in idle zones 310 and 320 where tiles are idle and not doing any work. Idle zone 310 is smaller than idle zone 320, but introduces a large number of stalls, resulting in additional compile-time or run-time control complexity not present in the approach in Figure 3B.

[0030] Figure 3C shows how differences in execution slopes can actually introduce additional idle zones: In Figure 3C, the steeper execution slope of Pass B results in a first idle zone 330. However, if Pass A were to be executed again after Pass B, this would introduce a second idle zone 340.

[0031] Figure 3D shows how parallelogram time slicing can be used to manage different execution slopes. To reduce the idle zones introduced by different execution slopes, the compiler can generate a schedule that effectively forces all paths to have the same execution slope. Thus, the different paths effectively fit together and there are no idle zones.

[0032] The compiler can use stalls or buffering to force a more gradual execution slope. For example, as shown in FIG. 3D, the execution slope of Pass B is made smaller by introducing buffering between different tiles. In other words, after Tile0 completes Pass B, the result is stored in buffer 360. Then, Tile0 can immediately start processing the second iteration of Pass A. Alternatively or additionally, the compiler may insert a stall between the end of Pass B and the start of the second iteration of Pass A. Note that although this solution introduces more stalls than the approach shown in FIG. 3A, the overall idle zone is substantially smaller.

[0033] Enforcing gradual execution slopes introduces additional control complexity that the compiler can take into account when evaluating candidate assignments. Specifically, enforcing gradual execution slopes introduces timing constraints for certain passes. For example, in Figure 3D, the timing constraint is that Pass B in Tile1 cannot start immediately after Pass B in Tile0.

[0034] 4A-4C show examples of oversharding. Oversharding refers to a technique for sharding a matrix more finely than the hardware requires. In other words, sometimes a computation can be done by a single tile, but the matrix is ​​sharded in some way for other reasons. Those other reasons are often related to load balancing.

[0035] 4A shows the execution of a single Pass A with 9 layers across 8 tiles. Thus, the 9th layer in this example is fully allocated to Tile 0. Tile 0 has the availability and memory capacity to fully perform all operations of the 9th layer, which leaves the remaining tiles completely unoccupied during this time.

[0036] FIG. 4B illustrates the oversharding of the 9th layer of Pass A. In other words, even if a single tile could perform all the operations of the 9th layer, the 9th layer is oversharded to distribute it across all eight of the other tiles. There may or may not be intra-layer data dependencies, so different parts of the 9th layer may be executed partially in parallel or fully in parallel.

[0037] Therefore, the load balancing of the device is better than if only one tile was doing the 9th layer, which can be especially advantageous if the device is also doing other computations as well.

[0038] FIG. 4C shows another application of oversharding to address runtime load balancing. In some applications, the runtime load for tiles is not known in advance. One such example is that a device may be used to perform automatic machine transformations using several different models trained for different transformations. For example, as shown on the left side of FIG. 4C, Tile0 may be assigned to run Model A, Tile1 may be assigned to run Model B, etc. However, the actual order in which these models are used is not known at compile time, especially when the device is used in a backend server servicing inbound Internet traffic.

[0039] So it may be the case that Tile0 handles the majority of requests and the other tiles are only rarely used.

[0040] To perform some runtime load balancing, the compiler can overshard the models and distribute all models across all chips, as shown on the right side of Figure 4C.

[0041] Then, at run-time, the utilization of all tiles can be made substantially equal, regardless of the actual distribution of run-time load.

[0042] 5A-5C show an example of another version of oversharding, called zigzag sharding. In many machine learning applications, including neural networks, a common pattern emerges where the compute complexity of a layer decreases but the memory requirements increase.

[0043] Thus, a memory pattern may result in each successive tile requiring more and more memory, as shown in Figure 5A, and a computational pattern requiring fewer and fewer computational resources, as shown in Figure 5B.

[0044] The compiler can use zigzag sharding to ensure that every tile has a better balance between memory allocation and computational allocation. Zigzag sharding essentially pairs different complementary layers together such that they are executed by the same tile. So, for example, zigzag sharding can ensure that the same tile will execute the first and eighth layers, the second and seventh layers, the third and sixth layers, and the fourth and fifth layers. This allocation is shown in FIG. 5C, where the computational pattern takes on a zigzag shape.

[0045] As shown in Figure 5C, the first layer, Layer 1, is oversharded and distributed across tiles 0-3 to partition the computational complexity of the first layer. In a similar but complementary manner, the eighth layer is also partitioned and distributed across the same tiles 0-3 to partition the memory requirements of the eighth layer.

[0046] Similarly, the second layer is always assigned to the same tile as the seventh layer, the third layer is always assigned to the same tile as the sixth layer, and the fourth layer is always assigned to the same tile as the fifth layer.

[0047] Each of the example sharding techniques described above in relation to Figures 2-4 may be used either alone or in combination with any one or more of the other sharding techniques.

[0048] 6 shows an example of a tile 600 that may be used to implement processing in a system having multiple synchronous processors. For example, the tile 600 may be used as one tile in a linear tile array or in a system in which the tiles operate as a linear tile array.

[0049] Each tile 600 includes a local memory 602 and a computing array 604 coupled to the memory 602. The local memory 602 includes a physical memory located proximate to the computing array 604. The computing array 604 includes a number of cells 606. Each cell 606 of the computing array 604 includes circuitry configured to perform a computing calculation (e.g., multiplication and accumulation operations) based on data inputs, such as activation inputs and weight inputs to the cell 606. Each cell can perform a computing calculation (e.g., multiplication and accumulation operations) on a cycle of a clock signal. The computing array 604 may have more rows than columns, more columns than rows, or an equal number of columns and rows. By way of example, in the example shown in FIG. 6, the computing array 604 includes 64 cells arranged in 8 rows and 8 columns. Other computing array sizes are possible, such as computing arrays having 16 cells, 32 cells, 128 cells, or 256 cells, among others. Each tile may contain the same number of cells and / or the same size of a computational array.

[0050] The memory 602 included in the tile 600 may include, for example, a random access memory (RAM), such as an SRAM. Each memory 602 may be configured to store 1 / nth of the total memory associated with the n tiles 502 of a system having multiple tiles. The memory 602 may be provided as a single chip or multiple chips. For example, the memory 602 shown in FIG. 6 is provided as four single-port SRAMs, each of which is coupled to a computing array 604. Alternatively, the memory 602 may be provided as two single-port SRAMs or eight single-port SRAMs, among other configurations. The combined capacity of the memory may be, for example, but not limited to, 16 kB, 32 kB, 64 kB, or 128 kB after error correction coding. By providing the physical memory 602 local to the computing array, the density of the wiring may be significantly reduced.

[0051] The tile 600 also includes controllable bus lines, which may be categorized into a number of different groups. For example, the controllable bus lines may include a first group of general purpose controllable bus lines 610 configured to transfer data between tiles for each cardinal direction. That is, the first group of controllable bus lines 610 includes bus lines 610a configured to transfer data toward a first orientation along the first dimension 101 of the grid tile (referred to as "East" in FIG. 6), bus lines 610b configured to transfer data toward a second orientation along the first dimension 101 of the grid tile that is opposite to the first orientation (referred to as "West" in FIG. 6), bus lines 610c configured to transfer data toward a third orientation along the second dimension 103 of the grid tile (referred to as "North" in FIG. 6), and bus lines 610d configured to transfer data toward a fourth orientation along the second dimension 103 of the grid tile that is opposite to the third orientation (referred to as "South" in FIG. 6). The general-purpose bus lines 610 may be configured to carry control data, activation input data, data from and / or to the communication interfaces, data from and / or to the vector processing units, and data (e.g., weight inputs) that are to be stored and / or used by the tile 600. The tile 600 may include one or more control elements 621 (e.g., flip-flops and multiplexers) for controlling the controllable bus lines and thus for routing data to and / or from the tile 600 and / or from the memory 602.

[0052] The controllable bus lines may also include a second group of controllable bus lines, referred to herein as computational array partial sum bus lines 620. The computational array partial sum bus lines 620 may be configured to carry data output from computations performed by the computational array 604. For example, the bus lines 620 may be configured to carry partial sum data from rows in the computational array 604, as shown in FIG. 6. In such a case, the number of bus lines 620 would match the number of rows in the array 604. As an example, for an 8×8 computational array, there would be eight partial sum bus lines 620, each coupled to an output of a corresponding row in the computational array 604. The computational array output bus lines 620 may be further configured to couple to another tile in the system, for example, as an input to the computational array of an adjacent tile in a linear array of tiles. For example, the array partial sum bus lines 620 of tile 600 may be configured to accept an input (e.g., partial sum 620a) of a computing array of a second tile located at least one tile away from tile 600. An output of computing array 604 may then be added to partial sum bus lines 620 to create a new partial sum 620b, which may be output from tile 600. Partial sum 620b may then be passed to another tile or, alternatively, to a vector processing unit. For example, each bus line 620 may be coupled to a corresponding section of a vector processing unit (such as section 506 in FIG. 5).

[0053] The embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, in tangible embodied computer software or firmware, in computer hardware, or in a combination of one or more of them, including the structures disclosed herein and their structural equivalents. The embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded in a tangible non-transitory storage medium for execution by or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information to be transmitted to a receiver device suitable for execution by a data processing device.

[0054] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for computer programs, e.g., code constituting a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0055] A computer program, which may also be referred to or written as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, for example in one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, for example files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0056] A system of one or more computers is configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. A system of one or more computer programs is configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0057] As used herein, an "engine" or "software engine" refers to a software-implemented input / output system that provides an output distinct from the input. An engine can be an encoded functional block, such as a library, a platform, a software development kit (SDK), an object, etc. Each engine can be implemented in any suitable type of computing device, such as a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smartphone, or other fixed or portable device, including one or more processors and a computer-readable medium. In addition, two or more of the engines may be implemented in the same computing device or in different computing devices.

[0058] The methods and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The methods and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0059] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. In general, the computer will also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data from them, transfer data to them, or both. However, a computer need not have such devices. Moreover, the computer may be embedded within another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0060] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks and DVD-ROM disks.

[0061] To provide for user interaction, embodiments of the subject matter described herein can be implemented in a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse, trackball, or presence-sensitive display or other surface, by which the user can provide input to the computer. Other types of devices can be used as well to provide for user interaction, e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0062] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0063] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server transmits data, e.g., HTML pages, to a user device, e.g., for purposes of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, e.g., a result of a user interaction, can be received at the server from the device.

[0064] In addition to the above-mentioned embodiments, the following embodiments are also innovative: In embodiment 1, receiving a representation of a data flow graph including a number of nodes each representing a respective matrix operation to be performed by a device having a number of synchronization tiles; generating a plurality of candidate assignments of respective portions of the dataflow graph to each tile of the plurality of synchronization tiles; evaluating each candidate allocation of the plurality of candidate allocations according to one or more resource constraints of the device; selecting one of the candidate assignments based on an evaluation of each candidate assignment; The method includes:

[0065]

[0023] Example 2 is the method of example 1, wherein each tile of the plurality of synchronization tiles comprises a computing array of independent processing elements.

[0066] Embodiment 3 is the method of embodiment 1 or 2, wherein generating the plurality of candidate allocations according to one or more resource constraints includes generating the candidate allocations according to an SRAM memory constraint, the SRAM memory constraint specifying that all weights of the associated operations must fit within a local SRAM that is local to the corresponding synchronization tile.

[0067] Example 4 is the method of example 3, wherein each local SRAM is a resident memory integrated in the same synchronous tile in which the operation is to be performed.

[0068]

[0023] Embodiment 5 is the method according to any one of embodiments 1 to 4, wherein the step of generating the plurality of candidate allocations comprises the step of allocating matrices of different respective layers to different respective tiles.

[0069] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the step of generating the plurality of candidate allocations includes a step of performing sub-layer sharding on a matrix for a particular node in the program, the step including a step of allocating different blocks of the matrix to different respective synchronization tiles.

[0070] Embodiment 7 is a method according to embodiment 6, in which the step of allocating different blocks of the matrix to each different synchronization tile includes a step of dividing the matrix along rows or columns, or a step of dividing the matrix diagonally.

[0071] Embodiment 8 is the method of embodiment 7, wherein the operations further include performing sublayer sharding on any matrices that exceed a size threshold.

[0072] Embodiment 9 is a method according to any one of embodiments 1 to 8, wherein the step of generating the plurality of candidate allocations includes a step of performing parallelogram time slicing such that different paths that initially have different execution slopes have substantially similar execution slopes.

[0073] In a tenth embodiment, the step of performing parallelogram time slicing includes: computing a respective running slope for each of the two different paths; determining that the first pass has a smaller execution slope than the second pass; in response, modifying the candidate allocations such that the first pass has a gentler performance slope; 10. The method of embodiment 9, comprising:

[0074]

[0023] Example 11 is the method of example 10, wherein modifying the candidate allocation comprises inserting buffering or stalls into the candidate allocation.

[0075] Embodiment 12 is a method according to any one of embodiments 1 to 11, wherein the step of generating the plurality of candidate assignments includes a step of oversharding such that nodes of the dataflow graph that can be entirely made by one of the tiles are divided among a number of different tiles.

[0076]

[0023] Example 13 is the method of example 12, wherein the step of oversharding comprises the step of allocating one respective layer to each tile and splitting the last layer among all said tiles.

[0077]

[0023] Example 14 is the method of example 12, wherein the oversharding step includes dividing a plurality of matrices across all of the tiles.

[0078] Embodiment 15 is a method according to any one of embodiments 1 to 14, wherein the step of generating the plurality of candidate allocations includes a step of performing zigzag sharding by pairing together different layers having different memory and computational requirements.

[0079] Example 16 is the method of example 15, wherein pairing different layers together includes pairing a first layer with a last layer such that the same tile performs both the first layer and the last layer.

[0080]

[0023] Example 17 is the method of example 16, wherein the first layer has the greatest computational requirements and the last layer has the greatest memory requirements.

[0081] Embodiment 18 is a method according to any one of embodiments 1 to 17, wherein the tiles are arranged in a linear array such that each tile provides data to only one or two adjacent tiles.

[0082] Embodiment 19 is a system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform a method according to any one of embodiments 1 to 18.

[0083] Embodiment 20 is a computer storage medium encoded with a computer program, the program including instructions operable, when executed by a data processing device, to cause the data processing device to perform a method according to any one of embodiments 1 to 18.

[0084] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as functioning in a particular combination, and even may be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0085] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0086] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the methods depicted in the appended figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0087] 101 First Dimension 103 Second Dimension 200 Matrix 301, 302, 303 stalls 310, 320, 330, 340 Idol Zone 360 Buffer 502 tiles 506 classification 600 tiles 602 Local Memory, Physical Memory 604 Computer Calculation Array 606 Cell 610 General Purpose Controllable Bus Line 610a, 610b, 610c, 610d bus lines 620 Computer Calculation Array Partial Sum Bus Line 620a, 620b partial sum 621 Control Elements

Claims

1. A method comprising: receiving a representation of a data flow graph illustrating matrix operations that can be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit; generating an assignment for assigning portions of the matrix operations to each of the plurality of synchronous tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the assignment, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; A method comprising:

2. The step of generating the allocation comprises: generating one or more candidate assignments; identifying, from among the one or more candidate allocations, a candidate allocation that satisfies the resource constraint; generating the allocation based on the candidate allocations that satisfy the resource constraints; 2. The method of claim 1, comprising:

3. The method of claim 2, wherein the resource constraint of the hardware integrated circuit is the respective capacity of static random access memory (SRAM) local to each of the plurality of synchronous tiles.

4. The step of generating the allocation comprises: generating one or more candidate assignments; determining a candidate allocation from among the one or more candidate allocations that optimizes utilization of the plurality of synchronization tiles; generating the allocation based on the candidate allocations that optimize utilization of the plurality of synchronization tiles; 2. The method of claim 1, comprising:

5. The method described in claim 4, wherein the resource constraint of the hardware integrated circuit is a bandwidth constraint that specifies how quickly data must move between nodes of the data flow graph when performing the matrix operation on the hardware integrated circuit.

6. The method of claim 5, wherein the matrix operation is performed using a matrix containing weights for neural network layers, and the step of generating the assignment comprises: performing sub-layer sharding on the matrix containing weights for neural network layers; generating the allocation based on the sublayer sharding performed on the matrix; 2. The method of claim 1, comprising:

7. The method of claim 6, further comprising the step of: performing sub-layer sharding on the matrix containing weights for a neural network layer, Dividing the matrix into different parts for particular nodes in the data flow graph.

7. The method of claim 6, comprising:

8. The step of generating the allocation, determining an allocation of the different portions to the plurality of synchronization tiles; 8. The method of claim 7, comprising:

9. The method of claim 1, wherein the data flow graph comprises:

7. The method of claim 6, wherein ii) each of the plurality of nodes in the dataflow graph represents a layer of a neural network.

10. The step of determining a minimum compute time, comprising: generating a plurality of candidate assignments by assigning different portions of the matrix operations to the plurality of synchronization tiles; calculating, for each of the plurality of candidate allocations, a maximum amount of compute time to perform the matrix operation across the plurality of synchronization tiles; selecting a candidate assignment from among the plurality of candidate assignments that has a minimum or maximum amount of compute time; 2. The method of claim 1, comprising:

11. A system comprising: A processing device; a non-transitory machine-readable storage device for storing instructions; the instructions, when executed by the processing device, are executable to cause performance of an operation, the operation comprising: receiving a representation of a data flow graph representing matrix operations to be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit; generating an assignment for assigning portions of the matrix operations to each of the plurality of synchronization tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the assignments, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; Including, the system.

12. The act of generating the allocation comprising: generating one or more candidate assignments; identifying, from among the one or more candidate allocations, a candidate allocation that satisfies the resource constraint; generating the assignment based on the candidate assignments that satisfy the resource constraints; The system of claim 11 , comprising:

13. The system of claim 12, wherein the resource constraint of the hardware integrated circuit is a respective amount of static random access memory (SRAM) local to each of the plurality of synchronous tiles.

14. The act of generating the allocation comprising: generating one or more candidate assignments; determining a candidate allocation from the one or more candidate allocations that optimizes utilization of the plurality of synchronization tiles; generating the assignment based on the candidate assignments that optimize utilization of the plurality of synchronization tiles; The system of claim 11 , comprising:

15. The system described in claim 14, wherein the resource constraint of the hardware integrated circuit is a bandwidth constraint that specifies how quickly data must move between nodes of the data flow graph when performing the matrix operation on the hardware integrated circuit.

16. The method of claim 15, wherein the matrix operation is performed using a matrix containing weights for neural network layers, and the operation of generating the assignment comprises: performing sub-layer sharding on the matrix containing weights for neural network layers; generating the allocation based on the sublayer sharding performed on the matrix; The system of claim 11 , comprising:

17. The method of claim 16, further comprising the steps of: performing sub-layer sharding on the matrix containing weights for a neural network layer, the sub-layer sharding comprising: splitting said matrix into different parts for a particular node in said data flow graph; The system of claim 16, comprising:

18. The method of claim 1, wherein: i) the data flow graph comprises a plurality of nodes; ii) each of the plurality of nodes in the dataflow graph represents a layer of a neural network.

19. The method of claim 18, further comprising: generating a plurality of candidate assignments by assigning different portions of the matrix operation to the plurality of synchronization tiles; calculating, for each of the plurality of candidate assignments, a maximum amount of compute time to perform the matrix operation across the plurality of synchronization tiles; selecting a candidate assignment from among the plurality of candidate assignments that has a minimum or maximum amount of compute time; Including, The system of claim 11.

20. A non-transitory machine-readable storage device for recording instructions, comprising: The instructions, when executed by a processing device, are executable to cause performance of an operation, the operation being: receiving a representation of a data flow graph representing matrix operations to be performed by a hardware integrated circuit comprising a plurality of synchronization tiles; determining a minimum compute time for performing the matrix operation using the plurality of synchronous tiles based on resource constraints of the hardware integrated circuit; generating an assignment for assigning portions of the matrix operations to each of the plurality of synchronization tiles of the hardware integrated circuit based on the minimum compute time; generating, based on the assignments, hardware instructions to the hardware integrated circuit that synchronize execution of each of the portions of the matrix operation across the plurality of synchronization tiles; A non-transitory machine-readable storage device comprising: