Slicing for synchronization processors

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for partitioning a dataflow graph for a device having multiple synchronous slices. One method includes receiving a representation of a dataflow graph comprising a plurality of nodes, each node representing a respective matrix operation to be performed by the device having multiple synchronous slices. Evaluating candidate allocations of respective portions of the dataflow graph to each slice of the multiple synchronous slices according to one or more resource constraints of the device. Selecting one of the candidate allocations based on evaluating each candidate allocation.

Description

Technical Field

[0001] The inventive concept of this application relates to compilation technology for integrated circuit accelerators. Specifically, the inventive concept of this application relates to techniques for compiling programs written for systems having multiple synchronous processors. Background Technology

[0002] This specification relates to compiler technology for integrated circuit accelerators.

[0003] Synchronous integrated circuit accelerators are application-specific integrated circuits (ASICs) designed to perform highly parallel synchronous operations. Parallelism is achieved by integrating many different independent processing elements that can execute concurrently.

[0004] This type of device is well-suited for accelerating the inference pass through neural networks, where each of the independent processing elements performs different multiplications or additions of the layer inputs with weights. A neural network is a machine learning model that uses multiple layers of computation to predict one or more outputs from one or more inputs. Neural networks typically consist of one or more hidden layers located between input and output layers. The output of each layer is used as the input to another layer in the network (e.g., the next hidden layer or output layer).

[0005] An accelerator can use multiple independent processing tiles. In this specification, a tile refers to any suitable device with an array of computational units capable of performing computations on a portion of a matrix in at least partial parallelism. Therefore, each unit includes circuitry that allows that unit to perform mathematical or other computations. In a typical scenario, a tile receives an input vector, multiplies the input vector by a weight matrix using the computational array, and generates an output vector.

[0006] In this specification, "slice is synchronous" means that operations performed by the computing array do not involve branched execution, such as in an if / else statement of a command program. Instead, operations can be partially or entirely pre-scheduled. For example, operations of some synchronous processors can be scheduled down to the single-cycle level, meaning that each operation of each processing element of the computing array is assigned to a specific time slot in the sequence of execution cycles. Summary of the Invention

[0007] This specification describes techniques for compiling programs written for systems with multiple synchronous processors. As mentioned above, for the sake of brevity, such a device will be referred to as a slice in this specification.

[0008] The compiler can utilize periodic precision, allowing scheduling of operations on programs executing on multiple synchronous processors at this level. This enables fine-grained slices of operations, as well as other performance enhancements. The techniques described in this specification are particularly useful for scheduling operations in systems operating on multiple slices of a linear array, where each slice communicates only with its adjacent slices in the linear array. However, the same techniques can also be used in systems lacking such hardware configurations.

[0009] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. The compiler can leverage various sharding techniques to improve the performance of devices with multiple computational slices. Performing tasks on such devices can utilize sharding techniques to adapt to device specifications / constraints. The compiler can use sub-layer sharding to improve device utilization. The compiler can also use parallelogram time partitioning to reduce or eliminate idle regions where slices do not perform any work because they are waiting for something to happen. The compiler can also use over-sharding to improve load balancing between slices and handle unpredictable load balancing situations at compile time. The compiler can also use zigzag sharding to balance memory and computational demands within slices.

[0010] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0011] Figure 1 This is a flowchart of an example process for matrix operations on a device with multiple slices.

[0012] Figure 2A-2B An example of sublayer fragmentation is shown.

[0013] Figures 3A-3D An example of parallelogram time segmentation is shown.

[0014] Figure 4A -C shows an example of excessive fragmentation.

[0015] Figures 5A-5C An example of another version of over-fractionation, called sawtooth fragmentation, is shown.

[0016] Figure 6 An example of a slice that can be used to implement processing in a system with multiple synchronous processors is shown.

[0017] The same reference numerals and names in different figures denote the same elements. Detailed Implementation

[0018] Figure 1This is a flowchart of an example process for matrix operations on a device sharded with multiple slices. As described above, the target device may have slices arranged in a linear array or configured to operate as a linear array where each slice communicates only with its adjacent slices. Therefore, in response to receiving a data flow graph, the system can generate a graph with nodes and slices that takes into account the ordering of slices in the linear array. For convenience, the example process will be described as being performed by a system of one or more computers located in one or more locations and appropriately programmed according to this specification.

[0019] The system receives a representation of a data flow graph that specifies the matrix operations to be performed by a device with multiple slices (110). In this context, matrix operations should be understood to include matrix or vector operations. Each node of the data flow graph specifies the matrix operation to be performed on a matrix dedicated to that node. For example, each node in the graph could represent a layer of a neural network, with the matrix storing the weights to be used and learned for that layer. Each edge of the graph represents the propagation of data, where the output of one node becomes the input of the next node. The system generates multiple candidate assignments of the data flow graph to each of the multiple slices (120). The slice problem can be formulated as a search problem to find the optimal assignment of the data flow graph to the slices on the device from among the multiple candidate assignments. Various techniques related to generating candidates to be evaluated are described in more detail below. Which candidate assignments are actually the best depends largely on the data flow graph and the specific application.

[0020] The system evaluates each candidate allocation (130) based on one or more resource constraints of the device. Important resource constraints include memory constraints and bandwidth constraints.

[0021] One example memory constraint for machine learning accelerators is that all weights of all matrices assigned to a particular slice must fit into the slice's local static random access memory (SRAM). In other words, each slice can have an integrated amount of resident SRAM, which becomes a memory constraint when performing slices. The reasons for this constraint involve speed and bandwidth. First, machine learning accelerators are designed to perform the inference process through the model as quickly as possible. For some applications, dynamic random access memory (DRAM) is not fast enough. Furthermore, the weights of the matrices can be large, making them difficult to move. Therefore, when all weights of all matrices assigned to a particular slice can fit into the slice's local SRAM, the slice does not need to spend time reading and writing weights during the inference process or during training updates.

[0022] Another example of a memory constraint is a bandwidth constraint. A bandwidth constraint specifies the rate at which data should move between nodes in a graph. If a proposed allocation would exceed the bandwidth threshold of a particular edge in the graph, the system can determine that a bandwidth constraint has been violated.

[0023] The system selects one of the candidate assignments based on evaluating each candidate assignment (140). As an example, the system can perform layer slicing, where different matrices from different layers of the input program are assigned to different corresponding slices. Thus, for example, a neural network with 8 layers can assign its weight matrices to 8 slices respectively.

[0024] Given candidate allocations that satisfy one or more resource constraints, the system can determine which allocation best optimizes the utilization of slices on the device. Typically, this involves calculating the maximum computation time for all slices and then selecting the candidate allocation with the minimum maximum computation time. For devices with synchronized slices, which can be scheduled ahead to the cycle level, these calculations are estimated by the compiler with much greater accuracy than for other types of devices.

[0025] After selecting candidate assignments, the system can perform further descent of the data flow graph procedure by generating hardware instructions that will be executed by the device slices. The system can then load the hardware instructions, along with the matrix weights, onto the device and execute the hardware instructions on the device to implement the operations represented by the data flow graph.

[0026] Figure 2A-2B An example of sub-layer sharding is shown. Sub-layer sharding refers to dividing a matrix into specific nodes in a data flow graph and assigning different parts to different slices. Figure 2A An example candidate split is shown, where the elements of matrix 200 are split along the diagonal. Figure 2B Another example candidate partition is shown, where the elements of matrix 200 are partitioned between columns. Alternatively or additionally, matrix 200 may be partitioned between rows. In some implementations, the system uses sub-layer partitioning to generate candidate assignments whenever the matrix exceeds a size threshold.

[0027] To generate candidate assignments for sublayer slices, some additional complexity needs to be added to the data flow graph. Specifically, the vector input for each layer typically must be broadcast to each slice that receives a portion of the sublayer slice results. Furthermore, partial sum / or activations generated from individual slices must be merged into the final sum / or activations at a later stage, either on the same slice or on different slices. The compiler can take these additional complexities into account when evaluating candidate assignments involving sublayer slices.

[0028] Figures 3A-3CAn example of parallelogram time segmentation is shown. Parallelogram time segmentation can be advantageous when slices perform certain operations in different passes with different execution slopes. The execution slope refers to how quickly a slice can fully execute all operations in a sequence. This application is common for neural networks; for example, a single pass through all slices can represent one inference process through the network.

[0029] Figure 3A The conflict caused by different execution slopes is shown. Figure 3A In the middle, the horizontal axis represents time. Figure 3A The slices in the array are arranged in a linear array structure, such that each slice can only provide data to its two immediate neighbors and can only receive data from its two immediate neighbors. In some implementations, the linear array structure is arranged in a ring, such as a circular structure, so that the last slice can communicate with the first slice and vice versa.

[0030] Therefore, the slices are executed sequentially first, following "Process A," starting with slice 0 and working upwards to slice 7. Process A shares a common property with machine learning models: each slice cannot begin execution until it receives the output of the previous slice. Thus, slice 1 must wait for slice 0 to complete, slice 2 must wait for slice 1 to complete, and so on.

[0031] Figure 3A The dashed lines in the diagram illustrate how this common pattern forms the parallelogram of process A.

[0032] Figure 3A The second process, process B, is also shown. Process B takes less time than process A, therefore, process B has a steeper execution slope than process A.

[0033] In this example, the different execution slopes of processes A and B cause a conflict between slices 5 and 6. The reason for the conflict is that slice 6 has not yet completed its part of process A when slice 5 has completed process B. Therefore, pauses 301 and 302 must be introduced by the compiler or runtime, causing slice 5 to wait to provide a result for process B until slice 6 has completed its operation on process A.

[0034] Figure 3B An alternative strategy for managing different execution slopes is shown, which is to introduce a single pause 303 even before process B begins, so that the last operation of process B on slice 7 can be executed immediately after the last operation of process A.

[0035] Both strategies result in free regions 310 and 320, where slices remain idle and do no work. Free region 310 is smaller than free region 320. Although introducing multiple pauses introduces additional compile-time or runtime control complexity, this is... Figure 3B The strategies lacking in the middle.

[0036] Figure 3C This illustrates how different execution slopes actually introduce additional free space. Figure 3C In this context, process B, with its steeper execution slope, results in a first free zone 330. However, if process A is executed again after process B, this will introduce a second free zone 340.

[0037] Figure 3D This section explains how to use parallelogram time partitioning to manage different execution slopes. To reduce the idle areas introduced by different execution slopes, the compiler can generate schedules that force all procedures to effectively have the same execution slope. Therefore, different procedures are effectively coordinated to eliminate idle areas.

[0038] The compiler can use pauses or buffers to force a shallower slope. For example, as... Figure 3D As shown, the process introduces buffering between different slices, which has reduced the execution slope of process B. In other words, after slice 0 has completed process B, the result is stored in buffer 360. Slice 0 can then immediately begin processing the second iteration of process A. Alternatively or additionally, the compiler can insert a pause between the end of process B and the start of the second iteration of process A. Note that although this solution is compared to Figure 3A The method shown introduces more pauses, but the overall free space is much smaller.

[0039] Forced shallowing of the execution slope introduces additional control complexity, which the compiler can take into account when evaluating candidate assignments. In particular, forced shallowing of the execution slope introduces timing constraints on certain procedures. For example, in... Figure 3D In this context, the timing constraint is that process B on slice 1 cannot start immediately after process B on slice 0.

[0040] Figure 4A -C illustrates an example of over-sharding. Over-sharding refers to a technique that shards a matrix more finely than the hardware requires. In other words, the computation could be performed by a single slice, but the matrix is ​​sharded for other reasons. Typically, these other reasons are related to load balancing.

[0041] Figure 4A The example illustrates a single process A with 9 layers executed across 8 slices. Therefore, in this example, the 9th layer is entirely allocated to slice 0. Although slice 0 has the availability and memory capacity to fully execute all operations of the 9th layer, this leaves the remaining slices completely unused during this time.

[0042] Figure 4BThe diagram illustrates over-sharding of layer 9 in process A. In other words, even though a single slice could perform all the operations of layer 9, layer 9 is over-sharded to distribute it across all eight other slices. Intra-layer data dependencies may or may not exist; therefore, different parts of layer 9 can be executed partially or fully in parallel.

[0043] Therefore, the device exhibits better load balancing compared to having only one slice performing layer 9. This is particularly advantageous when the device is also performing other computations.

[0044] Figure 4C This illustrates another application of over-sharding for handling runtime load balancing. For some applications, the runtime load on the slices is unknown beforehand. One such example is that the device could be used to provide automatic machine translation using multiple different models trained for different translations. For example, as... Figure 4C As shown on the left, slice 0 can be assigned to execute model A, slice 1 can be assigned to execute model B, and so on. However, the actual order in which these models are used is unknown at compile time, especially when the device is used in a backend server serving upcoming internet services.

[0045] Therefore, it is possible that slice 0 handles the vast majority of requests, while other slices are rarely used.

[0046] To provide some runtime load balancing, the compiler can over-shard the model and distribute all models across all chips, such as... Figure 4C As shown on the right.

[0047] Now, at runtime, all slices can have essentially equal utilization, regardless of the actual distribution of runtime load.

[0048] Figures 5A-5C An example of another version of over-partitioning, known as zigzag partitioning, is shown. A common pattern emerges for many machine learning applications, including neural networks: the computational complexity of layers decreases, but memory requirements increase.

[0049] Therefore, as Figure 5A As shown, the memory model results in each subsequent slice requiring increasingly more memory. And as... Figure 5B As shown, the computing model requires fewer and fewer computing resources.

[0050] The compiler can use zigzag partitioning to achieve a more balanced allocation of memory and computation across all partitions. Zigzag partitioning essentially pairs different complementary layers together so that they are executed by the same partition. Therefore, for example, zigzag partitioning can ensure that the same partition will execute layers 1 and 8, 2 and 7, 3 and 6, and 4 and 5. This allocation is as follows: Figure 5C As shown, the calculation pattern is zigzag-shaped.

[0051] like Figure 5C As shown, the first layer, layer 1, is over-sharded and distributed across slices 0-3 to distribute the computational complexity of the first layer. In a similar but complementary manner, the eighth layer is also partitioned and distributed across the same slices 0-3 to distribute the memory requirements of the eighth layer.

[0052] Similarly, the second layer is always assigned to the same slice as the seventh layer, the third layer is always assigned to the same slice as the sixth layer, and the fourth layer is always assigned to the same slice as the fifth layer.

[0053] Each of the sharding techniques described above in conjunction with Figure 2-4 can be used alone or in combination with any one or more other sharding techniques.

[0054] Figure 6 An example of a slice 600 that can be used to implement processing in a system with multiple synchronous processors is shown. For example, slice 600 can be used as a slice in a slice linear array or as a slice in a system that operates as a slice linear array.

[0055] Each slice 600 includes a local memory 602 and a computation array 604 coupled to the memory 602. The local memory 602 includes physical memory located in proximity to the computation array 604. The computation array 604 includes a plurality of cells 606. Each cell 606 of the computation array 604 includes circuitry configured to perform computations (e.g., multiplication and accumulation operations) based on data inputs to the cell 606 (such as activation inputs and weight inputs). Each cell can perform computations (e.g., multiplication and accumulation operations) over a clock cycle. The computation array 604 may have more rows than columns, more columns than rows, or an equal number of columns and rows. For example, in Figure 6 In the example shown, computation array 604 comprises 64 cells arranged in 8 rows and 8 columns. Other computation array sizes are also possible, such as computation arrays with 16 cells, 32 cells, 128 cells, or 256 cells. Each slice may include the same number of cells and / or the same size computation array.

[0056] The memory 602 included in slice 600 may include, for example, random access memory (RAM), such as SRAM. Each memory 602 may be configured to store the (1 / n)th of the total memory associated with n slices 502 of a system having multiple slices. The memory 602 may be provided as a single chip or multiple chips. For example, Figure 6 The memory 602 shown is provided as four single-port SRAMs, each coupled to the compute array 604. Alternatively, the memory 602 can be provided as two single-port SRAMs or eight single-port SRAMs, etc. After error correction coding, the combined capacity of the memory can be, but is not limited to, 16 kB, 32 kB, 64 kB, or 128 kB. By providing the physical memory 602 locally to the compute array, wiring density can be significantly reduced.

[0057] Slice 600 also includes controllable bus lines. These controllable bus lines can be divided into several different groups. For example, the controllable bus lines may include a first group of general-purpose controllable bus lines 610, which are configured to transmit data between slices in each main direction. That is, the first group of controllable bus lines 610 may include: bus line 610a, configured to travel along the first dimension 501 of the slice grid in a first direction (in... Figure 6 Data is transmitted in the middle (referred to as "East"); bus line 610b is configured to move along the first dimension 101 of the sliced ​​grid to the second direction (in the middle). Figure 6 Data is transmitted in the second direction (referred to as "west"), where the second direction is opposite to the first direction; bus line 610c is configured to travel along the second dimension 103 of the sliced ​​grid to the third direction (in Figure 6 Data is transmitted in the north (referred to as "north"); and bus line 610d is configured to move along the second dimension 103 of the sliced ​​grid in the fourth direction (in the north). Figure 6 Data is transmitted in the direction referred to as "South" (or "South"), where the fourth direction is opposite to the third direction. The general-purpose bus line 610 can be configured to carry control data, activation input data, data from and / or to the communication interface, data from and / or to the vector processing unit, and data to be stored and / or used by the slice 600 (e.g., weight input). The slice 600 may include one or more control elements 621 (e.g., triggers and multiplexers) for controlling the controllable bus line and thus routing data to and / or from the memory 602.

[0058] The controllable bus lines may also include a second set of controllable bus lines, referred to herein as the computing array section and bus line 620. The computing array section and bus line 620 can be configured to carry data outputs from calculations performed by the computing array 604. For example, as... Figure 6 As shown, bus lines 620 can be configured to carry partial sums and data obtained from rows in computing array 604. In this case, the number of bus lines 620 will match the number of rows in array 604. For example, for an 8×8 computing array, there will be 8 partial sums and bus lines 620, each coupled to the output of the corresponding row in computing array 604. Computing array output bus lines 620 can be further configured to couple to another slice within the system, for example, as input to a computing array of an adjacent slice in a slice linear array. For example, array partial sums and bus lines 620 of slice 600 can be configured to receive input (e.g., partial sum 620a) of a computing array of a second slice located at least one slice away from slice 600. The output of computing array 604 is then added to the partial sums and bus lines 620 to produce a new partial sum 620b, which can be output from slice 600. The partial sum 620b can then be passed to another slice, or, alternatively, to a vector processing unit. For example, each bus line 620 can be coupled to a corresponding segment of the vector processing unit (such as segment 506 in Figure 5).

[0059] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0060] The term "data processing apparatus" refers to data processing hardware and includes various means, devices, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0061] A computer program (also referred to or described as a program, software, software application, application, module, software module, script, or code) can be written in any version of a programming language (including compiled or interpreted languages, or declarative or procedural languages) and can be deployed in any form (including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment). A program can, but is not required to, correspond to a file in a file system. A program can be stored as a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple co-located files, such as a file storing one or more modules, subroutines, or code sections. A computer program can be deployed to execute on a single computer, located at a site, or distributed across multiple sites and interconnected via a data communication network.

[0062] For a system consisting of one or more computers, being configured to perform a specific operation or action means that the system has software, firmware, hardware, or a combination thereof installed on it, which, in operation, cause the system to perform those operations or actions. For one or more computer programs, being configured to perform a specific operation or action means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0063] As used herein, "engine" or "software engine" refers to an input / output system implemented in software that provides outputs other than inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device including one or more processors and computer-readable media. Additionally, two or more engines can be implemented on the same computing device or on different computing devices.

[0064] The processes and logical flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. These processes and logical flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0065] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or contained within special-purpose logic circuitry. Typically, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device such as a universal serial bus (USB) flash drive, etc.

[0066] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and optical disks and DVDs.

[0067] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display), and a keyboard and pointing device, such as a mouse, trackball, or presence-sensitive display or other surface on which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone), running a messaging application, and in turn receiving response messages from the user.

[0068] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components, such as a data server, or middleware components, such as an application server, or front-end components, such as a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0069] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device acting as a client, for example, to display data to a user interacting with that device and to receive user input from that user. Data generated at the user device, such as the result of user interaction, can be received from that device at the server.

[0070] In addition to the embodiments described above, the following embodiments are also innovative:

[0071] Example 1 is a method comprising:

[0072] Receive a representation of a data flow graph comprising multiple nodes, each node representing a corresponding matrix operation to be performed by a device with multiple synchronization slices;

[0073] Generate multiple candidate assignments for each slice of multiple synchronization slices to the corresponding parts of the data flow graph;

[0074] Evaluate each of a plurality of candidate allocations based on one or more resource constraints of the device; and

[0075] One of the candidate assignments is selected based on the evaluation of each candidate assignment.

[0076] Example 2 is the method of Example 1, wherein each of the multiple synchronous slices includes a computing array of independent processing elements.

[0077] Example 3 is a method of any of Examples 1-2, wherein generating multiple candidate allocations based on one or more resource constraints includes generating candidate allocations based on SRAM memory constraints, wherein the SRAM memory constraints specify that all weights of the associated operations must fit in the local SRAM of the corresponding synchronization slice.

[0078] Example 4 is the method of Example 3, wherein each local SRAM is a resident memory integrated into the same synchronous slice in which the operation will be performed.

[0079] Example 5 is a method of any of Examples 1-4, wherein generating multiple candidate assignments includes assigning matrices of different corresponding layers to different corresponding slices.

[0080] Example 6 is a method of any of Examples 1-5, wherein generating multiple candidate assignments includes performing sub-layer sharding on the matrix of a specific node in the program, including assigning different blocks of the matrix to different corresponding synchronization slices.

[0081] Example 7 is the method of Example 6, wherein assigning different blocks of the matrix to different corresponding synchronization slices includes splitting the matrix along rows or columns or diagonally.

[0082] Example 8 is the method of Example 7, wherein the operation further includes performing sub-layer sharding on any matrix that exceeds a size threshold.

[0083] Example 9 is a method of any of Examples 1-8, wherein generating multiple candidate assignments includes performing parallelogram time segmentation such that different processes that initially have different execution slopes have substantially similar execution slopes.

[0084] Example 10 is the method of Example 9, wherein performing parallelogram time segmentation includes:

[0085] Calculate the corresponding execution slope for each of the two different processes;

[0086] Determine that the first process has a smaller execution slope than the second process; and

[0087] In response, the candidate assignment is modified to give the first process a shallower execution slope.

[0088] Example 11 is the method of Example 10, wherein modifying the candidate allocation includes inserting a buffer or pause in the candidate allocation.

[0089] Example 12 is a method of any of Examples 1-11, wherein generating multiple candidate assignments includes performing oversharding such that nodes of a data flow graph that can be fully executed by one of the slices are split across multiple different slices.

[0090] Example 13 is the method of Example 12, wherein performing over-sharding includes assigning a corresponding layer to each slice and splitting the last layer into all slices.

[0091] Example 14 is the method of Example 12, wherein performing excessive sharding includes splitting multiple matrices across all slices.

[0092] Example 15 is a method of any of Examples 1-14, wherein generating multiple candidate allocations includes performing zigzag partitioning by pairing different layers with different memory and computational requirements together.

[0093] Example 16 is the method of Example 15, wherein pairing different layers together includes pairing a first layer with a last layer, such that the same slice performs both the first layer and the last layer.

[0094] Example 17 is the method of Example 16, wherein the first layer has the greatest computational requirement and the last layer has the greatest memory requirement.

[0095] Example 18 is a method of any of Examples 1-17, wherein slices are arranged in a linear array such that each slice is configured to provide data to only one or two adjacent slices.

[0096] Example 19 is a system comprising: one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform the method of any one of Examples 1 to 18.

[0097] Example 20 is a computer storage medium encoded with a computer program including instructions that, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform the method of any of Examples 1 to 18.

[0098] Although this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from that combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0099] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order or sequence shown, or requiring all of the shown operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions listed in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system for partitioning a data flow graph for a device with multiple synchronization slices, comprising: One or more computers and one or more storage devices storing instructions, wherein, when executed by the one or more computers, the instructions are operable to cause the one or more computers to perform operations, the operations including: Receive a representation of a data flow graph comprising multiple nodes, each node representing a corresponding matrix operation to be performed by a device with multiple synchronization slices; Generating multiple candidate assignments of corresponding portions of the data flow graph to each of the plurality of synchronization slices, the generation comprising: Calculate the corresponding execution slope for each of the two different processes through the data flow graph; The first process in the two distinct processes is determined to have a smaller execution slope than the second process in the two distinct processes, wherein the execution slope represents how quickly the set of synchronization slices in the plurality of synchronization slices can fully execute all operations in the sequence of operations represented in the data flow graph; and In response, the candidate assignment is modified so that the second process has a smaller execution slope that is substantially similar to that of the first process; Each of the plurality of candidate allocations is evaluated based on one or more resource constraints of the device; and Based on the evaluation of each candidate assignment, the candidate assignment that best utilizes the slice is selected for execution from the plurality of candidate assignments.

2. The system of claim 1, wherein each of the plurality of synchronous slices comprises a computing array of independent processing elements.

3. The system of claim 1, wherein generating the plurality of candidate allocations based on one or more resource constraints includes generating the candidate allocations based on SRAM memory constraints, the SRAM memory constraints specifying that all weights of the associated operations must fit in the local SRAM of the corresponding synchronization slice.

4. The system of claim 3, wherein each local SRAM is a resident memory integrated into the same synchronous slice in which the operation will be performed.

5. The system according to claim 1, wherein, Each node in the data flow graph represents a layer of the neural network, each node has a matrix storing the weights to be used and / or learned for that layer, and wherein generating the plurality of candidate assignments includes assigning matrices of different corresponding layers to different corresponding slices.

6. The system according to claim 1, wherein, Each node in the data flow graph specifies a matrix operation to be performed on a matrix dedicated to that node, and wherein generating the plurality of candidate assignments includes performing sub-layer sharding on the matrix of a particular node in the data flow graph, including assigning different blocks of the matrix to different corresponding synchronization slices.

7. The system of claim 6, wherein assigning different blocks of the matrix to different corresponding synchronization slices comprises splitting the matrix along rows or columns or splitting the matrix diagonally.

8. The system of claim 7, wherein the operation further comprises performing sub-layer sharding on any matrix exceeding a size threshold.

9. The system of claim 1, wherein modifying the candidate allocation includes inserting a buffer or pause into the candidate allocation.

10. The system of claim 1, wherein generating the plurality of candidate assignments includes performing oversharding such that nodes of the data flow graph that can be fully executed by one of the shards are split across a plurality of different shards.

11. The system of claim 10, wherein performing over-sharding includes assigning a corresponding layer to each slice and splitting the last layer into all said slices.

12. The system of claim 10, wherein performing excessive sharding includes dividing a plurality of matrices across all said shards.

13. The system of claim 1, wherein generating the plurality of candidate allocations comprises performing zigzag partitioning by pairing together different layers with different memory and computational requirements.

14. The system of claim 13, wherein pairing different layers together includes pairing a first layer with a last layer such that the same slice performs both the first layer and the last layer.

15. The system of claim 14, wherein the first layer has the greatest computational requirement and the last layer has the greatest memory requirement.

16. The system of claim 1, wherein the slices are arranged in a linear array such that each slice is configured to provide data to only one or two adjacent slices.

17. A method for fragmenting a data flow graph for a device having multiple synchronization slices, comprising operations performed by one or more computers executing the system of any one of claims 1 to 16.

18. A computer storage medium encoded with a computer program, the program including instructions that, when executed by a data processing apparatus, are operable to cause the data processing apparatus to perform operations performed by one or more computers of the system of any one of claims 1 to 16.

Images