Methods, systems, and computer storage media for optimizing hardware FIFO instructions.

Abstract

This disclosure relates to methods, systems, and computer storage media for optimizing hardware FIFO instructions. Methods, systems, and apparatus for scheduling First-In-First-Out (FIFO) instructions are described. In one aspect, a method includes receiving data representing code of a program to be executed by a processing unit having hardware processors. For each of one or more hardware processors, the order of individual FIFO instruction groups for execution by the hardware processors is identified in the data representing the code. For each individual FIFO instruction group for execution by the hardware processors, a path length metric representing how long it takes to reach the end of the program from the individual FIFO instruction group is determined. A new order of the individual FIFO instruction groups for execution by the hardware processors is generated, based at least on the path length metric for each individual FIFO instruction group for execution by the hardware processors.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201980093043.X, filed on November 26, 2019. Background Technology

[0003] Some processors (e.g., domain-specific processors) may include long-latency, high-throughput hardware units. For example, some processors may include systolic arrays to efficiently compute complex numerical workloads. The natural programming interface for such hardware units is a first-in, first-out (FIFO) queue defined in the processor's instruction set architecture (ISA), because there are instructions responsible for pushing and popping data from these units. However, hardware FIFOs impose ordering constraints on instructions and can retain additional state as configuration, making it difficult for compilers to efficiently optimize and schedule instructions. Summary of the Invention

[0004] This specification describes techniques involving, for example, improving the order of FIFO instructions at compile time to reduce latency when performing machine learning and other computations.

[0005] Typically, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising receiving data representing code of a program to be executed by a processing unit including a hardware processor. The method comprises, for each of one or more hardware processors: identifying in the data representing the program the order of independent First-In-First-Out (FIFO) instruction sets for execution by the hardware processor; for each independent FIFO instruction set to be executed by the hardware processor, determining a path length metric, the path length metric representing how long it will take to reach the end of the program from the independent FIFO instruction set; and generating a new order of the independent FIFO instruction sets to be executed by the hardware processor, based at least on the path length metric for each independent FIFO instruction set to be executed by the hardware processor. Updated code for the program is generated, based at least on each new order of the independent FIFO instruction sets for the one or more hardware processors. Output code for execution by the hardware processor is generated based on the updated code for the program. The output code is provided to the hardware processor. Other embodiments of this and other aspects include corresponding systems, methods, and computer programs configured to perform actions of a method encoded on a computer storage device. A system capable of configuring one or more computers by means of software, firmware, hardware, or a combination thereof installed on the system, which in operation cause the system to perform actions. A system capable of configuring one or more computer programs by means of instructions that, when executed by a data processing device, cause that device to perform actions.

[0006] These and other implementations may each optionally include one or more of the following features. In some aspects, the data representing the program's code includes a compiler intermediate representation of the program's source code. In some aspects, the output code includes compiled machine code executable by one or more hardware processors.

[0007] Some aspects include generating a graph representing the sequence of instructions for a program based on data representing the program's code. Generating a new order of independent FIFO instruction sets for a given hardware processor may include reordering the independent FIFO instruction sets in the graph to generate an updated graph. Generating updated code for a program based at least on each new order of independent FIFO instruction sets for one or more hardware processors may include generating updated instruction sequences based on the updated graph.

[0008] In some respects, generating update code for a program based at least on each new order of independent FIFO instruction groups for one or more hardware processors may include: identifying instructions not identified as FIFO instructions in the update graph; for one or more of the identified instructions: selecting an independent FIFO instruction group based at least on the path length between the independent FIFO instruction group and the identified instructions in the update graph; and scheduling the identified instructions and the selected long-latency group in the update code.

[0009] In some aspects, multiple hardware processors execute independent FIFO instruction sets. Some aspects may include, for at least one of the multiple hardware processors, determining the order of independent FIFO instruction sets executed by at least one hardware processor based on whether the independent FIFO instruction sets are on the critical path of the program. Some aspects may include determining that a first hardware processor comprises multiple independent FIFO instruction sets; and determining that an independent FIFO instruction set of a second hardware processor is an instruction that receives data output from one of the multiple independent FIFO instruction sets of the first hardware processor. In some aspects, generating a new order for the independent FIFO instruction sets executed by the first hardware processor may include, in response to an instruction determining that an independent FIFO instruction set of a second hardware processor is an instruction that receives data output from one of the multiple independent FIFO instruction sets of the first hardware processor, moving one independent FIFO instruction set of the first hardware processor to execute before one or more other independent FIFO instruction sets of the first hardware processor.

[0010] In some aspects, each individual FIFO instruction set includes one or more input instructions for providing input data to the hardware processor and one or more output instructions for outputting data from the hardware processor based on the input data. In some aspects, the specific hardware processor of the processing unit is a configurable hardware processor that executes instructions using a variety of different configurations. Some aspects may include identifying, in the data representing the program code, a set of FIFO instructions that will be executed by the specific hardware processor using the same configuration for the specific hardware processor; and determining, in the updated code for the application, a new order for the FIFO instruction sets, such that the FIFO instruction sets are executed sequentially without requiring any intervening FIFO instructions with different configurations for the specific hardware processor.

[0011] In some aspects, one or more hardware processors include multiple processors. Some aspects may include, for each of the multiple hardware processors, determining a performance metric indicating how long the hardware processor will take to complete all FIFO instructions for the hardware processor; determining an evaluation order for the multiple hardware processors based on the performance metric for each of the multiple hardware processors; and determining a new order of individual FIFO instruction groups for each of the multiple hardware processors, one at a time, according to the determined evaluation order.

[0012] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. By generating an improved order of FIFO instructions used by the hardware processor to input and output data, latency during program execution can be reduced. Non-FIFO instructions can be grouped with FIFO instructions (e.g., with the nearest FIFO instruction) to further improve and reduce latency during program execution. For example, by grouping another instruction that provides data to the FIFO instruction for loading into the hardware processor with the FIFO instruction, the instructions can be scheduled together such that the amount of time the FIFO instruction must wait for such data is reduced. By reducing latency during program execution, complex workloads, such as deep neural network workloads, can be completed faster and more efficiently. Therefore, by utilizing the described techniques at compile time, a given program can be executed at runtime with reduced latency compared to a regular compiler executed on the same hardware processor. Furthermore, by taking into account the characteristics of a particular hardware processor during the generation of the improved order of FIFO instructions, the execution of FIFO instructions can be customized / adapted to the specific hardware processor present in the system executing the FIFO instructions.

[0013] Details of one or more embodiments of the subject matter described herein are set forth in the following figures and description. Other potential features, aspects, and advantages of this subject matter will become apparent from the description, figures, and claims. Attached Figure Description

[0014] Figure 1 It is a block diagram of an exemplary environment in which a program processing system modifies a program and uses the modified program to perform calculations.

[0015] Figure 2 The directed graph of the program's instructions and the update graph of the program's instructions are depicted.

[0016] Figure 3 This is a flowchart illustrating an exemplary process for modifying a program.

[0017] In the various figures, similar reference numerals and names indicate similar elements. Detailed Implementation

[0018] Typically, the topics described in this specification relate to scheduling hardware FIFO instructions for a hardware processor to reduce the overall latency of a program executed by the processor. The processing unit may include multiple hardware processors, including long-latency, high-throughput hardware processors that efficiently perform complex computations. For example, a long-latency hardware processor may be configured to use dedicated hardware to perform matrix multiplication calculations for machine learning models or other complex computations (e.g., digital signal processing, video encoding, decoding, etc.), and such computations may take multiple processor loops to compute the output.

[0019] These long-latency hardware processors can be programmed via push instructions to send input data to the hardware processor and pop instructions to extract result data from the hardware processor for use with another hardware processor or another FIFO of the hardware processor. Push and pop typically occur in FIFO order, allowing the hardware processor to be modeled as a FIFO queue. In some cases, the hardware processor can also maintain additional state to represent its configuration, which can change via additional instructions. For example, a hardware processor can have multiple configurations and can maintain statistics specifying the current configuration.

[0020] Some hardware processors produce results based on a single input. In this example, a pop (e.g., output) from a FIFO corresponds to a push (e.g., input) to a FIFO. Other hardware processors require multiple inputs to produce one or more results. In this example, a group of one or more pops from a FIFO corresponds to a group of pushes to a FIFO. For example, a hardware processor performing matrix multiplication may include multiple pushes providing the values ​​of a matrix to the hardware processor, and one or more pops outputting the calculated values ​​to another hardware processor or FIFO. Such a group of associated pushes and corresponding pops based on the pushes represent an indivisible computational unit performed by the hardware processor. Each group of one or more pops from a FIFO corresponding to one or more pushes to the same FIFO (e.g., each group of one or more pops from a FIFO whose output is based on one or more results from one or more pushes to the same FIFO) is an independent group of FIFO instructions. Different independent groups of FIFO instructions can be reordered to reduce latency. In some implementations, FIFO instructions within an independent group are not reordered.

[0021] Dependencies between instructions can be modeled in directed graphs. Instructions are vertices or nodes in the graph, and dependencies are edges between vertices or nodes. If a subsequent instruction must wait for a preceding instruction to complete before it can begin, the graph can include directed edges from the preceding instruction to the subsequent instruction. The length of the edge can represent the waiting time between instructions. This graph can encode all types of dependencies between instructions, including producer-consumer, write-before-read, and FIFO ordering constraints.

[0022] There are several types of FIFO ordering dependencies, including control dependencies and data dependencies. Control dependencies are imposed by the FIFO processing order of the hardware processor. A control dependency edge from first instruction to second instruction means that the second instruction executes after the first instruction and must wait for the first instruction to complete before the second instruction can begin. A data dependency edge from first instruction to second instruction means that the second instruction uses the data produced by the first instruction. For example, a data dependency could mean that the popping of an operation result must wait for another instruction, such as the pushing of an input.

[0023] An example of control dependency is a subsequent push (or pop) from a FIFO queue that must wait for a previous push or pop from the same FIFO queue. In this example, the wait time between instructions is based on the hardware processor's throughput. Another example of control dependency is when the FIFO queue does not have enough space for a new push. The new push must wait for a pop, which will free up enough space in the FIFO queue for the new push. An example of data dependency is that, within a separate group of FIFO instructions, a result pop must wait for an input push. In this example, the hardware processor uses the data received from the input push to perform calculations to determine the data to be popped out by the output. The wait time in this example is determined by the hardware processor's latency. Each of these dependencies restricts how the order of FIFO instructions is determined or modified.

[0024] FIFO control dependencies can prevent optimal instruction scheduling by a hardware processor if the original order of FIFO instructions is not optimal. For example, consider two hardware FIFOs (e.g., a hardware processor modeled as a FIFO queue), FIFO1 and FIFO2. Also consider that FIFO1 has 10 independent pushes, and the last result from FIFO1 is used as the input for the first push to FIFO2. FIFO control dependencies require that all 10 FIFO1 operations be completed before scheduling the first FIFO1 push. To address this problem, a new order of FIFO instructions can be determined, for example, before determining the final control dependency. In this example, the new order of FIFO instructions could use the previous last result of FIFO1 as the first result of FIFO1, reducing the latency of FIFO2 waiting for that result. The techniques described in this paper use performance metrics, critical path heuristics, and other considerations to schedule independent groups of FIFO instructions and other related instructions, resulting in reduced program execution latency.

[0025] Figure 1 This is a block diagram of an exemplary environment 100 in which the program processing system 130 modifies a program and performs computations using the modified program. The machine learning processing system 130 can receive program 112 from user equipment 110 via a data communication network 120 (e.g., a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof).

[0026] Program 112 may include workloads such as machine learning workloads, digital signal processing workloads, video encoding / decoding workloads, or other suitable workloads. For example, each program 112 may define a directed computation graph, or in the form of a directed computation graph, that specifies the operators used for the workload. Each operator represents an operation, such as a numerical operation, to be performed by the processing unit 160 of the machine learning processing system 130. For example, operators may represent mathematical operations (e.g., addition, multiplication, maximum, minimum, etc.), tensor operations (e.g., integer, transpose, etc.), data copying operations, and / or other suitable operations.

[0027] The program processing system 130 includes a compiler 140 that compiles program 112 into machine instructions for processing unit 160. Processing unit 160 may include one or more hardware processors that execute the machine instructions to perform computations defined by program 112 and / or modified versions of the program, as described below. In this example, processing unit 160 includes a systolic array 162 of hardware processor 164. Other suitable arrangements of processors may also be used. For example, processing unit 160 may be a graphics processing unit (GPU) or an application-specific integrated circuit (ASIC), such as a machine learning accelerator ASIC, like a tensor processing unit (TPU).

[0028] The compiler 140 can also modify program 112 to improve the performance of processing unit 160 when executing machine instructions. For example, the compiler 140 can modify program 112 based on the characteristics of processing unit 160 that will perform the computation of program 112. The compiler 140 can also adjust or determine the scheduling of instructions to improve the performance of processing unit 160 when executing the program, for example, to reduce the latency of executing the program at runtime.

[0029] Specifically, compiler 140 includes a FIFO scheduler 142 that can determine whether one or more FIFO instructions can be moved in the instruction scheduler of processor 164. If so, the FIFO scheduler 142 can determine a new order of the FIFO instructions. The FIFO scheduler 142 can also, for example, determine a new order of other non-FIFO instructions based on the new order of the FIFO instructions.

[0030] In some implementations, compiler 140 generates or receives data of a directed graph defining instructions for program 112. For example, compiler 140 may receive program 112 and convert it into a compiler intermediate representation. The compiler intermediate representation may be in the form of a static, single allocation, for example, prior to register allocation. Compiler 140 may generate the graph based on the instruction set of the program in the compiler intermediate representation. As described above, instructions are vertices or nodes in the graph, and dependencies are edges between vertices or nodes. The graph may include nodes for FIFO instructions and non-FIFO instructions, such as operations, transferring data to hardware processors not modeled as FIFO queues, storing data, etc.

[0031] The FIFO scheduler 142 can analyze the graph and determine the order of individual FIFO instruction groups. As described above, an individual FIFO instruction group is a group of one or more pops from a FIFO, corresponding to one or more pushes onto the same FIFO. Since the processing unit 160 may include multiple hardware FIFOs, the FIFO scheduler 142 can determine the order of FIFO instructions for each hardware FIFO, for example, one at a time.

[0032] The FIFO scheduler 142 can remove control dependencies between individual FIFO instruction groups in the diagram. The FIFO scheduler 142 can preserve control dependencies within individual FIFO instruction groups as they are, because these instructions do not need to be reordered by the FIFO scheduler 142. The FIFO scheduler 142 can then determine the order of the individual FIFO instruction groups for each hardware FIFO.

[0033] In some implementations, the FIFO scheduler 142 evaluates instructions and determines the order of individual FIFO instruction groups for each hardware FIFO, one at a time (e.g., in a specific order). For example, the FIFO scheduler 142 may determine the order of individual FIFO instruction groups for each hardware FIFO sequentially based on the performance advantage of the hardware FIFO in program 112.

[0034] To determine the order in which hardware FIFOs are evaluated, FIFO scheduler 142 can determine a performance metric for each hardware FIFO in program 112. The hardware FIFO performance metric can represent the degree of performance advantage of the operations performed by the hardware FIFO in program 112. In some implementations, the hardware FIFO performance metric is based on, for example, proportionality to how long it takes to perform all pushes and pops of the hardware FIFO. For example, the FIFO performance metric can be based on an estimated or actual duration (e.g., proportional to) the execution of all pushes and pops, which can be based on the hardware FIFO throughput, the computation performed by the hardware FIFO on the data, and / or the amount of data on which the hardware FIFO performs the computation. Since pops are used to output data from computations performed by the hardware FIFO using the data pushed into the corresponding input, pops must wait until the computation is complete. Therefore, the hardware FIFO performance metric can be based on how long it takes to perform the computation, for example, based on the hardware FIFO throughput and the amount of computation.

[0035] The FIFO scheduler 142 can evaluate each hardware FIFO based on performance metrics, in order from the most performance-advantaged hardware FIFO (e.g., longest complete push and pop) to the least performance-advantaged FIFO (e.g., shortest complete push and pop). The order of the individual FIFO instruction groups of each hardware FIFO becomes a constraint on the other hardware FIFOs. Since the most performance-advantaged hardware FIFOs can have the greatest impact on the overall performance of program 112, the constraints imposed on the most performance-advantaged hardware FIFOs should be fewer than those imposed on the least performance-advantaged hardware FIFOs. Determining the order of the individual FIFO instruction groups of each hardware FIFO based on performance metrics in this order can result in better instruction scheduling with reduced latency for program 112 compared to other evaluation orders.

[0036] To determine the order of individual FIFO instruction groups for a given hardware FIFO, the FIFO scheduler 142 can determine a path length metric for each individual FIFO instruction group of the hardware FIFO. The path length metric for an individual FIFO instruction group can represent how long it will take to reach the end of program 112 from the individual FIFO instruction group. For example, the path length metric can be based on the length of the longest path from the individual FIFO instruction group to the end of program 112 (e.g., to the end of the graph representing the instructions of program 112). The length of each path can be based on the length of the edges along the path, since edges represent the latency between instructions. The length of each path can also be based on the instructions along each path, the latency of the hardware processor(s) that will execute the instructions, and / or the throughput of the hardware processor(s), since these parameters affect how long it takes to traverse the path to the end of program 112. For example, the length of each path can be based on the aggregate length of the edges along the path and the time it will take to execute each instruction on the path based on parameters (e.g., an estimated duration) (e.g., proportional to the aggregate length of the edges along the path and the time it will take to execute each instruction on the path based on parameters).

[0037] The FIFO scheduler 142 can classify the independent FIFO instruction groups of the hardware FIFO based on the path length determined for each independent FIFO instruction group. The FIFO scheduler 142 can determine the order of the independent FIFO instruction groups of the hardware FIFO based on the determined path length. For example, the FIFO scheduler 142 can sort the independent FIFO instruction groups of the hardware FIFO from longest path length to shortest path length. That is, the FIFO scheduler 142 can determine the order such that the independent FIFO instruction group with the longest determined path length (e.g., the shortest path length is the longest among the independent groups) is first for the hardware FIFO, the independent group with the second longest path length is second, and so on.

[0038] However, this ordering must respect the control dependencies of the hardware FIFO and any data dependencies between the independent FIFO instruction groups of the hardware FIFO. For example, if the independent FIFO instruction group with the longest path length pushes data output by different independent FIFO instruction groups into the hardware FIFO, then the different independent FIFO instruction groups can be scheduled before the independent FIFO instruction group with the longest path length.

[0039] After determining the order of the individual FIFO instruction groups for the hardware FIFOs, the FIFO scheduler 142 can add control dependencies to the graph based on the determined order. The FIFO scheduler 142 can then move to the next hardware FIFO with the best performance and determine the order of its hardware FIFO instructions. The FIFO scheduler 142 can continue this process until all hardware FIFOs have been evaluated and, if appropriate, a new order of the individual FIFO instruction groups for the hardware FIFOs has been determined.

[0040] In some implementations, the FIFO scheduler 142 may also use scheduling heuristics, such as the critical path on the graph and the interdependencies between hardware FIFOs, when determining the order of individual FIFO instruction groups. For example, the FIFO scheduler 142 may order individual FIFO instruction groups in a manner that shortens the critical path of the graph, which reduces the overall latency of program 112.

[0041] In some implementations, the FIFO scheduler 142 evaluates the graph to identify individual FIFO instruction groups on the critical path of the graph. The critical path of the graph may be the longest path from the beginning to the end of the graph. The length of the path used to determine the critical path may be based on how long it takes to traverse the path, for example, using the length of the edges, the latency of the hardware FIFOs on the path, the throughput of the hardware FIFOs on the path, and / or other appropriate data, as described above.

[0042] The FIFO scheduler 142 can evaluate each individual FIFO instruction group on the critical path to determine whether reordering other individual FIFO instruction groups will shorten the critical path. If so, the order of FIFO instructions not on the critical path can be reordered to shorten the critical path.

[0043] For example, a first hardware FIFO (FIFO1) may have multiple independent FIFO instruction groups, each including one or more pushes and one or more pops. A second hardware FIFO (FIFO2) may have an independent FIFO instruction group, which is on the critical path of program 112 and has a push of the result of a pop from FIFO1. If the pop from FIFO1 is not the first in the sequence of FIFO instructions of FIFO1, the FIFO scheduler 142 can determine a new sequence of FIFO instructions of FIFO1 such that the pop of the data required by FIFO2 is the first in that new sequence. This reduces the amount of time that FIFO2 on the critical path must wait for the required data to perform its computation, which reduces the overall latency of program 112.

[0044] If the output of other independent FIFO instruction groups from FIFO1 is input to other FIFOs on the critical path of the program, then the FIFO scheduler 142 can determine the order of these instructions based on when other FIFOs need data and / or the performance advantage of other FIFOs in program 112. For example, if another FIFO is more performance-advantageous than FIFO2 and requires the output of a different independent FIFO instruction group from FIFO1, then for FIFO1, that different independent group could be the first in the order.

[0045] In some implementations, the FIFO scheduler 142 may omit or combine some FIFO operations on the hardware processor 164 to improve efficiency. For example, when the hardware processor 164 is configurable, changing its configuration requires additional instructions. Executing these additional instructions can be expensive in terms of processing requirements and / or latency. In some cases, the configuration of the hardware processor 164 may be periodically switched between configurations based on operations to be performed by the hardware processor 164 and / or data input to the hardware processor 164 using push instructions.

[0046] If multiple FIFO operations sharing the same configuration can be co-located in the graph or in a schedule generated by the graph, the frequency of switching the configuration of hardware processor 164 can be reduced, resulting in faster and more efficient computation. FIFO scheduler 142 can identify FIFO instructions of hardware processor 164 that will change the configuration of hardware processor 164 during program execution.

[0047] For each such hardware processor 164, the FIFO scheduler 142 can identify candidate FIFO instructions that can be grouped together. For example, for each configuration, the FIFO scheduler 142 can identify each FIFO instruction of the hardware processor 164 that requires the hardware processor 164 to have that configuration. The FIFO scheduler 142 can then determine whether grouping candidate FIFO instructions (or at least a portion of FIFO instructions with the same configuration) will result in a reduction in the latency of program 112. For example, the FIFO scheduler 142 can determine whether adjusting the position of one or more FIFO instructions in the graph to group FIFO instructions with the same configuration will negatively impact the critical path of the graph, for example, whether the grouping will increase the length of the critical path. If so, the FIFO scheduler 142 can determine not to group the FIFO instructions. If not, the FIFO scheduler 142 can group the FIFO instructions in the graph together such that all FIFO instructions in the group are executed without having to switch the configuration of the hardware processor 164.

[0048] Another example of a shared FIFO instruction is used in a hardware processor that processes data larger than its input data. For example, hardware processor 164 could be configured to process 32-bit data, but the input data is only 16-bit data. The two 16-bit inputs can be combined, and hardware processor 164 can process the combined data. However, in this example, combining the data could also negatively impact the critical path of the graph.

[0049] FIFO scheduler 142 can identify FIFO instructions for hardware processor 164, where hardware processor 164 processes data with a larger data size than the data input to hardware processor 164 by the FIFO instructions. Then, for each such hardware processor 164, FIFO scheduler 142 can determine whether adding instructions to combine data and / or whether making the FIFO instructions wait for some data to be combined would negatively affect the critical path of the graph (e.g., by extending the length of the critical path). If yes, FIFO scheduler 142 can determine not to combine data for hardware processor 164. If not, FIFO scheduler 142 can adjust the graph to combine inputs to hardware processor 164, for example, by adding instructions to combine inputs and edge data from input sources to the instructions used to combine that data.

[0050] When determining whether an independent FIFO instruction group should be co-located or grouped with other independent FIFO instruction groups, the FIFO scheduler 142 can search for potential candidates (e.g., candidate FIFO instructions with the same configuration as those processed by the hardware processor or with a smaller data size). The FIFO scheduler 142 can then determine where the critical path from the independent FIFO instruction group to the end of the graph will still include control dependency edges around the independent FIFO instruction group. That is, this improvement is made when program 112 is constrained by the control dependencies of the FIFO hardware processor.

[0051] After determining the order of FIFO instructions, the FIFO scheduler 142 can generate a schedule of instructions that will be compiled by compiler 140 and executed by hardware processor 164, such as a linear schedule. As mentioned above, in addition to FIFO instructions, the graph can also include non-FIFO instructions. Non-FIFO instructions typically have shorter latency than FIFO instructions and generally do not have similar ordering constraints. Therefore, critical path heuristics is less useful for scheduling non-FIFO instructions.

[0052] The FIFO scheduler 142 can cluster non-FIFO instructions with FIFO instructions. In some implementations, the FIFO scheduler 142 clusters each non-FIFO instruction with the FIFO instruction in the graph that is closest to the non-FIFO instruction. As described above, the length of the edges in the graph can represent the waiting time between instructions. The FIFO scheduler 142 can evaluate the length of the edges between a non-FIFO instruction and one or more FIFO instructions (e.g., one or more FIFO instructions connected to a non-FIFO instruction) to determine the FIFO instruction closest to the non-FIFO instruction. For example, if there are intermediary instructions between non-FIFO and FIFO instructions, the graph can include multiple edges between non-FIFO and FIFO instructions. In this case, the FIFO scheduler 142 can aggregate the lengths of the edges (or the waiting times represented by the edges) to determine the total length between non-FIFO instructions. The FIFO scheduler 142 can then cluster the non-FIFO instructions with independent groups of FIFO instructions that include the FIFO instructions with the shortest distance from the non-FIFO instructions. The FIFO scheduler 142 can do this for each non-FIFO instruction until all non-FIFO instructions are in the cluster containing FIFO instructions.

[0053] The FIFO scheduler 142 can classify instruction groups using a critical path heuristic method. For example, for each cluster of instructions, the FIFO scheduler can determine the length of the critical path from the cluster's FIFO instructions to the end of the program. This critical path can represent the longest path from the cluster's FIFO instructions to the end of the program. The FIFO scheduler 142 can then schedule the clusters of instructions based on the length of the critical path. For example, the FIFO scheduler 142 can schedule the clusters of instructions in order from longest to shortest length.

[0054] When a program involves multiple hardware FIFOs and there are data dependencies between them, this method can lead to a large effective range of registers if the utilization rates of the different FIFOs differ. For example, if one out of every three results popped from FIFO1 is pushed into FIFO2, the critical path to push into FIFO2 is much shorter than the critical path to pop from FIFO1. Therefore, after classification based on the length of the critical path, the effective range of pops can be very long. In this case, without increasing the critical path of the graph, the FIFO scheduler 142 can move data-consuming FIFO instructions closer to their corresponding data-generating FIFO instructions. For example, the FIFO scheduler 142 can move data-generating pushes to the earliest position in the graph that can move data-generating pushes without increasing the critical path length of the graph.

[0055] The FIFO scheduler 142 can also use one or more scheduling techniques to schedule instructions within each cluster. For example, the FIFO scheduler 142 can use a list scheduling technique with a graphical topology and feature-determined priorities, based on a hardware architecture of the hardware processor 164.

[0056] Compiler 140 can complete the compilation based on instruction scheduling and provide compiler 150 to processing unit 160. Hardware processor 164 of processing unit 160 can execute compiler 150. For example, hardware processor 164 can execute FIFO instructions and non-FIFO instructions respectively according to the scheduling order determined as defined by compiler 150.

[0057] Figure 2 A directed graph 205 depicts the instructions of the program, and an update graph 235 depicts the instructions of the program. Directed graph 205 may be part of a graph representing dependencies between the instructions of the program. In this example, graph 205 includes separate FIFO instruction groups A1-A4 for a first hardware FIFO (FIFO1). The separate FIFO instruction groups A1-A4 are in a first order 210 based on a first version of the program (e.g., the version of the program received from the user equipment). Graph 205 also includes separate FIFO instruction groups B1-B4 for the first hardware FIFO (FIFO1). The separate FIFO instruction groups B1-B4 are also in a first order 220 based on the first version of the program (e.g., the version of the program received from the user equipment).

[0058] The FIFO scheduler 142 can determine a new order of FIFO instructions for each FIFO based on the performance advantage of FIFO1 and FIFO2 in the program, for example, one at a time. In this example, it is assumed that FIFO2 has a performance advantage over FIFO1.

[0059] The individual FIFO instruction groups for each FIFO can be ordered based on the length of the longest path from each individual FIFO instruction group to the end of the program. In this example, each individual FIFO instruction group has an associated estimated duration based on the longest path of the individual FIFO instruction group. For example, the individual FIFO instruction group A1 has an estimated duration of 2.1 milliseconds (ms). As mentioned above, the duration can be the estimated duration of the traversal path.

[0060] The FIFO scheduler 142 can determine the new order 225 of the individual FIFO instruction groups B1-B4 based on the path length or estimated duration. In this example, the new order 225 of the individual FIFO instruction groups B1-B4 is from the longest duration to the shortest duration. Since the individual FIFO instruction group B4 has the longest duration (4.9 ms), it is the first in the new order 215 of updating Figure 235.

[0061] The FIFO scheduler 142 can also determine a new order 215 for the independent FIFO instruction groups A1-A4 based on path length or estimated duration. For example, the FIFO scheduler 142 can determine a new order 215 based on estimated duration, similar to the new order 225 for the independent instruction groups B1-B4. The FIFO scheduler 142 can also consider the critical path heuristic and the interdependencies between FIFOs. In this example, Figure 205 includes the data dependency between the independent FIFO instruction groups A4 and B1, as shown by edge 212. Therefore, B1 must wait for output data to be popped from A4 in order to push the data into its hardware processor. To reduce this waiting time, the FIFO scheduler 142 can place A4 first in the new order and then sort the other independent FIFO instruction groups A1-A3 based on their estimated duration. In this example, the new order 215 has A4 first, followed by A3, A1, and A2.

[0062] Figure 3 This is a flowchart illustrating an exemplary process 300 for modifying a program. The operation of process 300 can be performed by a system including one or more data processing devices. For example, the operation of process 300 can be performed by… Figure 1 The program processing system 130 executes.

[0063] The system receives a program (310) to be executed by a processing unit including a hardware processor. The program may include one or more workloads, such as machine learning workloads, digital signal processing workloads, video encoding / decoding workloads, or other suitable workloads. For example, the program may define a directed computation graph or in the form of a directed computation graph that specifies the operators of the workload.

[0064] The processing unit may include one or more long-latency, high-throughput hardware processors that can be modeled as a FIFO queue. The processing unit may also include other types of low-latency hardware processors that are not modeled as a FIFO queue.

[0065] The system determines a new order (320) of independent FIFO instruction sets to be executed by each of one or more hardware processors. For example, the system may receive or determine a compiler intermediate representation of the program's source code based on the hardware processor of the processing unit. This compiler intermediate representation of the program may include a directed graph of instructions to be executed by the hardware processor. The graph may also indicate the hardware processor to execute each instruction.

[0066] The system can identify each hardware processor modeled as a FIFO queue in the graph as one or more hardware processors. These processors are also referred to as hardware FIFOs in this document. For each hardware FIFO, the graph may include one or more independent FIFO instruction groups. For hardware FIFOs comprising multiple independent groups, the system can evaluate the independent instruction groups of each hardware FIFO to determine whether a new order of the independent FIFO instruction groups of the hardware FIFOs will reduce program latency. If so, the system can determine a new order of the hardware FIFOs. The system can use composition operations 321-324 to determine a new order of FIFO instructions for each of the one or more hardware FIFOs. If the one or more hardware processors comprise multiple hardware processors, the system can evaluate each hardware processor sequentially based on the performance advantage of the hardware processor in the program, as described above.

[0067] The system identifies the order of the individual FIFO instruction groups for execution by the hardware processor in the data representing the program code (321). The system can then determine the path length metric for each identified individual FIFO instruction group (322).

[0068] A path length metric for individual FIFO instruction groups can represent how long it will take to reach the end of the program from an individual FIFO instruction group. For example, the path length metric can be based on the length of the longest path from an individual FIFO instruction group to the end of the program. As mentioned above, the length of each path can be based on the length of the edges along the path, the instructions along each path, the latency of the hardware processor executing the instructions, and / or the throughput of the hardware processor. For example, the length of each path can be based on the aggregate length of the edges along the path and how long it will take to execute each instruction on the path based on parameters (e.g., an estimated duration) (e.g., proportional to the aggregate length of the edges along the path and how long it will take to execute each instruction on the path based on parameters).

[0069] The system generates a new order of independent FIFO instruction groups based on a path length metric (323). For example, the system can generate a new order such that independent FIFO instruction groups are ordered from independent groups with a path length metric representing the longest path length to independent groups with a path length metric representing the longest path length. As described above, when determining the order of independent FIFO instruction groups of a hardware FIFO, the system can also consider other critical path heuristics and interdependencies between hardware FIFOs. The system can update the graph based on the determined order and add control dependencies between FIFO instructions based on the new order.

[0070] The system determines whether there is an additional hardware processor (324) that evaluates the order of independent FIFO instruction groups. If so, the system returns to operation 321 to evaluate another hardware processor. If not, the system generates a schedule of instructions. The instruction schedule may include a linear instruction schedule, and the instructions include both FIFO and non-FIFO instructions.

[0071] The system can sort the independent FIFO instruction groups in the schedule based on the length of the critical path from each independent FIFO instruction group to the end of the updated graph. For example, the order of the independent FIFO instruction groups in the schedule could be from the group with the longest critical path to the group with the shortest critical path. For non-FIFO instructions, the system can group the non-FIFO instructions with the closest independent FIFO instruction group in the graph, as described above.

[0072] The system generates updated code (340) for the program. The system can generate the updated code based on instruction scheduling. Then, the system can generate output code (350) for execution by the hardware processor based on the updated code. For example, the system can compile the code into machine instructions for each hardware processor.

[0073] The system can then provide the output code to each hardware processor (360). While the processing unit is executing the program, the hardware processor can execute the instructions.

[0074] Embodiments of the subject matter and functional operation described in this specification may be implemented using digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their equivalents, or a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination thereof.

[0075] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as an FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), or GPGPU (General-Purpose Graphics Processing Unit).

[0076] As an example, a computer suitable for executing computer programs includes a central processing unit (CPU) that may be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type. Typically, the CPU receives instructions and data from read-only memory (ROM) or random access memory (RAM), or both. Essential components of a computer are the CPU for executing or carrying out instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said mass storage devices, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0077] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by dedicated logic circuitry or incorporated into dedicated logic circuitry.

[0078] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or potentially claimed claims, but rather as descriptions of features specific to particular embodiments that may be specific to a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0079] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in sequence, or to perform all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0080] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: Receive data representing code of a program to be executed by a processing unit including a hardware processor; For each of the plurality of said hardware processors: In the data representing the code of the program, one or more separate groups are identified for first-in-first-out (FIFO) instructions to be executed by the hardware processor; as well as Determine a performance metric that represents the degree of performance advantage of the operations performed by the hardware processor in the program; The evaluation order of the plurality of hardware processors is determined based on the performance metric of each of the plurality of hardware processors; as well as Each of the plurality of hardware processors is evaluated in a determined evaluation order to determine whether the FIFO instructions of each of the plurality of hardware processors should be reordered. For at least one of the plurality of hardware processors, determine a new order of the FIFO instructions of the hardware processor; The updated code for the program is generated at least based on each new order of the FIFO instructions of the plurality of hardware processors; Output code is generated based on the updated code of the program for execution by the hardware processor; as well as The output code is provided to the hardware processor.

2. The method according to claim 1, wherein, Determining the new order of the FIFO instructions of the hardware processor includes: For each independent group of FIFO instructions executed by the hardware processor, a path length metric is determined, representing the time required to reach the end of the program from that independent group of FIFO instructions; and A new order for the independent groups of FIFO instructions executed by the hardware processor is generated, based at least on the path length metric for each independent group of FIFO instructions.

3. The method of claim 1, further comprising identifying one or more hardware FIFO units as the plurality of hardware processors, each of the one or more hardware FIFO units including a hardware processor modeled using a FIFO queue.

4. The method according to claim 1, wherein, The performance metric for each hardware processor is based, at least in part, on how much time the hardware processor will spend completing the one or more independent groups of FIFO instructions of the hardware processor.

5. The method according to claim 1, wherein, The performance metric for each hardware processor is based, at least in part, on how much time will be spent executing all the pushes and pops of the FIFO instructions for that hardware processor.

6. The method according to claim 1, wherein, The performance metric for each hardware processor is based at least in part on (i) the throughput of the hardware processor, (ii) the computation performed by the hardware processor on data, (iii) the amount of data on which the hardware processor performs the computation, or (iv) any combination of (i), (ii) and (iii).

7. The method according to claim 1, wherein, The data representing the code of the program includes a compiler intermediate representation of the program's source code.

8. The method according to claim 1, wherein, The output code includes compiled machine code that can be executed by the hardware processor.

9. The method according to claim 1, wherein, For at least one of the plurality of hardware processors, determining the new order of the FIFO instructions of the hardware processor includes: Determine the new order of FIFO instructions for the first hardware processor; and For each of one or more subsequent hardware processors in the determined evaluation order, a new order of FIFO instructions for that subsequent hardware processor is determined by subjecting it to one or more constraints on the order of FIFO instructions of that subsequent hardware processor based on the new order of FIFO instructions of the first hardware processor.

10. The method according to claim 9, wherein, Evaluating each of the plurality of hardware processors in the determined evaluation order reduces a certain number of constraints imposed on the new order of FIFO instructions of the hardware processor having a high performance metric relative to the low performance metric, which would otherwise be imposed on the new order of FIFO instructions of the hardware processor having the high performance metric by the new order of FIFO instructions of the hardware processor having the low performance metric.

11. A system for optimizing hardware FIFO instructions, the system comprising: Data processing device; as well as A memory storage device that communicates with the data processing device, the memory storage device storing instructions that can be executed by the data processing device and, upon execution, cause the data processing device to perform operations, the operations including: Receive data representing code of a program to be executed by a processing unit including a hardware processor; For each of the plurality of said hardware processors: In the data representing the code of the program, one or more independent groups are identified for FIFO instructions to be executed by the hardware processor; and Determine a performance metric that represents the degree of performance advantage of the operations performed by the hardware processor in the program; The evaluation order of the plurality of hardware processors is determined based on the performance metric of each of the plurality of hardware processors; and Each of the plurality of hardware processors is evaluated in a determined evaluation order to determine whether the FIFO instructions of each of the plurality of hardware processors should be reordered. For at least one of the plurality of hardware processors, determine a new order of the FIFO instructions of the hardware processor; The updated code for the program is generated at least based on each new order of the FIFO instructions of the plurality of hardware processors; Output code is generated based on the updated code of the program for execution by the hardware processor; and The output code is provided to the hardware processor.

12. The system according to claim 11, wherein, Determining the new order of the FIFO instructions of the hardware processor includes: For each independent group of FIFO instructions executed by the hardware processor, a path length metric is determined, representing the time required to reach the end of the program from that independent group of FIFO instructions; and A new order for the independent groups of FIFO instructions executed by the hardware processor is generated, based at least on the path length metric for each independent group of FIFO instructions.

13. The system according to claim 11, wherein, The operation also includes identifying one or more hardware FIFO units as the plurality of hardware processors, each of the one or more hardware FIFO units including a hardware processor modeled using a FIFO queue.

14. The system according to claim 11, wherein, The performance metric for each hardware processor is based, at least in part, on how much time the hardware processor will spend completing the one or more independent groups of FIFO instructions of the hardware processor.

15. The system according to claim 11, wherein, The performance metric for each hardware processor is based, at least in part, on how much time will be spent executing all the pushes and pops of the FIFO instructions for that hardware processor.

16. The system according to claim 11, wherein, The performance metric for each hardware processor is based at least in part on (i) the throughput of the hardware processor, (ii) the computation performed by the hardware processor on data, (iii) the amount of data on which the hardware processor performs the computation, or (iv) any combination of (i), (ii) and (iii).

17. The system according to claim 11, wherein, The data representing the code of the program includes a compiler intermediate representation of the program's source code.

18. The system according to claim 11, wherein, The output code includes compiled machine code that can be executed by the hardware processor.

19. The system according to claim 11, wherein, For at least one of the plurality of hardware processors, determining the new order of the FIFO instructions of the hardware processor includes: Determine the new order of FIFO instructions for the first hardware processor; and For each of one or more subsequent hardware processors in the determined evaluation order, a new order of FIFO instructions for that subsequent hardware processor is determined by subjecting it to one or more constraints on the order of FIFO instructions of that subsequent hardware processor based on the new order of FIFO instructions of the first hardware processor.

20. A non-transitory computer storage medium encoded with a computer program, the program including instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations, the operations including: Receive data representing code of a program to be executed by a processing unit including a hardware processor; For each of the plurality of said hardware processors: In the data representing the code of the program, one or more separate groups are identified for first-in-first-out (FIFO) instructions to be executed by the hardware processor; as well as Determine a performance metric that represents the degree of performance advantage of the operations performed by the hardware processor in the program; The evaluation order of the plurality of hardware processors is determined based on the performance metric of each of the plurality of hardware processors; as well as Each of the plurality of hardware processors is evaluated in a determined evaluation order to determine whether the FIFO instructions of each of the plurality of hardware processors should be reordered. For at least one of the plurality of hardware processors, determine a new order of the FIFO instructions of the hardware processor; The updated code for the program is generated at least based on each new order of the FIFO instructions of the plurality of hardware processors; Output code is generated based on the updated code of the program for execution by the hardware processor; as well as The output code is provided to the hardware processor.

Images