Executing computational graphs in graphics processing units

Abstract

A computer program for generating a data entity from a computational graph, comprising instructions that, when executed by a computer, cause it to perform steps, including: Receiving data that identifies a computation graph, where the computation graph contains multiple nodes representing operations and edges representing input and output relationships between the nodes; Obtaining compilation artifacts for processing the computational graph in a graphics processing unit, wherein the compilation artifacts comprise machine code, the machine code for the nodes of the computational graph, a sequence of operations in the computational graph, and certain array sizes of operations in the computational graph are generated; and Generating a data entity from the compilation artifacts, where, when invoked, the data entity causes the graphics processing unit to process the computational graph by performing the operations represented by the multiple nodes.

Description

BACKGROUND

[0001] This application relates to the processing of computational graphs representing neural networks.

[0002] Neural networks are machine learning models that use one or more layers of models to generate an output, such as one or more classifications, from a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer of the network. Each layer of the network generates an output for that layer from a received input, in accordance with the current values ​​of its respective set of parameters.

[0003] In accordance with the requirements of the Utility Model Act, only devices as defined in the accompanying claims, but not methods, are to be protected as utility models and are the subject matter of the utility model. If the description refers to methods, these references serve only to illustrate the device or devices for which protection is sought under the accompanying claims. SUMMARY

[0004] This application generally describes a system implemented as computer programs on one or more computers at one or more locations, which generates a data entity. When the data entity is invoked, it causes a graphics processing unit (GPU) to process a computational graph by performing operations associated with the computational graph.

[0005] In general, an innovative aspect of the subject matter described in this application can be embodied in a process of operations comprising: receiving data identifying a computation graph, wherein the computation graph contains multiple nodes representing operations; obtaining compilation artifacts for processing the computation graph in a processing unit; and generating a data entity from the compilation artifacts, wherein, when invoked, the data entity causes the processing unit to process the computation graph by executing the operations represented by the multiple nodes.

[0006] Optionally, the foregoing and other embodiments may each include one or more of the following features, alone or in combination. In particular, one embodiment includes all of the foregoing features together. The compilation artifacts further include: buffer data representing multiple buffer parameters and mappings between the multiple buffer parameters and the operations, each of the multiple buffer parameters being associated with a respective operation of the operations. The compilation artifacts further include: respective logical numbers assigned to the multiple buffer parameters and data specifying mappings between the logical numbers and the operations represented by the multiple nodes. The compilation artifacts further include: a data structure representing (i) a sequence of operations and (ii) dependencies between the operations. The data structure is a directed acyclic graph (DAG).Furthermore, the method includes maintaining multiple libraries, each of which stores one or more subroutines. The compilation artifacts also include: library data representing multiple buffer parameters and mappings between these parameters and the libraries, with each parameter corresponding to a specific library. Additionally, the compilation artifacts include: machine code configured to process the computation graph. The operations are those performed by processing an input to a neural network through one or more layers of the neural network to generate an output. The operations are those performed to train a neural network by adjusting the values ​​of its parameters. The processing unit is a GPU or a central processing unit (CPU).

[0007] In general, another innovative aspect of the subject matter described in this application can be embodied in a process of operations comprising: receiving input buffer parameters that are user-specific inputs to a computation graph, wherein the computation graph contains multiple nodes representing operations; receiving a data entity containing buffer data that represents (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations; and calling the data entity containing the input buffer parameters to cause a processing unit to process the computation graph in accordance with the input buffer parameters by executing the operations.

[0008] Optionally, the foregoing and other embodiments may each include one or more of the following features, alone or in combination. In particular, one embodiment includes all of the foregoing features together. Calling the data entity with the input buffer parameters includes: identifying one or more operations from the computation graph based on the data entity, wherein the one or more operations correspond to the input buffer parameters, and queuing the one or more operations into the processing unit to process the computation graph.The operations contain multiple groups of operations, and the data entity contains a data structure that represents (i) multiple data streams, each representing a sequence of operations, and (ii) dependencies between the operations. Calling the data entity with the input buffer parameters includes: identifying multiple data streams from the computation graph based on the data entity, with each data stream associated with a specific group of operations; enqueuing the respective group of operations into the data stream associated with that group of operations for each of the multiple data streams; and defining dependencies between the multiple groups of operations based on the data entity. The processing unit is a GPU or a CPU.

[0009] The subject matter described in this application can be implemented in certain embodiments to realize one or more of the following advantages. A conventional compilation system requires a host CPU to process operations. In the conventional compilation system, a GPU is idle while the host CPU processes some operations. Thus, the conventional compilation system uses the GPU's hardware resources inefficiently. Unlike the conventional compilation system, the system described in this application generates a data entity containing all the necessary descriptions for executing operations. In particular, the data entity is generated at compile time and can then be called at runtime to cause the GPU to process the computation graph.This allows all operations represented by the computation graph to be integrated into the GPU as a whole, thus improving the efficiency of the machine containing a GPU.

[0010] The details of one or more embodiments of the subject matter of this application are set out in the accompanying drawings and in the following description. Further features, aspects and advantages of the subject matter will become apparent from the description, the drawings and the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A presents an exemplary computational graph system for generating a data entity from a computational graph. Fig. 1B presents exemplary machines for processing a computational graph. Fig. Figure 2 is a flowchart of an exemplary process for generating a data entity from a computational graph. Fig. Figure 3 is a flowchart of an exemplary process for calling a data entity to process a computation graph by performing operations represented by the computation graph. Fig. Figure 4 is a flowchart of an exemplary process for calling the data entity to process a computation graph by performing operations represented by the computation graph.

[0011] Identical reference symbols and designations in the different drawings refer to the same elements. DETAILED DESCRIPTION

[0012] This application generally describes a computational graph system that creates a data entity which, when invoked, causes a graphics processing unit (GPU) to process a computational graph by performing operations represented by the computational graph. Specifically, the data entity as a whole is queued into a GPU, and the GPU processes the computational graph in accordance with the input buffer parameters when the data entity is invoked with those parameters.

[0013] Fig. 1A presents an exemplary computational graph system 100 for generating a data entity from a computational graph. The computational graph system 100 is an example of a system implemented as computer programs on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0014] A user of client 102 can request that operations be performed on a computation graph representing a neural network. The client 102 can be an application running on a computer. As part of the request, the client 102 provides data that identifies a computation graph for the computation graph system 100 and specifies the types of operations to be performed on the computation graph. For example, the request can identify a computation graph representing an inference for a particular neural network and identify an input on which to perform the inference. Alternatively, the request can identify a computation graph representing a neural network to be trained and an input, such as training data, on which to perform the training.

[0015] The computation graph system 100 receives data as input, including a computation graph. The computation graph represents computations of a machine learning model with nodes representing operations. Specifically, a computation graph expresses computations, such as those of a machine learning model, with nodes representing operations and directed edges representing data dependencies between operations. An incoming edge to a node represents a flow of input into the node, i.e., an input argument into the operation represented by the node. If all arguments required for an operation are available in the operation node, the node is released and the operation can be executed.

[0016] An outgoing edge from a node represents a flow of output from the operation represented by that node, which is to be used as input to an operation represented by another node. Thus, a directed edge connecting a first node in the graph to a second node in the graph indicates that an output produced by the operation represented by the first node is used as input to the operation represented by the second node.

[0017] In some implementations, the operations represented in the computation graph are operations of linear algebra, such as matrix multiplication, neural network operations, or operations for various types of machine learning models. A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks, which, in addition to an output layer, contain one or more hidden layers. The output layer of each hidden layer is used as input to another layer in the network—that is, to another hidden layer, the output layer, or both. Some layers of the network produce an output from a received input in accordance with the current value of a particular set of parameters, while other layers of the network may have no parameters.

[0018] The operations represented by the computation graph can be operations necessary for the neural network to compute a conclusion, i.e., to process an input across the layers of the neural network to generate an output. Additionally, or perhaps more precisely, the operations represented by the computation graph can be operations necessary to train the neural network by executing a training procedure to set the values ​​of the neural network's parameters, for example, to determine trained parameter values ​​from initial values ​​using backward propagation. In some cases, such as during neural network training, the operations represented by the computation graph may include operations performed by multiple replicas of the neural network.

[0019] For example, a layer of a neural network receiving input from a preceding layer can use a parameter matrix to perform matrix multiplication between the parameter matrix and the input. In some cases, the matrix multiplication can be represented as multiple nodes in the computation graph. For instance, matrix multiplication can be broken down into multiple multiplication and addition operations, and each operation can be represented by a different node in the computation graph. The operation represented by each node can produce a corresponding output that flows along a directed edge to a subsequent node. After the operation represented by a final node has produced a result of the matrix multiplication, the result flows along a directed edge to another node.The result is equivalent to an output from the layer of the neural network that performs the matrix multiplication.

[0020] In some other cases, matrix multiplication is represented as a node in the graph. The operations represented by the node can receive inputs on a first directed edge, such as an input tensor, and on a second directed edge, a weight tensor, e.g., a parameter matrix. The node can process the input and the weight tensors, e.g., perform a matrix multiplication of them to output an output tensor on a third directed edge, which is equivalent to an output of the neural network layer.

[0021] Other operations of a neural network, which can be represented by nodes in the computation graph, include other mathematical operations, such as subtraction, division, and gradient calculations; array operations, such as concatenation, joining, dividing, or ordering; and operations of building blocks of a neural network, such as SoftMax, Sigmoid, Rectified Linear Unit (ReLU), or convolutions.

[0022] In an example system, one or more sets of nodes in the computation graph can represent operations that control the data flow through the graph. These sets of nodes can represent, for example, conditional, recursive, and / or iterative control instructions, including: IF statements, while loops, do-while loops, for loops, for-each loops, or nested control flow instructions that contain a combination of these instructions.

[0023] The one or more sets of nodes in the computation graph can represent some operations that can be translated into operations of high-performance libraries containing high-performance implementations of linear algebra, e.g., matrix multiplication, or neural network operations, e.g., reconvolution.

[0024] In an exemplary compilation system, the system merges multiple operations into a single merge operation, which can be translated at code generation time into a call that executes all the merged operations. This merging process produces efficient code for devices such as CPUs or GPUs.

[0025] The computation graph system 100 includes a compiler 103 and a data entity generator 105. The compiler 103 can translate the computation graph of operations to produce machine code which, when executed by one or more facilities, causes the facilities to perform the operations represented by the computation graph. In some implementations, a machine may contain one or more facilities, such as GPUs and CPUs. A GPU may be controlled by a host CPU. The GPU receives various requests from the host program run by the host CPU. For example, the GPU may receive requests to start single-data parallel subroutines concurrently, to mark points of interest in a sequence of startup operations, or to allocate or unallocate GPU memory, which is dedicated memory in the GPU facility.

[0026] In particular, if the device on which a computational graph is to be executed is a GPU, the compiler generates compilation artifacts for processing the computational graph on the GPU. These compilation artifacts contain all the descriptions necessary for executing the operations represented by the nodes of the computational graph. In some implementations, the compilation artifacts include machine code, a data structure, buffer data, and library data.

[0027] Machine code is generated to process the computation graph. This machine code can be generated for specific nodes of the computation graph, for a sequence of operations within the graph, and for the array sizes of operations within the graph. Because this machine code is generated for a specific computation graph, it doesn't need to consider all possible array sizes of operations. This can improve processing speed. The machine code is used by the GPU to process the computation graph. For example, the GPU can use the machine code to execute certain operations represented by the computation graph. In some implementations, the machine code may not contain any CPU code. Even if the machine code doesn't contain CPU code, the GPU can still execute the operations by invoking a data entity.The following details of the data entity are described using . Fig. 1B is described in more detail.

[0028] In some implementations, the compilation artifacts contain the data structure. In some implementations, the data structure can be a DAG. The data structure can represent (i) a sequence of operations represented by the nodes of the computation graph and (ii) dependencies between the operations.

[0029] In some implementations, the compilation artifacts contain buffer data. This buffer data specifies which operation to execute in which buffer when the GPU processes the computation graph by performing the operations represented by the graph. The buffer data represents buffer parameters and mappings between these parameters and operations. In some implementations, the buffer parameters can be logical numbers. For example, each buffer can be assigned a specific logical number, and a particular operation can be performed on a specific buffer by specifying the logical number assigned to that buffer.

[0030] In some implementations, the compilation artifacts contain library data. This library data represents which GPU library should be invoked for a specific operation. In some implementations, the library data can also represent which buffer is used for which GPU library. Like the buffer data, each library can be assigned a logical number, and a specific GPU library can be invoked for a particular buffer by specifying the logical number assigned to that buffer.

[0031] The data entry generator 105 creates a data entity from the compilation artifacts. This data entity can contain all the descriptions necessary to execute the operations represented by the nodes of the computation graph. For example, all the compilation artifacts described above can be bundled together to form the data entity. As a result, the data entity contains the machine code, the data structure (e.g., a DAG), the buffer data, and the library data. In some implementations, properties of the machine code, the data structure, the buffer data, and the library data in the compilation artifacts can be preserved in the data entry. For example, dependencies between the operations in the compilation artifacts can be preserved in the data entry. In some implementations, the data entry can be loaded into a machine's memory.

[0032] The computation graph system 100 provides the data entity for one of the machines 160-166 via a data communication network 114. Examples of machines 160-166 can include a computer system, a personal computer, a mobile device, and a server. Each machine can contain, for example, one or more devices 120-126, such as a GPU and a CPU. Based on Fig. In 1A, machines 160-166 each contain facilities 120-126, such as GPUs. Because the data entity contains all the descriptions necessary to perform the operations, the machine receiving the data entity can instruct a facility, such as a GPU, to process the entire computation graph using the data entity. Since in some implementations all the necessary descriptions, such as those for configuring the GPU and buffers, are determined at compile time and contained within the data entity, the GPU can proceed to process the entire computation graph, thus reducing or avoiding the frequent interaction between the GPU and a host CPU that is typical in traditional systems. When the data entity is invoked, the facility, such as...A GPU computes the computation graph by executing the operations represented by the nodes of the computation graph and generating output. The machine containing the facility can return the output to the computation graph system 100, which can then return the requested output to the client 102. In some implementations, the network 114 can be a local area network (LAN) or a wide area network (WAN). In some implementations, the machines 160-166 may additionally contain memory, such as read / write memory (RAM), for storing instructions and data, and a processor for executing stored instructions. Generally, each machine is a hardware resource that performs operations independently of other machines.

[0033] Fig. Section 1B presents exemplary machines 160-166 and exemplary devices 120-126 for processing a computational graph. Fig. In example 1B, machines 160-166 each contain facilities 120-126, such as GPUs. Machine 160 receives input buffer parameters 130. These input buffer parameters are user-specific inputs to a computation graph. Additionally, machines 160-166 can receive the data entity from the computation graph system 100. In this example, machine 160, which contains facility 120, receives the data entity from the computation graph system 100. Machine 160 invokes the data entity containing the input buffer parameters and causes facility 120 to process the computation graph. Since the data entity is generated from the compilation artifacts containing buffer data, it also contains the buffer data. The data entity contains the input buffer parameters and mappings between the input buffer parameters and the operations.Thus, the operations can be identified by the computation graph based on the input buffer parameters. The operations are queued as a whole in facility 120. When the operations are queued in facility 120, facility 120 processes the computation graph by executing the queued operations and generating an output. Machine 160 returns the output to computation graph system 100, which can then return the output to client 102.

[0034] In some implementations, multiple machines 160-166 can each receive the input buffer parameters 130 and the data entity. In this example, the operations are queued in each of the facilities 120-126, and the facilities 120-126 can process the computation graph by executing the operations concurrently. The facilities 120-126 produce respective outputs, and the machines 160-166 return the outputs to the computation graph system 100.

[0035] Fig. Figure 2 is an example flowchart for generating a data entity from a computational graph. In some implementations, the generation of the data entity is performed in the context of a compilation task before the processing of the computational graph. In other examples, the generation is performed on a host CPU, and the processing of the computational graph based on the generated data entity takes place on a GPU.

[0036] For the sake of expediency, the process 200 is described as being executed by a system of one or more computers located at one or more sites and suitably programmed in accordance with this application. For example, a suitably programmed exemplary computational graph system 100 may consist of Fig. 1A Execute process 200.

[0037] The system receives data that identifies a computation graph (202). The computation graph can contain nodes that represent operations.

[0038] The system receives compilation artifacts for processing the computation graph in a GPU (204). The compilation artifacts are generated by a compiler that compiles the computation graph for processing by the GPU. The compilation artifacts contain descriptions necessary to execute the operations represented by the nodes of the computation graph. For example, the compilation artifacts contain machine code, a data structure representing multiple sequences of operations and dependencies between the operations, buffer data, and library data.

[0039] The system generates a data entity (206) from the compilation artifacts. When the data entity is invoked, it causes the GPU to process the computation graph by executing the operations represented by the multiple nodes. The data entity can contain all the descriptions necessary to execute the operations represented by the nodes of the computation graph. For example, all the compilation artifacts described above can be bundled together to form the data entity. As a result, the data entity contains the machine code, the data structure (e.g., a DAG), the buffer data, and the library data to cause the GPU to process the computation graph by executing the operations. In some implementations, properties of the machine code, the data structure, the buffer data, and the library data can be preserved in the data entry within the compilation artifacts.For example, dependencies between operations in the compilation artifacts may be preserved in the data entry. In some implementations, the data entry can be loaded into a machine's memory.

[0040] Fig. Figure 3 is an exemplary flowchart for invoking a data entity to process a computational graph by executing operations represented by the computational graph. For convenience, process 300 is described as being executed by one or more GPUs. For example, the exemplary machines 160-166 can be derived from Fig. 1B, which are appropriately programmed, execute process 300.

[0041] The machine receives input buffer parameters (302). The machine can contain one or more facilities, such as a GPU and a CPU. The input buffer parameters are user-specific inputs to a computation graph. In some implementations, the machine can receive the input buffer parameters from a user. The computation graph contains several nodes that represent operations.

[0042] The machine receives a data entity (304). The data entity can contain buffer data representing (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations. In some implementations, the machine can receive the data entity from a computation graph system.

[0043] The machine calls the data entity with the input buffer parameters (306). When the data entity is called, it causes a facility, such as a GPU, to process the computation graph in accordance with the input buffer parameters by executing the operations. In some implementations, operations from the computation graph that correspond to the input parameter buffers can be identified based on the data entity. The machine queues the identified operations as a whole for the facility to process the computation graph.

[0044] Fig. Figure 4 is an exemplary flowchart for invoking a data entity to process a computational graph by executing operations represented by the computational graph. For convenience, process 400 is described as being executed by one or more GPUs. For example, the exemplary machines 160-166 can be derived from Fig. 1B, which are appropriately programmed, execute process 400. The machines can contain one or more facilities, such as a GPU and a CPU. In this example, the operations comprise several groups of operations, and the data entity contains a data structure that represents (i) data streams and (ii) dependencies between the operations in the data streams.

[0045] The machine identifies data streams from the computation graph (402) based on the data entity. Each data stream can be assigned to a respective group of operations.

[0046] The machine inserts the respective group of operations for each data stream into the data stream that is associated with that group of operations (404). The machine can identify the respective group of operations for each data stream based on the data entity. Once the machine has identified the respective group of operations for the data streams, it inserts that respective group of operations into each data stream.

[0047] The machine defines dependencies between multiple groups of operations based on the data entity (406). The data entity contains the data structure that represents dependencies between the operations. From the dependencies between the operations, dependencies between the multiple groups of operations can be determined. In some implementations, dependencies can be established for a specific data stream across all data streams. The specific data stream can be defined as a primary data stream.

[0048] Embodiments of the subject matter and functional operations described in this application can be implemented in a digital electronic circuit arrangement, in concretely embodied computer software or computer firmware, in computer hardware, including the structures disclosed in this application and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded in a concrete non-transitory program carrier for execution by or control of the operation of data processing devices. Alternatively or additionally, the program instructions can be implemented in an artificially generated propagated signal, e.g.,The computer storage medium may be encoded in a machine-generated electrical, optical, or electromagnetic signal produced to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a read / write storage device, or a serial-access storage device, or a combination of one or more of these. However, the computer storage medium is not a propagated signal.

[0049] The term "data processing device" encompasses all types of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may include a specialized logic circuit arrangement, such as an FPGA (free programmable logic assembly) or an ASIC (application-specific integrated circuit). Furthermore, in addition to hardware, the device may include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0050] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) may be written in any form of a programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computer environment. A computer program may correspond to a file in a file system, but it need not. A program may be contained in a section of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file designated for the program in question, or in several coordinated files, such as...Files are stored that contain one or more modules, subroutines, or sections of code. A computer program can be used to run on one computer or on multiple computers located at one site or distributed across multiple sites and connected by a communication network.

[0051] As used in this application, an “engine” or “software engine” refers to a software-implemented input / output system that provides an output different from the input. An engine can be a coded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Any engine can be implemented in any suitable type of computing device, such as servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smartphones, or other fixed or portable devices containing one or more processors and computer-readable media. Furthermore, two or more of the engines can be implemented in the same computing device or in different computing devices.

[0052] The processes and logic sequences described in this application can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by processing input data and generating an output. The processes and logic sequences can also be executed by a special logic circuit arrangement, such as an FPGA (a free programmable logic array) or an ASIC (an application-specific integrated circuit), and the devices can also be implemented as such.

[0053] Computers capable of running a computer program contain, for example, general-purpose and specialized microprocessors, or both, or some type of central processing unit (CPU). Generally, a CPU receives instructions and data from read-only memory, read / write memory, or both. The essential elements of a computer are a CPU for executing or carrying out instructions and one or more storage devices for storing instructions and data. Generally, a computer also contains data from one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is functionally coupled to receive data from them, transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be integrated into another device, such as a storage device.be embedded in a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus flash drive (USB flash drive), to name a few.

[0054] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated into a special logic circuit arrangement.

[0055] To provide interaction with a user, embodiments of the subject matter described in this application can be implemented in a computer with a display device, e.g., a CRT monitor (cathode ray tube monitor) or an LCD monitor (liquid crystal display monitor), for displaying information to the user, and with a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user; e.g., feedback provided to the user can be any form of sensory feedback, e.g.,This feedback can be visual, audible, or tactile; and input can be received from the user in any form, including acoustic, speech, or keystroke input. Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0056] Implementations of the subject matter described in this application can be implemented in a computer system that includes a backend component, such as a data server, or a middleware component, such as an application server, or a frontend component, such as a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this application, or any combination of one or more such backend, middleware, or frontend components. The system components can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the Internet.

[0057] A computer system can contain clients and servers. Generally, a client and a server are located remotely and typically interact via a communication network. The client-server relationship arises from computer programs running on the respective computers, which establish a client-server relationship with each other.

[0058] Although this application contains many specific implementation details, these are not to be understood as limitations on the scope of protection of any invention or on what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of certain inventions. Certain features described in this application in the context of separate embodiments may also be implemented together in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in several embodiments or in any suitable combination thereof.Although features may be described above in such a way that they act in certain combinations, and may even initially be claimed in this manner, in addition, one or more features from a claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a sub-combination or to variants of a sub-combination.

[0059] Although operations are shown in the drawings in a specific sequence, this should not be interpreted as requiring that these operations be performed in the specific order shown, or in a continuous sequence, or that all depicted operations be performed to achieve desirable results. Multitasking and parallel processing may be advantageous under certain circumstances. Furthermore, the separation of different system modules and system components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and the described program components and systems can, of course, generally be integrated together into a single software product or packaged into multiple software products.

[0060] Further implementations are summarized in the following examples:

[0061] Example 1. A procedure comprises: receiving data that identifies a computation graph, wherein the computation graph contains multiple nodes representing operations; receiving compilation artifacts for processing the computation graph in a processing unit; and generating a data entity from the compilation artifacts, wherein, when invoked, the data entity causes the processing unit to process the computation graph by performing the operations represented by the multiple nodes.

[0062] Example 2. Method according to Example 1, wherein the compilation artifacts further include: buffer data representing multiple buffer parameters and mappings between the multiple buffer parameters and the operations, wherein each of the multiple buffer parameters is associated with a respective operation of the operations.

[0063] Example 3. Procedure according to Example 2, wherein the compilation artifacts further include: respective logical numbers assigned to the multiple buffer parameters, and data specifying mappings between the logical numbers and the operations represented by the multiple nodes.

[0064] Example 4. Procedure according to one of Examples 1 to 3, wherein the compilation artifacts further include: a data structure that represents (i) a sequence of operations and (ii) dependencies between the operations.

[0065] Example 5. Procedure according to Example 4, where the data structure is a directed acyclic graph.

[0066] Example 6. Method according to one of Examples 1 to 5, further comprising: maintaining several libraries, each of which stores one or more subroutines.

[0067] Example 7. Method according to Example 6, wherein the compilation artifacts further include: library data representing multiple buffer parameters and mappings between the multiple buffer parameters and the multiple libraries, each of the multiple buffer parameters being associated with a respective library of the multiple libraries.

[0068] Example 8. Method according to one of Examples 1 to 7, wherein the compilation artifacts further include: machine code configured to process the computation graph.

[0069] Example 9. Method according to any of Examples 1 to 8, wherein the operations are operations for processing an input to a neural network by one or more layers of the neural network to produce an output of the neural network.

[0070] Example 10. Procedure according to one of Examples 1 to 9, wherein the operations are operations for training a neural network by setting values ​​of parameters of the neural network.

[0071] Example 11. Method according to one of Examples 1 to 10, wherein the processing unit is a graphics processing unit (GPU) or a central processing unit (CPU).

[0072] Example 12. A method comprising receiving input buffer parameters that are user-specific inputs to a computation graph, wherein the computation graph contains multiple nodes representing operations; receiving a data entity containing buffer data that represents (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations; and calling the data entity containing the input buffer parameters to cause a processing unit to process the computation graph in accordance with the input buffer parameters by executing the operations.

[0073] Example 13. Procedure according to Example 12, wherein calling the data entity with the input buffer parameters includes: identifying one or more operations from the computation graph based on the data entity, wherein the one or more operations correspond to the input buffer parameters, and queuing the one or more operations into the processing unit to process the computation graph.

[0074] Example 14. Procedure according to Example 12 or 13, wherein the operations contain multiple groups of operations and wherein the data entity contains a data structure that (i) represents multiple data streams, each data stream representing a sequence of operations, and (ii) represents dependencies between the operations, and wherein calling the data entity with the input buffer parameters includes: identifying multiple data streams from the computation graph based on the data entity, each data stream being associated with a respective group of operations from the operations, enqueuing the respective group of operations into the data stream associated with the respective group of operations for each data stream of the multiple data streams, and defining dependencies between the multiple groups of operations based on the data entity.

[0075] Example 15. Method according to one of Examples 12 to 14, wherein the processing unit is a GPU or a CPU.

[0076] Example 16. System comprising one or more computers and one or more storage storage facilities that store instructions which, when executed by the one or more computers, cause the one or more computers to perform system operations that include: receiving data identifying a computation graph, wherein the computation graph contains multiple nodes representing operations; receiving compilation artifacts for processing the computation graph in a processing unit; and generating a data entity from the compilation artifacts, wherein the data entity, when invoked, causes the processing unit to process the computation graph by performing the operations represented by the multiple nodes.

[0077] Example 17. System according to Example 16, wherein the compilation artifacts further include: buffer data representing multiple buffer parameters and mappings between the multiple buffer parameters and the operations, each of the multiple buffer parameters being associated with a respective operation of the operations.

[0078] Example 18. System according to Example 17, wherein the compilation artifacts further include: respective logical numbers assigned to the multiple buffer parameters, and data specifying mappings between the logical numbers and the operations represented by the multiple nodes.

[0079] Example 19. System according to one of Examples 16 to 18, wherein the compilation artifacts further include: a data structure representing (i) a sequence of operations and (ii) dependencies between the operations.

[0080] Example 20. System according to Example 19, where the data structure is a directed acyclic graph.

[0081] Example 21. System according to one of Examples 16 to 20, which further comprises: maintaining several libraries, each of which stores one or more subroutines.

[0082] Example 22. System according to Example 21, wherein the compilation artifacts further include: library data representing multiple buffer parameters and mappings between the multiple buffer parameters and the multiple libraries, each of the multiple buffer parameters being associated with a respective library of the multiple libraries.

[0083] Example 23. System according to one of Examples 16 to 22, wherein the compilation artifacts further include: machine code configured to process the computation graph.

[0084] Example 24. System according to one of Examples 16 to 23, wherein the operations are operations for processing an input to a neural network by one or more layers of the neural network to produce an output of the neural network.

[0085] Example 25. System according to one of Examples 16 to 24, wherein the operations are operations for training a neural network by setting values ​​of parameters of the neural network.

[0086] Example 26. System according to one of Examples 16 to 25, where the processing unit is a GPU or a CPU.

[0087] Example 27. System comprising one or more computers and one or more storage storage facilities that store instructions which, when executed by the one or more computers, cause the one or more computers to perform system operations that include: receiving input buffer parameters that are user-specific inputs to a computation graph, the computation graph containing multiple nodes that represent operations; receiving a data entity containing buffer data that represents (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations; and calling the data entity containing the input buffer parameters to cause a processing unit to process the computation graph in accordance with the input buffer parameters by performing the operations.

[0088] Example 28. System according to Example 27, wherein calling the data entity with the input buffer parameters includes: identifying one or more operations from the computation graph based on the data entity, wherein the one or more operations correspond to the input buffer parameters, and queuing the one or more operations into the processing unit to process the computation graph.

[0089] Example 29. System according to Example 27 or 28, wherein the operations contain multiple groups of operations and wherein the data entity contains a data structure that (i) represents multiple data streams, each data stream representing a sequence of operations, and (ii) represents dependencies between the operations, and wherein calling the data entity with the input buffer parameters includes: identifying multiple data streams from the computation graph based on the data entity, each data stream being associated with a respective group of operations from the operations, enqueuing the respective group of operations into the data stream associated with the respective group of operations for each data stream of the multiple data streams, and defining dependencies between the multiple groups of operations based on the data entity.

[0090] Example 30. System according to one of Examples 27 to 29, where the processing unit is a GPU or a CPU.

[0091] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in a different sequence and still achieve desirable results. As an example, the processes shown in the accompanying figures do not necessarily require the specific sequence or running order shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0092] What is being claimed is:

Claims

[1] Computer program for generating a data entity from a computational graph, comprising instructions which, when the program is executed by a computer, cause it to perform steps, comprising: Receiving data that identifies a computation graph, where the computation graph contains multiple nodes representing operations and edges representing input and output relationships between the nodes; Obtaining compilation artifacts for processing the computational graph in a graphics processing unit, wherein the compilation artifacts comprise machine code, the machine code for the nodes of the computational graph, a sequence of operations in the computational graph, and certain array sizes of operations in the computational graph are generated; and Generating a data entity from the compilation artifacts, where, when invoked, the data entity causes the graphics processing unit to process the computational graph by performing the operations represented by the multiple nodes. [2] Computer program according to claim 1, wherein the compilation artifacts further include: Buffer data representing multiple buffer parameters and mappings between the multiple buffer parameters and the operations, where each of the multiple buffer parameters is associated with a respective operation of the operations. [3] Computer program according to claim 2, wherein the compilation artifacts further include: respective logical numbers assigned to the multiple buffer parameters, and Data that specifies mappings between the logical numbers and the operations represented by the multiple nodes. [4] Computer program according to any one of claims 1 to 3, wherein the compilation artifacts further include: a data structure that represents (i) a sequence of operations and (ii) dependencies between the operations. [5] Computer program according to claim 4, wherein the data structure is a directed acyclic graph. [6] Computer program according to any one of claims 1 to 5, further comprising: Maintaining multiple libraries, where each of the multiple libraries stores one or more subroutines. [7] Computer program according to claim 6, wherein the compilation artifacts further include: Library data representing multiple buffer parameters and mappings between the multiple buffer parameters and the multiple libraries, where each of the multiple buffer parameters is mapped to a respective library of the multiple libraries. [8] Computer program according to any one of claims 1 to 7, wherein the compilation artifacts further include: Machine code configured to process the computation graph. [9] Computer program according to any one of claims 1 to 8, wherein the operations are operations for processing an input to a neural network by one or more layers of the neural network to produce an output of the neural network. [10] Computer program according to any one of claims 1 to 8, wherein the operations are operations for training a neural network by setting values ​​of parameters of the neural network. [11] Computer program for calling a data entity for processing a computational graph in a graphics processing unit, wherein the program includes instructions which, when executed by a computer, cause it to perform steps, comprising: Receiving input buffer parameters that are user-specific inputs into a computation graph, where the computation graph contains multiple nodes representing operations and edges representing input and output relationships between the nodes; Receiving a data entity containing buffer data representing (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations, wherein the data entity comprises machine code generated for the nodes of the computation graph, a sequence of operations in the computation graph, and certain array sizes of operations in the computation graph; and Calling the data entity with the input buffer parameters to cause a graphics processing unit to process the computational graph in accordance with the input buffer parameters by executing the operations. [12] Computer program according to claim 11, wherein calling the data entity with the input buffer parameters comprises: Identifying one or more operations from the computation graph based on the data entity, where the one or more operations correspond to the input buffer parameters, and Scheduling one or more operations into the graphics processing unit to process the computational graph. [13] Computer program according to claim 11 or 12, wherein the operations include multiple groups of operations and wherein the data entity includes a data structure that (i) contains multiple data streams, each data stream representing a sequence of operations, and (ii) represents dependencies between the operations, and wherein calling the data entity with the input buffer parameters comprises: Identifying multiple data streams from the computation graph based on the data entity, where each data stream is assigned a respective group of operations from the operations, Inserting the respective group of operations into the data stream assigned to that group of operations, for each of the multiple data streams, and Defining dependencies between multiple groups of operations based on the data entity. [14] System comprising one or more computers and one or more storage devices that store instructions which, when executed by the one or more computers, cause the one or more computers to perform system operations which include: Receiving data that identifies a computation graph, where the computation graph contains multiple nodes representing operations and edges representing input and output relationships between the nodes; Obtaining compilation artifacts for processing the computation graph in a processing unit, wherein the compilation artifacts comprise machine code, the machine code for the nodes of the computation graph, a sequence of operations in the computation graph, and certain array sizes of operations in the computation graph are generated; and Generating a data entity from the compilation artifacts, where, when invoked, the data entity causes the graphics processing unit to process the computational graph by performing the operations represented by the multiple nodes. [15] System according to claim 14, wherein the compilation artifacts further include: Buffer data representing multiple buffer parameters and mappings between the multiple buffer parameters and the operations, where each of the multiple buffer parameters is associated with a respective operation of the operations. [16] System according to claim 14 or 15, wherein the compilation artifacts further include: respective logical numbers assigned to the multiple buffer parameters, and Data that specifies mappings between the logical numbers and the operations represented by the multiple nodes. [17] System according to any one of claims 14 to 16, wherein the compilation artifacts further include: a data structure that represents (i) a sequence of operations and (ii) dependencies between the operations. [18] System according to claim 17, wherein the data structure is a directed acyclic graph. [19] System according to any one of claims 14 to 18, further comprising: Maintaining multiple libraries, where each of the multiple libraries stores one or more subroutines. [20] System according to claim 19, wherein the compilation artifacts further include: Library data representing multiple buffer parameters and mappings between the multiple buffer parameters and the multiple libraries, where each of the multiple buffer parameters is mapped to a respective library of the multiple libraries. [21] System according to any one of claims 14 to 20, wherein the compilation artifacts further include: Machine code configured to process the computation graph. [22] System according to any one of claims 14 to 21, wherein the operations are operations for processing an input to a neural network by one or more layers of the neural network to produce an output of the neural network. [23] System according to one of claims 14 to 22, wherein the operations are operations for training a neural network by setting values ​​of parameters of the neural network. [24] System comprising one or more computers and one or more storage devices that store instructions which, when executed by the one or more computers, cause the one or more computers to perform system operations which include: Receiving input buffer parameters that are user-specific inputs into a computation graph, where the computation graph contains multiple nodes representing operations and edges representing input and output relationships between the nodes; Receiving a data entity containing buffer data representing (i) multiple buffer parameters containing the input buffer parameters, and (ii) mappings between the multiple buffer parameters and the operations, wherein the data entity comprises machine code generated for the nodes of the computation graph, a sequence of operations in the computation graph, and certain array sizes of operations in the computation graph; and Calling the data entity with the input buffer parameters to cause a graphics processing unit to process the computational graph in accordance with the input buffer parameters by executing the operations. [25] System according to claim 24, wherein calling the data entity with the input buffer parameters comprises: Identifying one or more operations from the computation graph based on the data entity, where the one or more operations correspond to the input buffer parameters, and Scheduling one or more operations into the graphics processing unit to process the computational graph. [26] System according to claim 24 or 25, wherein the operations include multiple groups of operations and wherein the data entity includes a data structure that (i) represents multiple data streams, each data stream representing a sequence of operations, and (ii) represents dependencies between the operations, and wherein calling the data entity with the input buffer parameters comprises: Identifying multiple data streams from the computation graph based on the data entity, where each data stream is assigned a respective group of operations from the operations, Inserting the respective group of operations into the data stream assigned to that group of operations, for each of the multiple data streams, and Defining dependencies between multiple groups of operations based on the data entity.

Images