Processing a computation graph

Abstract

The present disclosure relates to processing a computation graph. Methods, systems and apparatus, including computer programs encoded on computer storage media, for receiving a request from a client to process a computation graph; obtaining data representing the computation graph, the computation graph comprising a plurality of nodes and directed edges, wherein each node represents a respective operation, wherein each directed edge connects a respective first node to a respective second node, the respective second node representing an operation that receives as input an output of an operation represented by the respective first node; identifying a plurality of available devices for performing the requested operations; dividing the computation graph into a plurality of subgraphs, each subgraph comprising one or more nodes in the computation graph; and for each subgraph, assigning the operations represented by the one or more nodes in the subgraph to respective available devices of the plurality of available devices for operation.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201680063236.7, filed on October 28, 2016. Background Technology

[0003] This specification relates to processing computational graphs representing neural networks and / or using processed computational graphs to process model inputs.

[0004] Neural networks are machine learning models that use one or more layers to generate outputs, such as one or more categories, from received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current values ​​of its corresponding set of parameters.

[0005] The layers of a neural network can be processed by a separate device. This device can have a processor that performs operations, such as generating outputs from inputs at one layer and storing the outputs in memory. Because generating outputs in a neural network typically requires a large number and scale of computations, a single device may spend a significant amount of time processing the layers of the neural network. Summary of the Invention

[0006] Generally speaking, this specification describes a system and method for processing computational graphs representing neural networks or other machine learning models.

[0007] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: receiving a request from a client to process a computation graph; obtaining data representing the computation graph, the computation graph comprising a plurality of nodes and directed edges, wherein each node represents a corresponding operation, wherein each directed edge connects a corresponding first node to a corresponding second node, the corresponding second node representing an operation that receives the output of the operation represented by the corresponding first node as input; identifying a plurality of available devices for performing the requested operation; dividing the computation graph into a plurality of subgraphs, each subgraph comprising one or more nodes in the computation graph; and for each subgraph, assigning the operation represented by one or more nodes in that subgraph to a corresponding available device among the plurality of available devices for operation. The method may be a computer-implemented method.

[0008] The implementation may include one or more of the following features. The request specifies one or more specific outputs from one or more corresponding nodes, further including: receiving the one or more specific outputs from a device to which the one or more corresponding nodes are assigned; and providing the one or more specific outputs to the client. Operations through the computation graph include inference or training operations for a neural network. The request includes labels for partitioning the computation graph into a plurality of predetermined subgraphs, and wherein partitioning the computation graph includes partitioning the computation graph into the plurality of predetermined subgraphs. Each device is a hardware resource that performs operations independently of other devices among the plurality of devices. Assigning each subgraph to a corresponding device includes assigning the subgraph to a device having the computational capabilities required to perform operations represented by the nodes in the subgraph. Analyzing the computation graph to identify groups of nodes arranged in a chain structure; wherein the partitioning includes generating a corresponding subgraph including the identified group of nodes for each identified group. Analyzing the computation graph to identify groups of nodes that perform operations on shared data flowing to the group of nodes on directed edges; wherein the partitioning includes generating a corresponding subgraph including the identified group of nodes for each identified group. The process involves: determining an initial allocation of a subgraph to a device; monitoring the device to determine statistics; using the statistics to adjust the initial allocation; and reallocating the subgraph to a device based on the adjusted initial allocation. This monitoring, adjustment, and reallocation process is repeated until a threshold improvement is reached. The statistics include the corresponding computation time or idle time for each subgraph.

[0009] In another embodiment, the method further includes: receiving model input; and processing the model input according to operations represented by the processed computation graph.

[0010] Another innovative aspect of the subject matter described in this specification can be embodied in a method that may include: providing a machine learning model corresponding to the processed computational graph obtained through the first aspect; and using said machine learning model to process model inputs. Processing the model inputs may constitute training of said machine learning model, or it may constitute generating inference from said model inputs.

[0011] Another innovative aspect of the subject matter described in this specification can be embodied in a method that may include the following actions: the processing of the computational graph obtained through the first aspect is performed by multiple devices.

[0012] In these respects, the computational graph can be a representation of a machine learning model, such as a neural network.

[0013] Another innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: using multiple devices to process model inputs according to a computation graph comprising multiple nodes and directed edges, wherein each node represents a corresponding operation, wherein each directed edge connects a corresponding first node to a corresponding second node, the corresponding second node representing an operation that receives the output of the operation represented by the corresponding first node as input, wherein the method comprises, for each of the multiple devices: receiving data representing a subgraph of the computation graph assigned to the device, the subgraph comprising multiple nodes and directed edges from the computation graph; and performing the operations represented by the nodes in the subgraph.

[0014] Implementations of this aspect may include one or more of the following features. The request to specify one or more specific outputs from one or more corresponding nodes further includes: receiving a request to identify one or more specific outputs from one or more corresponding nodes in the subgraph; and providing the one or more specific outputs to a client. The method further includes monitoring statistics; and providing the statistics to a client. The statistics include a corresponding computation time or a corresponding idle time for each subgraph. Executing the computation represented by a node in the subgraph includes executing the computation asynchronously. Executing the computation asynchronously includes using a queue, a non-blocking kernel, or both to execute the computation.

[0015] Other aspects provide a system and a computer-readable medium corresponding to any of the foregoing aspects. The computer-readable medium may be a non-transitory computer-readable medium, but the invention is not limited thereto.

[0016] Specific embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. The operations of a neural network, such as operations generating inference from input, can be represented as a computational graph of nodes and directed edges. The system processes this computational graph representation to efficiently perform the operations of the neural network. The reason the system achieves this efficiency is that the computational graph has lower abstraction compared to a conventional neural network represented as a series of layers. In particular, the computational graph is easier to partition for parallel operations compared to a conventional neural network representation. For example, subgraphs of the computational graph can be assigned to unique devices; for instance, each subgraph can be assigned to a different device than the other subgraphs, where each device performs the operations in the corresponding subgraph, thereby reducing the overall time required to perform neural network operations.

[0017] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. It will be appreciated that these aspects and embodiments can be combined, and features described in the context of one aspect or embodiment can be implemented in the context of other aspects or embodiments. Attached Figure Description

[0018] Figure 1 The figure illustrates an example computational graph system used for operations on a neural network whose distribution is represented as a computational graph.

[0019] Figure 2 This is a flowchart of an example method for processing computation graphs.

[0020] Figure 3 This is an example computation graph.

[0021] Figure 4 This is a flowchart of an example process for assigning a subgraph to a device.

[0022] Similar icon numbers and markings in the figures indicate similar elements. Detailed Implementation

[0023] This specification generally describes a computation graph system that performs operations represented by a computation graph in a distributed manner.

[0024] A computation graph consists of nodes connected by directed edges. Each node in the computation graph represents an operation. An input edge entering a node represents the input stream entering the node, that is, the input to the operation represented by the node. An output edge leaving a node represents the output stream of the operation represented by the node, which will be used as the input to an operation represented by another node. Therefore, a directed edge connecting a first node in the graph to a second node in the graph indicates that the output generated by the operation represented by the first node is used as the input to the operation represented by the second node.

[0025] Generally, the inputs and outputs flowing along directed edges in a computation graph are tensors. A tensor is a multidimensional array of numbers or other values, such as strings, with a specific order corresponding to the array's dimensions. For example, scalar values ​​are 0th-order tensors, numerical vectors are 1st-order tensors, and matrices are 2nd-order tensors.

[0026] In some implementations, the operations represented in the computation graph are neural network operations or operations used in different types of machine learning models. A neural network is a machine learning model that uses one or more non-linear units to predict outputs from received inputs. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network—another hidden layer, an output layer, or both. Some layers of the network generate outputs from received inputs based on the current values ​​of their respective parameter sets, while other layers may not have parameters.

[0027] For example, the operations represented by a computation graph can be those required for neural network computation and inference; that is, the operations required for the neural network to process inputs through neural network layers to generate neural network outputs in response to those inputs. As another example, the operations represented by a computation graph can be those required to train a neural network by performing a neural network training process to adjust the parameter values ​​of the neural network, for example, to determine trained parameter values ​​from initial parameter values. In some cases, such as during neural network training, the operations represented by a computation graph can include operations performed by multiple copies of the neural network.

[0028] For example, a neural network layer that receives input from preceding layers can perform matrix multiplication between the parameter matrix and the input using the parameter matrix. In some cases, this matrix multiplication can be represented as multiple nodes in a computation graph. For instance, matrix multiplication can be divided into multiple multiplication and addition operations, and each operation can be represented by a different node in the computation graph. The operation represented by each node generates a corresponding output, which flows along directed edges to subsequent nodes. After the operation represented by the final node generates the result of the matrix multiplication, the result flows along directed edges to another node. This result is equivalent to the output of the neural network layer that performed the matrix multiplication.

[0029] In other cases, matrix multiplication is represented as a node in a graph. As input, the operation represented by the node can receive an input tensor on a first directed edge and a weight tensor, such as a parameter matrix, on a second directed edge. The node can process the input and weight tensors, for example, performing matrix multiplication of the input and weight tensors to output an output tensor on a third directed edge, equivalent to the output of a neural network layer.

[0030] Other neural network operations that can be represented by nodes in a computation graph include: other mathematical operations, such as subtraction, division, and gradient calculation; array operations, such as concatenation, splicing, splitting, or sorting; and neural network block operations, such as SoftMax, Sigmoid, Rectified Linear Unit (ReLU), or convolution.

[0031] Representing neural networks as computation graphs provides a flexible and nuanced way to efficiently implement neural networks, especially when the network's operations are distributed across multiple devices with different hardware profiles.

[0032] Figure 1 The figure illustrates an example computational graph system 100 for operations on a neural network whose distribution is represented as a computational graph. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the system, components, and techniques described below can be implemented.

[0033] The user of client 102 can request to perform operations on the computation graph representing the neural network. Client 102 can be an application running on a computer.

[0034] As part of the request, client 102 provides system 100 with data that identifies the computation graph and specifies the type of operation to be performed on the computation graph.

[0035] For example, a request could identify computational graphs representing reasoning for a particular neural network and identify the inputs that respond to it in performing reasoning.

[0036] As another example, the request could identify a computational graph representing the training process for a specific neural network and the inputs, such as training data, required to perform training on it. In this example, upon receiving a request to process the computational graph representing the training process, system 100 could, for example, use conventional backpropagation or other neural network training techniques to determine parameter modification values ​​for one or more edges of the computational graph. System 100 could store the modified parameters in the device's memory, and executor 106 could retrieve and store the address of the modified weights at system 100. Upon further requests from client 102 for inference, training, or other operations requiring weight modification, system 100 could access the modified weights using the address.

[0037] In some cases, a request can specify the response to be sent in response to the request. For example, for a neural network training request, client 102 can request an indication that the requested neural network training operation has been completed, and optionally, an indication of the neural network's parameter training values ​​or the storage location where client 102 can access the training values. As another example, for a neural network inference request, client 102 can request the output values ​​representing the inference operation from one or more specific nodes in the computation graph.

[0038] System 100 performs operations to generate specific outputs by dividing the computation graph represented by multiple devices 116 to 122. System 100 divides the operations across multiple devices 116 to 122 via a data communication network 114, such as a local area network (LAN) or a wide area network (WAN). Devices 116 to 122 perform the operations and, if applicable, return the corresponding output or indication to system 100, which is capable of returning the requested output or indication to client 102.

[0039] Any device performing neural network operations, such as devices 116 to 122, can include memory for storing instructions and data, such as random access memory (RAM), and a processor for executing the stored instructions. Generally, each device is a hardware resource that performs operations independently of other devices. For example, each device can have its own processing unit. These devices can be graphics processing units (GPUs) or central processing units (CPUs). For instance, a machine can host one or more devices, such as multiple CPUs and GPUs.

[0040] Each device can also possess corresponding computing capabilities. That is, devices can have different memory capacities, processing speeds, or other architectural features. Therefore, some devices can perform operations that other devices cannot. For example, some operations require a certain memory capacity that only specific devices possess, or some devices are configured to perform only specific types of operations, such as inference operations.

[0041] The session manager 104 in system 100 receives a request to start a session from the client 102, during which computation graph operations are performed. The session manager 104 manages a set of devices, such as devices 116 to 122, capable of performing computation graph operations, and is able to provide a set of devices available for performing operations to the placer 108.

[0042] For each operation to be performed in the computation graph, the layouter 108 determines the appropriate target device, such as device 116, to perform the operation, and in some embodiments, determines the timing at which the appropriate target device performs the operation. Some operations can be performed in parallel, while others require the completion of prior operations in the computation graph, such as other operations processing the output of prior operations as input.

[0043] After the device performs the operations assigned by the layoutr 108 to generate output, the executor 106 can retrieve the output. The executor 106 can generate an appropriate response to the request, such as output or indication that processing has been completed. The executor 106 can then return the response to the client 102.

[0044] Session manager 104 also provides the set of operations to be executed in the computation graph to executor 106. Executor 106 periodically retrieves computation time statistics from devices 116 to 122 associated with the graph execution of the operations. Executor 106 provides the computation time statistics to layoutr 108, which can then re-optimize the layout and scheduling of further operations. (See below for further details.) Figure 2 This re-optimization will be described further.

[0045] Figure 2 This is a flowchart of an example process 200 for processing a computation graph. For convenience, process 200 is described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed computation graph system, such as... Figure 1 The computational graph system 100 is capable of executing process 200.

[0046] The system receives a request from the client to process the computation graph (step 202). For example, this request could be a request to perform neural network inference represented by the computation graph on a specified input, a request to perform neural network training operations represented by the computation graph on a specified training dataset, or a request to perform other neural network operations represented by the computation graph, as described above. Figure 1 As described.

[0047] The system obtains data representing the computation graph (step 204). In some cases, this data is sent along with a request from the client. In other cases, the request identifies the computation graph and the system retrieves data representing the identified graph from memory. For example, the data representing the graph can be an array of nodes in the graph. Each node can contain information specifying the operation type, name, and a form containing the node's input and output edges.

[0048] The system identifies multiple available devices for performing the requested operation (step 206). The system can connect to several devices, for example, in a data center. The system can maintain the state of each device, for example, using... Figure 1 The actuator 106. Each device may be occupied or available. If a device is currently performing other operations and cannot be allocated further operations or cannot perform graph processing operations, the device is occupied. If a device can be allocated further operations, for example, if further operations can be queued for the device to perform, the device is available.

[0049] The system divides the computation graph into multiple subgraphs (step 208). Each subgraph includes one or more nodes from the computation graph. In some implementations, the request from the client includes a label specifying how the computation graph should be divided into predetermined subgraphs. For example, the user can manually generate labels for the computation graph and include the labels in the request. If the request includes this label, the system divides the computation graph into the predetermined subgraphs.

[0050] In some other implementations, the system partitions the computation graph based on how it is arranged. Specifically, the system can analyze the graph to identify directed edges connecting one or more nodes arranged in a chain structure. Nodes in a chain are connected to each other along a directed edge from node to node. Therefore, a node in a chain must wait for the computation of the preceding node in the chain to complete before it can compute its own operation. (See also...) Figure 3 Further description of the subgraph partitioning.

[0051] In other implementations, the system clusters nodes in the graph and then assigns nodes in the same cluster to the same subgraph. Specifically, the system can analyze the graph to identify nodes that perform operations on shared data flowing along directed edges. For example, multiple nodes may receive the same data as input from a prior node. The system can cluster such nodes that receive the same data in the same subgraph so that when the subgraph is assigned to a particular device, the device can reuse memory to store the same data used for multiple operations represented by the nodes. (See also...) Figure 3 To describe this further.

[0052] The methods by which the system generates subgraphs will be discussed in more detail below.

[0053] For each subgraph, the system allocates the operations represented by one or more nodes in that subgraph to the appropriate available devices (step 210). In some implementations, the system allocates each subgraph to a device with the computing power required to perform the operations represented by the nodes in the subgraph. In some implementations, the request from the client includes user-specified data that identifies a specific type of device to perform operations on a specific node. For example, the user can specify that a specific node with a large number of mathematical operations should be allocated to a GPU. The system can allocate subgraphs including specific nodes to devices of a specific type.

[0054] In some other implementations, the system determines the device to which the subgraph will be allocated by estimating the maximum amount of resources consumed by operations representing the nodes in the subgraph. For example, the system can calculate the maximum memory capacity that any node in the subgraph will consume. In particular, the system can traverse the subgraph to calculate the dimension of the tensor on each directed edge traveling to and from each node in the subgraph. The dimension of the tensor indicates the memory size consumed by the device to perform the operations. The system can then allocate the subgraph to a device with memory capable of storing the maximum number of tensors flowing through the subgraph.

[0055] The following will refer to Figure 4 Another implementation of assigning subgraphs to devices is further described below, and the way the system assigns subgraphs to devices will be discussed in more detail below.

[0056] The system prompts the device to perform the computation assigned to the node on the device (step 212). In some implementations, the system sends a request to each device to begin computation. The device receives the request and, in response, begins performing the computation assigned to the node on that device. In some implementations, the device performs the computation assigned to the node on the device asynchronously. For example, the device can perform the computation asynchronously by using a queue, a non-blocking kernel, or both. The asynchronous execution of computation is described below.

[0057] Figure 3 The diagram illustrates an example computational graph. For instance, a computational graph system, such as... Figure 1 System 100 can receive requests from clients, given a set of inputs, and use the computation graph to compute inference. Specifically, the client can request the output of node 316. This set of inputs can be provided to node 302 via directed edges.

[0058] The system can divide the computation graph into three subgraphs 318 to 322. To generate subgraphs 318 to 322, the system can analyze the computation graph to identify node chains. For example, the system can identify the first chain of nodes 304 and 316, the second chain of nodes 302, 306, and 310, and the third chain of nodes 308, 312, and 314. Although other possible node chains are feasible, the system can select the chain that minimizes the number of subgraphs. The system can then group these node chains into corresponding subgraphs.

[0059] In some implementations, if the outputs of node 306 are identical, the system groups nodes 306, 308, and 310 into a single subgraph. This is because nodes 310 and 308 both receive the same output from node 306. In this case, the operations represented by nodes 310 and 308 are performed on the same device to minimize memory consumption. That is, when operations are performed on nodes 310 and 308, the device can access the same memory location storing the outputs from node 306.

[0060] The system is able to assign three subgraphs 318 to 322 to three corresponding available devices. The system can begin by assigning the first subgraph 322 because it contains the initial node 302 and none of the nodes in it depends on the output of the other subgraphs. Once the first subgraph 322 is assigned, the system is able to assign the second subgraph 318. Node 304 in the second subgraph 318 requires the output of node 302, which will be computed by the device assigned to the first subgraph 322.

[0061] In some implementations, the system waits to allocate the second subgraph 318 until it receives an indication that the operation represented by node 302 has been completed. This allows the system to dynamically allocate the subgraph based on current information such as memory availability or device availability, which improves efficiency. Upon receiving the indication, the system can allocate the second subgraph 318 to a device capable of processing the output size of node 302. In some other implementations, the system analyzes the graph to determine the dimension of the tensors flowing along the directed edges from nodes 302 to 304. The system can then allocate the second subgraph 318 based on the tensor dimension. That is, the system allocates the second subgraph 318 to a device capable of processing the memory requirements of the tensors representing the second subgraph 318.

[0062] Similarly, the initial node 308 of the third subgraph 320 requires the output of node 306. The system can wait to allocate the third subgraph 320 until the device allocated to the first subgraph completes the operation represented by node 306. Once the operation represented by node 306 is completed, the system can analyze the output of node 306 to allocate the third subgraph 320 to the corresponding available device.

[0063] The device can pause computation at nodes that require input that has not yet been computed, for example, by entering an idle state. For instance, after executing the computation at node 308, the device assigned to the third subgraph 320 can execute the computation at node 312. Then, the device assigned to the third subgraph 320 determines whether input from node 310 has been received. The device can wait to execute the computation at node 312 until it receives input from node 310.

[0064] At the last node, node 316, after the operation is performed, the device assigned to that node can return the node's output or the instruction to complete graph processing to the system. If necessary, the system can then return the output to the client.

[0065] Figure 4 This is a flowchart of an example process 400 for assigning a subgraph to a device. For convenience, process 400 is described as being executed by a system of one or more computers located in one or more locations. For example, a properly programmed computational graph system, such as... Figure 1The computational graph system 100 is capable of executing process 200.

[0066] The system determines the initial allocation of the subgraph to devices (step 402). The system can determine the initial allocation to devices by using a greedy algorithm. That is, the system determines the devices to which the subgraph is allocated by analyzing one or more initial nodes in the subgraph. The initial node is the node from which data begins to flow in the subgraph.

[0067] In some implementations, the system determines the memory capacity consumed by the operation represented by the initial node or by the operations represented by nodes connected to the initial node. (Refer to the above.) Figure 2 and Figure 3 The system can analyze the dimension of tensors to or from the initial node to determine the memory capacity to be consumed, as described above. Figure 2 As stated above.

[0068] Based on the determined capacity, the system allocates subgraphs to devices with at least the determined memory capacity. By considering the initial node rather than subsequent nodes, the system can quickly allocate subgraphs to devices, but these allocations may not be optimal because subsequent nodes may require resources that the allocated device may not be able to handle efficiently. For example, if the allocated device does not have enough memory, paging must be implemented to execute subsequent operations represented in the subgraph.

[0069] The system monitors the devices' processing of the graph to determine statistics (step 404). For example, the system can monitor the computation time, idle time, or both for each device. Computation time is the time a device spends fulfilling a request from a client. That is, the system measures the time each device spends completing its assigned subgraph computation. The system can also measure the idle time each device spends waiting for subsequent computations while processing its assigned subgraph.

[0070] The system adjusts the initial allocation using statistical data (step 406). Specifically, the system can adjust the initial allocation to minimize computation time, idle time, or both. For example, based on the corresponding initial nodes of the first and second subgraphs, the system can first allocate a first device to perform computations on the first subgraph and allocate a second device to perform computations on the second subgraph. After tracking the computation execution time, the system can compare the resource utilization between the first and second devices. If the first device has a longer idle time than the second device, but the first device has more processing power and memory, then for subsequent requests using the first and second subgraphs for computation, the system can adjust the allocation from the first subgraph to the second device and from the second subgraph to the first device.

[0071] The system reallocates the subgraph to the device according to the adjusted allocation (step 408). That is, continuing from the above description, in response to a subsequent request to perform operations using the first and second subgraphs, the system allocates the first subgraph to the second device and the second subgraph to the first device.

[0072] The system can repeat steps 404 to 408 to continuously update the allocation to improve performance. For example, the system can identify multiple possibilities for adjusting the allocation to minimize idle time. The system can selectively allocate a specific subgraph to numerous different devices. In subsequent operations on the specific subgraph, the system selects the first possibility and measures the first operation time to complete the operation. In another subsequent operation, the system selects the second possibility during a second iteration and measures the second operation time to complete the operation. In yet another subsequent operation, the system selects the possibility with the shortest operation time and can select different allocation possibilities for different subgraphs. In some implementations, the system can repeat these steps until a threshold improvement has been reached.

[0073] After a device is assigned to a corresponding subgraph, it performs operations on that subgraph, for example, using a neural network (or other machine learning model) represented by a computation graph to process the model input. Upon completion of the operation, the device can notify the system of the completion of the operation or the output of the operation (if any). In some cases, the request received by the system can specify that the response includes one or more outputs of a specific node in the computation graph. After completion of the operation, the system can receive the output of a specific node from one or more devices to which that specific device is assigned. The system can then provide the output to the client, as described above. Figure 1 As stated above.

[0074] In some implementations, a user can specify, for example, a subgraph of a computation graph, a node in a computation graph, or a portion of a computation graph consisting of different sets of nodes, as a function that can be reused as part of another computation graph. Specifically, in these implementations, after providing the system with data identifying the computation graph, the user can submit a request to specify a particular portion of the computation graph as a reusable function and associate that reusable function with a function name, such as a system-generated identifier or a user-specified logical name. The system can then save data identifying the nodes and edges in that particular portion and associate that portion with the function name. At a later time, the system can receive a request to process another computation graph that includes references to the function, for example, an indication that the output of a specific node in another computation graph should be provided as input to a function with that function name and that the output of that function should be provided as input to another specific node in another computation graph. In response to this request, the system can identify the graph portion associated with the function name and can generate an augmented computation graph that includes that portion at the appropriate location. The system can then process the augmented computation graph as described above. Therefore, users can easily include certain common reusable operations in their computation graphs, such as operations for specific configurations of neural network layers, without having to regenerate the graph portion representing those operations each time.

[0075] Embodiments of the subject matter and functions described in this specification can be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling its operation. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof. However, the computer storage medium is not a propagating signal.

[0076] The term "data processing apparatus" encompasses various types of devices, equipment, and machines for processing data, including programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the aforementioned computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0077] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored in a file containing other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the aforementioned program, or multiple co-files, such as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0078] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device comprising one or more processors and computer-readable media. Furthermore, two or more of these engines can be implemented on the same computing device or on different computing devices.

[0079] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by processing input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the devices can also be implemented as special-purpose logic circuits.

[0080] For example, a computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random access memory, or both. The basic elements of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled thereto to receive data from, transfer data to, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0081] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented with dedicated logic circuitry or incorporated therein.

[0082] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) monitor, an LCD (liquid crystal display) monitor, or an OLED display, and an input device for providing input to the computer, such as a keyboard, a mouse, or a presence-sensitive display or other surface. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any type of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending resources to and receiving resources from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0083] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, such as a data server; middleware components, such as an application server; or front-end components, such as a client computer having a graphical user interface or web browser that a user can interact with embodiments of the subject matter described herein; or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0084] A computing system can include both clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is implemented using computer programs that run on the respective computers and have a client-server relationship with each other.

[0085] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations or even as initially claimed features, one or more features from a claimed combination may be excluded from said combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0086] Similarly, although operations are depicted in a specific order in the diagrams, this should not be construed as requiring the operations to be performed in the indicated order or sequentially, or performing all illustrated operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments. Rather, it should be understood that the program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0087] Specific embodiments of the subject matter have been described. Other embodiments fall within the scope of the appended claims. For example, the actions described in the claims can be performed in a different order and the desired result can still be obtained. For instance, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for performing neural network or machine learning operations, comprising: Receive requests from the client to process a computation graph representing a neural network or machine learning model; Obtain data representing the computation graph, the computation graph including multiple nodes and directed edges representing the neural network or machine learning model, wherein each node represents a corresponding operation, and each directed edge connects a corresponding first node to a corresponding second node, the corresponding second node representing an operation that receives the output of the operation represented by the corresponding first node as input; Identify a plurality of available devices for processing the computation graph, wherein the available devices include devices having processing units and memory and include devices having different memory capacities; The computation graph is divided into multiple subgraphs, each subgraph including one or more nodes in the computation graph; as well as For each subgraph, the operations represented by one or more nodes in the subgraph are allocated to corresponding available devices among the plurality of available devices for processing, wherein the allocation of the corresponding available devices includes determining the devices allocated to the subgraph by estimating the maximum memory capacity to be used by any node in the subgraph. The allocation includes assigning each subgraph to a device determined by estimating the maximum memory capacity to be consumed by operations representing nodes in the subgraph.

2. The method according to claim 1, The request from the client identifies a specific device type to perform an operation represented by a specific node in the computation graph, and in, The operation of assigning each subgraph to a node represented by one or more nodes in the subgraph includes assigning the subgraphs in the plurality of subgraphs and assigning the specific node to a device having the specific device type specified by the request.

3. The method according to claim 1, wherein, For each subgraph, the operations assigned to one or more nodes in that subgraph include: Assign a first subgraph to the first device, the first subgraph containing an initial node and having no node in the first subgraph that depends on the output of other subgraphs; A second subgraph with second subgraph nodes is assigned to a second device. These second subgraph nodes require the output from the initial node as input. The output from the initial node will be computed by the device assigned to the first subgraph. Waiting to allocate the second subgraph to the second device until the operation represented by the initial node in the first subgraph has been completed.

4. The method of claim 3, further comprising determining the dimension of a tensor flowing along a directed edge from the initial node to a node of the second subgraph, and assigning the second subgraph to the second device according to the dimension of the tensor.

5. The method according to claim 1, wherein, Estimating the maximum memory capacity to be used by any node in the subgraph involves traversing the subgraph to compute the dimension of the tensor on each directed edge traveling to and from each node in the subgraph.

6. The method according to claim 5, wherein, The allocation includes assigning the subgraph to a device having a memory capable of storing the largest tensor flowing in the subgraph as indicated by the dimensions of the tensor.

7. The method according to claim 1, in, The request identifies the output from the corresponding node in the computation graph, and The method further includes: For each of the devices assigned to the subgraph, cause that device to perform the operation represented by the nodes of the subgraph assigned to that device; and After performing the operation on the final node of the subgraph assigned to the device, the output is provided to the client.

8. The method according to claim 1, further comprising: Determine the initial allocation of subgraphs to devices; Monitor the device to determine statistical data; The initial allocation is adjusted using the statistical data. as well as The subgraph is reassigned to the device based on the adjusted initial assignment.

9. The method according to claim 1, The computation graph represents the operations corresponding to the execution of a neural network comprising multiple layers, each layer being associated with one or more operations. in, Each of the plurality of subgraphs includes at least one operation corresponding to a corresponding layer among the plurality of layers.

10. The method according to claim 1, wherein: Estimating the maximum memory capacity to be used by any node in the subgraph includes traversing the subgraph to compute the dimension of the tensor on each directed edge traveling to and from each node in the subgraph; and The allocation includes assigning the subgraph to a device having a memory capable of storing the largest tensor flowing in the subgraph as indicated by the dimensions of the tensor.

11. A computer-readable medium having instructions stored thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform a method, the method comprising: Receive requests from the client to process a computation graph representing a neural network or machine learning model; Obtain data representing the computation graph, the computation graph including multiple nodes and directed edges representing the neural network or machine learning model, wherein each node represents a corresponding operation, and each directed edge connects a corresponding first node to a corresponding second node, the corresponding second node representing an operation that receives the output of the operation represented by the corresponding first node as input; Identify a plurality of available devices for processing the computation graph, wherein the available devices include devices having processing units and memory and include devices having different memory capacities; The computation graph is divided into multiple subgraphs, each subgraph including one or more nodes in the computation graph; as well as For each subgraph, the operations represented by one or more nodes in the subgraph are allocated to corresponding available devices among the plurality of available devices for processing, wherein the allocation of the corresponding available devices includes determining the devices allocated to the subgraph by estimating the maximum memory capacity to be used by any node in the subgraph. The allocation includes assigning each subgraph to a device determined by estimating the maximum memory capacity to be consumed by operations representing nodes in the subgraph.

12. The computer-readable medium according to claim 11, The request from the client identifies a specific device type to perform an operation represented by a specific node in the computation graph, and in, The operation of assigning each subgraph to a node represented by one or more nodes in the subgraph includes assigning the subgraphs in the plurality of subgraphs and assigning the specific node to a device having the specific device type specified by the request.

13. The computer-readable medium of claim 11, wherein, For each subgraph, the operations assigned to one or more nodes in that subgraph include: Assign a first subgraph to the first device, the first subgraph containing an initial node and having no node in the first subgraph that depends on the output of other subgraphs; A second subgraph with second subgraph nodes is assigned to a second device. These second subgraph nodes require the output from the initial node as input. The output from the initial node will be computed by the device assigned to the first subgraph. Waiting to allocate the second subgraph to the second device until the operation represented by the initial node in the first subgraph has been completed.

14. The computer-readable medium according to claim 13, wherein, The method further includes determining the dimension of the tensor flowing along the directed edges from the initial node to the second subgraph node, and assigning the second subgraph to the second device based on the dimension of the tensor.

15. The computer-readable medium of claim 11, wherein, Estimating the maximum memory capacity to be used by any node in the subgraph involves traversing the subgraph to compute the dimension of the tensor on each directed edge traveling to and from each node in the subgraph.

16. The computer-readable medium of claim 15, wherein, The allocation includes assigning the subgraph to a device having a memory capable of storing the largest tensor flowing in the subgraph as indicated by the dimensions of the tensor.

17. The computer-readable medium according to claim 11, in, The request identifies the output from the corresponding node in the computation graph, and The method further includes: For each of the devices assigned to the subgraph, cause that device to perform the operation represented by the nodes of the subgraph assigned to that device; and After performing the operation on the final node of the subgraph assigned to the device, the output is provided to the client.

18. The computer-readable medium of claim 11, wherein, The method further includes: Determine the initial allocation of subgraphs to devices; Monitor the device to determine statistical data; The initial allocation is adjusted using the aforementioned statistical data; and The subgraph is reassigned to the device based on the adjusted initial assignment.

19. The computer-readable medium according to claim 11, The computation graph represents the operations corresponding to the execution of a neural network comprising multiple layers, each layer being associated with one or more operations. in, Each of the plurality of subgraphs includes at least one operation corresponding to a corresponding layer among the plurality of layers.

20. The computer-readable medium of claim 11, wherein: Estimating the maximum memory capacity to be used by any node in the subgraph includes traversing the subgraph to compute the dimension of the tensor on each directed edge traveling to and from each node in the subgraph; and The allocation includes assigning the subgraph to a device having a memory capable of storing the largest tensor flowing in the subgraph as indicated by the dimensions of the tensor.

Images