Job Scheduling for Distributed Computing Devices

Abstract

The present disclosure relates to methods, systems, and apparatus for scheduling jobs on distributed computing devices, including a computer program encoded on a computer storage medium for scheduling operations represented as a computation graph on a distributed computing network. The method includes: receiving data representing operations to be performed to execute a job on a plurality of hardware accelerators of a plurality of different accelerator types; generating, for the job and from at least the data representing the operations, features representing predicted performance of the job on the hardware accelerators of the plurality of different accelerator types; generating, from the features, a corresponding predicted performance metric for the job for each of the plurality of different accelerator types according to a performance objective function; and providing one or more recommendations to a scheduling system for scheduling the job on the hardware accelerators of the one or more recommended types.

Description

Technical Field

[0001] This specification relates to scheduling jobs on distributed computing devices, and in particular to training machine learning models including neural networks. Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output based on a given input. Some neural networks also include one or more hidden layers in addition to the output layer. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from its input based on the current values ​​of its corresponding set of network parameters.

[0003] Neural networks and other machine learning models can be processed for training or inference on multiple computing devices, which are collectively referred to in this specification as distributed computing networks.

[0004] A distributed computing network can be heterogeneous, meaning that the computing devices of the distributed computing network can include one or more different types of processing units, for example, central processing units ("CPUs"), graphics processing units ("GPUs"), and tensor processing units ("TPUs"), as well as other forms of specialized logic circuitry, such as field programmable gate arrays ("FPGAs") and application-specific integrated circuits ("ASICs"). A heterogeneous distributed computing network can also include different generations of the same type of processing unit. For example, a distributed computing network is considered heterogeneous if it includes a combination of first-generation TPUs, second-generation TPUs, and third-generation TPUs.

[0005] A single computing device can be heterogeneous, meaning that a single computing device can include multiple processing units of different types. In this specification, a distributed computing network that includes heterogeneous computing devices is referred to as a heterogeneous distributed computing network. A heterogeneous distributed computing network can receive jobs that represent a set of operations to be performed on the network. Summary of the Invention

[0006] This specification describes techniques for scheduling jobs on a heterogeneous distributed computing network having multiple computing devices of multiple different computing device types.

[0007] These techniques generally involve a recommendation engine configured to receive operations to be performed on a distributed computing network as part of a job. The recommendation engine can extract features representing the predicted performance of the job on each type of computing device present in the distributed computing network.

[0008] From these characteristics, a recommendation engine can predict a performance metric for each different type of computing device present in the distributed computing network. The recommendation engine is configured to generate a recommendation for one or more types of computing devices for scheduling a job, and specify a total amount of each type of computing resource recommended for assignment. The recommendation can be provided as an input to a scheduling system, which can use the predicted performance metric and other metrics to generate a scheduler for partitioning and assigning a computation graph across multiple computing devices in the distributed computing network.

[0009] The set of performance metrics can be ranked. Higher-ranked metrics can correspond to types of computing devices that are predicted to more efficiently execute operations of the computation graph than types of computing devices corresponding to lower-ranked metrics.

[0010] The scheduler may use the received recommendations in addition to other metrics such as cost, equipment availability, and preferences among users requesting scheduling to schedule operations on the distributed computing network more efficiently than without using performance metrics.

[0011] In some implementations, the recommendation is a report that can be provided to the user that details different types of computing devices and predicts which type will perform better than others.

[0012] The performance metric may vary depending on the corresponding objective function used by the recommendation engine to measure performance. The types of performance measured can include runtime performance, energy performance, and operating cost performance, but in general can be any metric corresponding to the performance of a job across a distributed computing network.

[0013] Regardless of the type of performance that the recommendation engine measures, the recommendation engine can predict a set of performance metrics even when historical information about the performance of previous jobs on different types of computing devices is sparse or largely unavailable.

[0014] In general, one innovative aspect of the subject matter described herein includes a recommendation engine configured to generate performance metrics by training and executing a machine learning model. The machine learning model is configured to receive as input a computation graph representing operations of an input job and generate as output a set of performance metrics that measure the predicted performance of the computation graph on each type of computing device in a distributed computing network. The recommendation engine can train the machine learning model and generate a set of performance metrics for the computation graph based on an objective function.

[0015] The machine learning model may include a graph convolutional neural network ("GCN"). The GCN may be trained to receive as input a computational graph of a job and generate as output an embedding representing features of the computational graph. An embedding is a data structure of lower dimensionality, such as a vector, that maps values ​​of a data structure of higher dimensionality, such as an image or a chart. Embeddings represent the transformation of categorical values ​​into real numbers and are collectively referred to as part of an embedding space. Embeddings in an embedding space may be compared and visualized as clusters to facilitate identifying "similar" embeddings, where similarity is defined by the objective function used to train the machine learning model.

[0016] The recommendation engine may further include a collaborative filtering system, such as a neural collaborative filtering network, configured to receive the embedding of the computation graph as input and generate a set of performance metrics as output according to an objective function. The set includes performance metrics for each type of computing device in the distributed computing network.

[0017] The recommendation engine can generate a set of performance metrics without processing the computation graph through a machine learning model. In some embodiments, the recommendation engine can be configured to use a simulator to simulate the performance of the computation graph across different types of computing devices. The simulator can simulate the performance of executing the input computation graph for each type of computing device. The output of the simulator is a set of performance metrics, similar to the machine learning model approach briefly described above.

[0018] The subject matter described herein can be implemented in certain embodiments, including methods, systems, and computer-readable storage media, such that one or more of the following advantages are achieved. A recommendation engine implementing the subject matter described herein can generate performance metrics as recommendations to a scheduler to improve the scheduler's efficiency in assigning jobs to one or more computing devices in a distributed computing network. The recommendation engine can generate recommendations using sparse historical information about previous jobs executed on different types of computing devices, which is otherwise unavailable to conventional recommendation systems due to a lack of information, i.e., due to the cold start problem.

[0019] The cold start problem generally refers to the problem of drawing inferences from sparse information. For example, the cold start problem can arise in the context of predicting qualitative information about the performance of machine learning jobs across heterogeneous distributed computing networks. Specifically, the historical performance of machine learning jobs is limited for a variety of reasons.

[0020] One reason is that machine learning jobs are often executed on distributed networks of hardware accelerators. Hardware accelerators are computing devices that include specialized hardware designed to perform certain types of operations, such as matrix multiplication, more efficiently than non-specialized or "general-purpose" computing devices. Hardware accelerators are generally more computationally efficient than their general-purpose counterparts, but are also typically more expensive due to the cost of the hardware and the associated energy costs required to power and maintain the accelerators. As a result, historical information about the performance of machine learning jobs on different types of hardware accelerators is often limited to the few types of hardware accelerators available to the specific user executing the job.

[0021] Another reason why historical performance information for machine learning jobs is limited is due to the volatile nature of machine learning models. The metadata describing a machine learning job (e.g., the job name or architectural features describing the job's machine learning model) often changes, particularly during training, when the optimal architecture is still being learned. As a result, it is difficult to compare machine learning jobs executed on different types of computing devices to predict their performance.

[0022] A recommendation engine implementing the subject matter can improve the performance of heterogeneous distributed computing networks, which often include multiple non-interchangeable computing devices, i.e., computing devices that simply cannot perform the same operations or cannot perform them with the same efficiency and accuracy as other computing devices in the network. Homogenizing a distributed computing network to include computing devices of a single device is cost-prohibitive and inefficient, particularly because heterogeneous distributed computing networks can flexibly execute different jobs optimized for certain types of computing devices. Therefore, a recommendation engine can leverage the diversity of computing devices in a distributed computing network to facilitate the generation of schedules and / or recommendations customized for each input job.

[0023] Also from the set of performance metrics, the recommendation engine can generate recommendations for user analysis. The recommendations can be a report indicating recommended computing device types on which to execute a job, to provide new insights into which types of computing devices perform a particular job better than other types.

[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Illustration of an example recommendation engine.

[0026] Figure 2 Illustration of an example scheduling system.

[0027] Figure 3Diagram of the recommendation pipeline using a machine learning model.

[0028] Figure 4 Figure 1 shows the recommended pipeline for using the simulator.

[0029] Figure 5 is a flow chart of an example process for generating performance metrics for a scheduling system.

[0030] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0031] Figure 1 An example recommendation engine 100 is illustrated. The recommendation engine 100 may receive a job 105 as input and generate a recommendation 110 as output. Alternatively, the recommendation engine 100 may be configured to receive multiple jobs 105A-N and generate multiple recommendations 110A-N. Jobs and recommendations will be described with reference to the job 105 and the recommendation 110.

[0032] Computing device types are distinct categories of computing devices grouped by common characteristics. Computing device types can include: architectural features, such as memory or processing power; processor or circuit logic, such as FPGA, ASIC, or TPU; and generation within the same family of computing devices (e.g., first-generation TPU or second-generation TPU).

[0033] Recommendation 110 is a recommendation made by recommendation engine 100 for certain types of computing devices from among a plurality of different types of computing devices in a distributed computing network for executing job 105. Recommendation 110 may provide a user with a suggestion, for example, as a text report, which type of computing device is recommended for executing job 105 based on different characteristics of the computing device. These characteristics may include a recommended device memory size, model, and how the computing devices in the distributed computing network assigned to job 105 should be arranged relative to each other to improve execution of job 105.

[0034] In addition to recommendations for one or more types of computing devices to assign the job to, recommendation 110 may also include a value quantifying how much computing resources from each type of computing device should be allocated to the job. For example, recommendation 110 may include the total amount of each type of device, such as two accelerators of the first type and three accelerators of the second type; or, the recommendation may include a ratio of devices, such as a 3:1 ratio between two different types of computing devices.

[0035] In some implementations, a value quantifying the computing resources of each type of computing device can be specified in terms of aggregate computing resources. For example, recommendation 110 recommends assigning enough computing devices of the first type so that the total computing resources assigned from the computing devices of the first type meet the recommended threshold.

[0036] The recommendation 110 may also include a hint or reason why the recommendation engine 100 generated the recommendation, for example, explaining why the performance of the recommended type of computing device is similar relative to other types. The reason may also include previous observations of the performance of jobs similar to the job 105 on the recommended type of computing device in the past, for example, an observation that jobs including many convolution operations perform better on the recommended type of computing device, for example, due to the hardware configuration of the computing device.

[0037] The recommendation engine 100 is configured to rank each type of computing device corresponding to a corresponding performance metric in the performance metric set. The ranking depends on the objective function used to generate the performance metric. The objective function can be any function that measures a characteristic of a given job of interest (e.g., in terms of runtime performance, energy cost, or productivity of computing resources dedicated to the job).

[0038] For example, if the objective function measures the performance of each type of computing device by computing throughput when performing a given job, the recommendation engine 100 will rank computing devices of a type having a corresponding metric indicating higher throughput over other types of computing devices having a corresponding metric indicating lower throughput.

[0039] As another example, the recommendation engine 100 can generate a performance metric using an objective function that measures the performance of each type of computing device by energy consumption while performing a given job. Using this example objective function, the recommendation engine 100 is configured to rank certain types of computing devices with generally lower energy consumption higher than other types of computing devices with generally higher energy consumption.

[0040] The recommendations may be handwritten as a product of expert analysis, the recommendation engine 100 may be configured to automatically infer reasons or prompts, or the recommendation engine 100 may be configured to automatically and use expert analysis to generate recommendations. In some implementations, the recommendation engine 100 is configured to infer reasons or prompts by correlating characteristics of a type of computing device with a performance ranking of that type of computing device.

[0041] For example, the recommendation engine 100 may generate a recommendation identifying a greater memory capacity for a computing device of a better performing type because the recommendation engine 100 correlates higher-ranked types of computing devices with having higher memory capacity. The recommendation engine 100 may correlate multiple characteristics of each type of computing device with its corresponding performance metrics to generate richer recommendations.

[0042] Jobs 105 can collectively represent operations represented in a computation graph 115 and optional metadata 120 associated with the computation graph 115 or the job 105 itself. A computation graph is a diagram of nodes that are each connected to at least one other node by an edge. Each node represents an operation to be performed by one or more computing devices in a distributed computing network. For any two nodes u and v in a computation graph, the edge (u, v) is a directed edge and represents a data dependency from u to v. A data dependency from u to v means that the operation represented by node u generates an output that is an input to the operation represented by node v.

[0043] The operations in the computation graph 115 may be operations of a software program configured for execution on a distributed computing network. For example, the operations in the computation graph 115 may be operations for training or executing a machine learning model (e.g., a neural network).

[0044] The metadata 120 may include information about the job 105 (e.g., the job name), information about the user who sent the job 105 to the recommendation engine 100, the priority assigned to the job 105 compared to other jobs, and the predetermined resource allocation requirements of the job 105. The metadata 120 may also include statistics of the computation graph collected when the computation graph is compiled, such as floating point operations per second ("FLOPS") and average or peak memory usage. The metadata 120 may also include statistics of the computation graph collected when the computation graph is executed, such as memory usage, processor core usage, and the type of statistics on the computing device that has previously executed the job 105, such as FLOPS or duty cycle.

[0045] Figure 2 An example scheduling system 200 is shown. The scheduling system 200 includes a recommendation engine 210, a scheduler engine 220, and a distributed computing network 230. The scheduling system 200 can receive as input a job 240 including a computation graph 250 and optional metadata 260. Although the example scheduling system 200 is shown as being configured to receive a job 240 including operations represented as a computation graph, the scheduling system 200 can be configured in other embodiments to receive data representing the operations of the job 240 in other formats, for example, as a series of function calls of a suitably configured application program interface ("API").

[0046] The scheduler engine 220 is configured to send the computation graph 250 and optional metadata 260 to the recommendation engine 210. The recommendation engine 210 is configured to predict a set of performance metrics for each type of computing device in the distributed computing network 230 and use the performance metrics to generate recommendations for scheduling operations to different types of hardware accelerators. The scheduler engine 220 can then receive recommendations from the recommendation engine 210.

[0047] The scheduler engine 220 can be any conventional scheduling system for scheduling jobs on a distributed computing environment. The scheduler engine 220 can implement any conventional scheduling algorithm to schedule work to the distributed computing network 230 and be enhanced with recommendations from the recommendation engine 210.

[0048] For example, in addition to the recommendations generated by the recommendation engine 210, the scheduler engine 220 can also use a set of characteristics for the job 240 to implement priority scheduling. The set of characteristics that the scheduler engine 220 uses to schedule the job 240 can include, for example, a user-assigned priority level and characteristics about the user submitting the job 240 for scheduling, such as the priority level that the user has access to for sending the job to the distributed computing network 230. The characteristics can be obtained from the metadata 260 or from another source, such as provided by the user.

[0049] In addition to the characteristic sets for jobs 240, scheduler engine 220 may also schedule jobs according to globally imposed requirements, such as scheduling certain types of jobs with higher priority than others, or favoring some types of computing devices over others due to energy costs, for example.

[0050] When scheduling a job 240, the scheduler engine 220 may partition the computation graph 250 into multiple subgraphs. Each subgraph is linked to another subgraph by an edge that represents the flow of data from the output of one subgraph to the input of another subgraph.

[0051] The scheduler engine 220 can decide which computing device to assign to the corresponding subgraph based on the recommendations generated by the recommendation engine 210. For example, the recommendation engine 210 can indicate that a computing device of the first type is better suited to process the job 240, predicting that the performance metric of the computing device of the first type is higher than the performance metric of other types of computing devices in the distributed computing network 230. Then, using the recommendations, the scheduler engine 220 can partition the computation graph 250 into multiple subgraphs and assign a preference for the computing device of the first type to the subgraphs.

[0052] The scheduling is described as "biased" toward the first type of computing device because other imposed requirements on the scheduler engine 220 may still cause the scheduler engine 220 to schedule the subgraph to a computing device type that is ranked lower by performance metric and subsequently by recommendation. In general, the set of performance metrics from the recommendation engine 210 enriches the decision-making of the scheduler engine 220 when assigning subgraphs to the distributed computing network 230.

[0053] For example, if the computing devices with the highest-ranked performance metrics are not available in the distributed computing network 230, e.g., because those computing devices are not currently running, the scheduler engine 220 may schedule the job on a type of computing device that is less recommended based on the received recommendation by the recommendation engine 210. As another example, the scheduler engine 220 may be configured to assign a computing device of a more likely recommended type to the job 240 depending on the priority level assigned to the job 240 or the status of the user from whom the job 240 originated.

[0054] Figure 3 The recommendation pipeline 300 using a machine learning model is shown. The recommendation engine can be configured to process input jobs 310 using the recommendation pipeline 300 and generate performance metrics. Figure 1 and Figure 2 The performance metrics may then be sent to a scheduling system for scheduling jobs 310 and / or used to generate recommendations, as described above.

[0055] The recommendation pipeline 300 starts with an input job 310 having a computation graph and optional metadata, as described above. A graph convolutional neural network 320 can receive the metadata and the computation graph of the input job 310.

[0056] A graph convolutional neural network is a neural network with multiple layers including an input layer and an output layer that receives a graph as input and performs a convolution operation on the input graph. The graph convolutional neural network 320 can be trained to receive a computational graph as input and generate an embedding representing features of the computational graph as output. The embedding from the graph convolutional neural network can represent features of neighboring nodes from a given node in the computational graph to generate an embedding representing a "similar" computational graph. Features of neighboring nodes from a given node can be performed by performing a convolution operation on the neighboring nodes.

[0057] The graph convolutional neural network 320 can be jointly trained with the neural collaborative filtering network described below to generate embeddings of similar computation graphs according to the objective function. When the graph convolutional neural network 320 and the neural collaborative filtering network are jointly trained, the resulting output from the neural collaborative filtering network is a set of performance metrics for each type of computing device, even for computing device types where historical data on the performance of the type of input job is sparse or non-existent.

[0058] Turning to the architectural details of the graph convolutional neural network ("GCN") 320, at each layer, the GCN 320 can perform one or more activation functions on the corresponding input received at the layer. The one or more activation functions can be any conventional activation function, such as ReLU, sigmoid, or tanh.

[0059] Similarly, at each layer, the GCN 320 can generate multiple outputs from one or more activation functions at that layer to extract features from each node in the computation graph, as well as extract separate features from the inputs and outputs of each node in the computation graph, and extract features from the neighbors of that node. To this end, the recommendation engine can, for example, enhance the computation graph to include self-loops at each node in the computation graph to extract features not only from the neighboring nodes of any given node in the computation graph but also from the given node itself.

[0060] To extract features separately from the input and output of a given node in the computation graph, GCN 320 may aggregate activation function outputs separately for the output and input of the node to learn whether the performance of the computation graph on a specific type of computing device is dominated by the input or the output.

[0061] The recommendation engine can use backpropagation techniques (e.g., backpropagation using stochastic gradient descent) to update the weights. For each layer, the recommendation engine can calculate the gradient of the activation function with respect to the ground truth value. The ground truth value can be a historical performance metric of the computation graph on each type of computing device. The recommendation engine can update the weights based on the gradient of the calculated activation output, where the activation output includes at least one of the following: (i) the corresponding output calculated node by node; (ii) the corresponding activation output of the aggregation of each node output of each node in the layer; or (iii) the corresponding activation output of the aggregation of each node input of each node in the layer.

[0062] Equation 1 is an example layer-by-layer propagation rule for GCN 320:

[0063]

[0064] l is a layer in GCN 320, and H l is the tensor of the activation function at layer l. A tensor is a multidimensional array of numbers or other values ​​(e.g., strings) with a specific rank corresponding to the number of dimensions of the array. For example, a scalar value is a rank-0 tensor, a numeric vector is a rank-1 tensor, and a matrix is ​​a rank-2 tensor.

[0065] Equation 1 defines H l+1 , where σ(·) represents the activation function and [] represents the cascade. H 0 , that is, the input layer, defined as H 0 =X∈RN*D , where X is the node feature tensor of the computation graph, which represents the N-dimensional feature vector of each operation represented by the computation graph with a total of D nodes.

[0066] A is the adjacency matrix of the input computational graph. An adjacency matrix is ​​a matrix where the elements of the matrix indicate whether the nodes within the graph are adjacent (i.e., whether they are linked by a common edge). In Equation 1, and represents the adjacency matrix in which each node is considered to be adjacent to itself (i.e., each node has self-loops to allow GCN 320 to learn the characteristics of each given node, not just the characteristics of its neighbors, as described above).

[0067] Also in Equation 1, D in And D out Is a matrix representing the input and output degrees of each node in the computational graph. Denote the computational graph as D in and D out Enabling GCN 320 to learn the features of the input and output of each node in the computation graph separately can improve how GCN 320 ultimately generates embeddings of the computation graph.

[0068] Also in Equation 1, V l and W l is a separate trainable weight tensor for layer l. The recommendation engine can update the elements of the trainable weight tensor during training.

[0069] Equation 2 is another example layer-by-layer propagation rule of GCN 320 used in some embodiments:

[0070]

[0071] Equation 2 is functionally equivalent to Equation 1. Instead of using the adjacency matrix using self-loop enhancement as described above with reference to Equation 1, Equation 2 includes a single term H l U l , where U l is another trainable weight tensor for this layer l. Therefore, Equation 2 can also be used to define a layer to learn the features of each single node at layer l.

[0072] Whether GCN 320 defines each layer according to Equation 1 or Equation 2, GCN 320 may learn the following combination at each layer:

[0073] 1. Node-by-node calculation to capture situations where the type of operation represented by the node and other characteristics of the operation determine the performance, expressed in Equation 1 by and Expressed and in Equation 2 by H l U lexpress;

[0074] 2. Aggregate activation outputs for each node output in the computation graph to capture the situation where the output in the computation graph at each node determines the performance, expressed in Equation 1 by and in Equation 2 by indicates; and

[0075] 3. The activation output of the aggregation of each node input in the computation graph to capture the situation where the input in the computation graph at each node determines the performance, which is expressed in Equation 1 by and in Equation 2 by express.

[0076] Next, in the recommendation pipeline 300, the embeddings from the GCN 320 can be provided as input to a neural collaborative filtering network 330. The neural collaborative filtering network 330 can be trained to receive the embeddings as input and generate a set of performance metrics for the input job 310 as output. In general, collaborative filtering refers to a class of techniques for making predictions or recommendations for one agent in response to a task, using information collected from the actions or preferences of multiple other agents in response to the same task.

[0077] The Neural Collaborative Filtering Network ("NCF Network") 330 is a neural network having multiple layers, for example, a feed-forward neural network such as a multilayer perceptron, which has been trained to generate a performance metric corresponding to the position of the input embedding in the embedding space. The GCN 320 and the NCF Network 330 can be jointly trained to generate embeddings for multiple computation graphs such that embeddings that are close to each other in the embedding space correspond to computation graphs that perform similarly according to the objective function used by the GCN 320 and the NCF Network 330.

[0078] Jointly training the GCN 320 and the NCF network 330 means that the activation outputs during the forward pass are calculated as if the GCN 320 and the NCF network 330 were one neural network. Specifically, the output embeddings for the GCN 320 are provided as input to the NCF network 330. The input embeddings for the NCF network 330 are processed and a set of performance metrics are generated for the computation graph.

[0079] In the backward pass, for example, stochastic gradient descent and the loss function are used to update the weights of each layer of the NCF network 330, for example, backpropagation. The gradient obtained from the input layer of the NCF network 330 is passed to the output layer of the GCN 320, and backpropagation can continue to update the weights of each layer in the GCN 320.

[0080] The loss function used to train the NCF network 330 and the GCN 320 may be the loss between an expected set of performance metrics for a job and a set of predicted performance metrics generated as outputs of the NCF network 330. The NCF network 330 and the GCN 320 may be trained until a predetermined training condition is met, such as the number of iterations for weight updates, or until a calculated loss function reaches a predetermined threshold.

[0081] After the recommendation engine generates a set of performance metrics using the NCF network 330, if the recommendation engine is processing multiple input jobs at once, the recommendation engine can then perform a clustering step 340 on the corresponding performance metrics for each job according to the recommendation pipeline 300. The clustering step 340 can be performed by a clustering machine learning model that is trained to receive multiple sets of performance metrics and generate as output a set of performance metrics clustered by similar performance metrics within each set. Clustering allows the recommendation engine to further identify types of computing devices that may perform better or worse for the input job 310 based on the predicted performance of other jobs on those types of computing devices. In some embodiments, this step is skipped.

[0082] The recommendation engine can convert the performance metric set into one or more recommendations 370. The recommendation engine can generate recommendations based on a set of rules (e.g., recommending types of hardware accelerators over other types based on which types rank higher in the performance metric set; or through statistical analysis), for example, as described above with reference to Figure 1 As stated.

[0083] Although transforming step 350 is shown as occurring after clustering step 340 and before runnability check 360 (described below), in some implementations, transforming step 350 is performed after runnability check 360 or in a processing pipeline where clustering step 340 is not performed at all.

[0084] The recommendation engine may perform a runnability check 360 before providing recommendations 370 converted from the performance metric set. Runnability check 360 is a check to ensure that the type of device recommended for performing job 310 can actually run the job. If a type of computing device fails runnability check 360, the recommendation engine may remove the recommendation corresponding to that type of computing device. In some embodiments, runnability check 360 is not performed.

[0085] For various reasons, a computing device of a certain type may fail the runnability check for an input job 310. The reasons may be related to the hardware specifications and overall configuration of the computing device of that type. For example, if a computing device of that type does not support one or more operations represented by the computation graph of the job 310, the computing device of that type may fail the runnability check.

[0086] Figure 4 The diagram illustrates a recommended pipeline 400 using a simulator 420. The simulator can be configured to simulate the execution of a computation graph on different types of computing devices on a distributed computing network. To this end, the simulator 420 can receive (i) an input job 410 comprising a computation graph and optional metadata; and (ii) corresponding hardware specifications 415 corresponding to each type of computing device present in the distributed computing network. Figure 4 The solid arrows in represent the flow of the proposed pipeline 400, while the dashed arrows indicate that the hardware specification 415 may be provided to the simulator before the proposed pipeline 400 is executed.

[0087] As mentioned above, reference Figure 2 In the scheduling system 200 , the simulator 420 may be configured to receive data representing operations in formats other than those specified in the computation graph.

[0088] The simulator 420 can be configured to simulate the performance of a particular distributed computing network by receiving as input the hardware specifications 415 representing each type of computing device in the distributed computing network (e.g., the number of computing devices, the memory capacity / bandwidth and computing performance of each computing device, and other low-level details of the hardware specifications of each computing device). The simulator can be configured to generate a set of performance metrics based on an objective function corresponding to the type of performance metric desired.

[0089] Simulator 420 can be configured to simulate the performance of executing a computation graph with additional functionality and compatibility guarantees. In some embodiments, simulator 420 determines whether a type of computing device is compatible to execute the operations represented in the computation graph. In these embodiments, simulator 420 acts as a runnability check in the pipeline.

[0090] The simulator 420 can then use the hardware specifications of a given type of computing device and predict a performance metric for executing the input job 410 on the distributed computing network. Specifically, the simulator 420 can generate simulation results of executing the computation graph on one type of computing device and measure the performance metric of those simulation results according to the objective function. Figure 3 and Figure 4 In an embodiment using the simulator 420 , the simulator 420 replaces the GCN 320 and the NCF network 330 .

[0091] The recommendation engine may perform a clustering step 430 and a runnability check 450 before generating recommendations 450, similar to the above reference Figure 3The clustering step 340 and the runnability check 360 in the recommended pipeline 300 are described above. In some embodiments, the clustering step 430 is skipped. Furthermore, if the simulator 420 is configured to simulate the execution of the computation graph based on whether each type of computing device is compatible with executing the operations represented in the computation graph, the runnability check 440 can be skipped.

[0092] The processing pipeline 400 includes a performance metric recommendation conversion step 440, and the recommendation engine can be configured to receive the performance metric set from the simulator 420 (optionally after the clustering step 430) and generate recommendations, as described above with reference to Figure 3 The recommendation may generate a recommendation 460 that may be provided to a scheduler for scheduling a job 410 and / or for generating a recommendation for consumption by a user.

[0093] The recommendation engine may be configured to generate performance metrics without recommendations corresponding to the performance metrics.For example, the recommendation engine may send the performance metrics for analysis by a user or another system configured to receive a set of performance metrics as input.

[0094] In some implementations, the recommendation engine includes or is interconnected with engines that are individually configured to translate recommendations from a set of performance metrics according to different rules.

[0095] In some embodiments, the recommendation engine may send the performance metrics as a user-readable report and / or for display on a display of a user device. The sent performance metrics may be further processed for further analysis, for example, the recommendation engine may generate statistics from the performance metrics for analysis.

[0096] Figure 5 is a flow chart of an example process 500 for generating performance metrics by a recommendation engine. For convenience, the process 500 will be described as being performed by an engine of one or more computers located in one or more locations and appropriately programmed according to this specification. For example, a recommendation engine, for example, appropriately programmed Figure 1 The recommendation engine 100 can perform process 500.

[0097] Data representing operations to be performed is received by the recommendation engine in order to perform the operation on a plurality of computing devices of a plurality of different computing device types (step 510). Figure 1 As described above, the recommendation engine can receive data that specifies a portion of a computational graph and a job. These operations can correspond to any software program, including operations that train and / or execute machine learning models.

[0098] The recommendation engine generates features representing the predicted performance of the job on a plurality of computing devices of different computing device types for the job and from at least the data representing the operation (step 520). Figure 1 and Figure 2 As described, the recommendation engine may extract features, for example, as embeddings. In embodiments where the recommendation engine simulates a distributed computing network, the extracted features may be simulation results for each type of computing device in the distributed computing network.

[0099] The recommendation engine generates a corresponding predicted performance metric for the job for each of the plurality of different computing device types from the features according to the performance objective function (step 530). Figure 3 and Figure 4 As described, the recommendation engine can use a machine learning model or a simulator for a distributed computing network to generate a predictive performance metric.

[0100] One or more recommendations are provided to the scheduling system for scheduling the job on one or more recommended types of hardware accelerators (step 540). The recommendation may be for the recommended type and the amount of computing resources to be assigned. The scheduling system is configured to schedule the job for execution by one or more of the plurality of hardware accelerators and (ii) based on the data including the one or more recommendations (step 540). As described above with reference to Figure 2 As described, the recommendation engine can provide recommendations to the scheduler engine, which in turn can use the performance metrics and additional data to schedule input jobs.

[0101] The embodiments of the subject matter and the actions and operations described in this specification can be implemented using digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of these. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a computer program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The carrier can be a tangible, non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these, or a portion thereof. A computer storage medium is not a propagated signal.

[0102] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. A data processing apparatus can include specialized logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPU (graphics processing unit). In addition to hardware, an apparatus can also include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0103] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, engine, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, engine, subroutine or other unit suitable for execution in a computing environment, which environment may include one or more computers in one or more locations interconnected by a data communications network.

[0104] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, such as a file that stores one or more modules, subroutines, or portions of code.

[0105] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA, ASIC, or GPU, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0106] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor, or both, or any other kind of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated into, special-purpose logic circuitry.

[0107] Typically, a computer will also include one or more mass storage devices, or be operationally coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices. A mass storage device can be, for example, a magnetic disk, a magneto-optical disk, or an optical disk, or a solid-state drive. However, a computer need not have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0108] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer, or configured to communicate with a computer, having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, and input devices, such as a keyboard and a pointing device, such as a mouse, trackball, or touchpad, that the user can use to provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user's device in response to a request received from the web browser, or by interacting with an app running on a user's device, such as a smartphone or electronic tablet. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0109] This specification uses the term "configured to" in connection with systems, devices, and computer program components. With respect to a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. With respect to one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action. With respect to a dedicated logic circuit to be configured to perform a particular operation or action, it means that the circuit has electronic logic to perform the operation or action.

[0110] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with implementations of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs") (e.g., the Internet).

[0111] The computing system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from a user interacting with a device acting as a client). Data generated at the user device (e.g., the result of a user interaction) may be received at the server from the client device.

[0112] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of what is or may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from the claimed combination can be removed from the combination in some cases, and the claims may be directed to subcombinations or variations of subcombinations.

[0113] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. In general, multitasking and parallel processing can be advantageous. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

[0115] Embodiment 1 is a method comprising: receiving data representing operations to be performed in order to execute a job on a plurality of hardware accelerators of a plurality of different accelerator types; generating, for the job and from at least the data representing the operations, features representing predicted performance of the job on the hardware accelerators of the plurality of different accelerator types; generating, from the features, a corresponding predicted performance metric for the job for each of the plurality of different accelerator types according to a performance objective function; and providing one or more recommendations to a scheduling system for scheduling the job on one or more recommended types of hardware accelerators, wherein the scheduling system is configured to schedule the job for (i) execution by one or more of the plurality of hardware accelerators and (ii) based on data including the one or more recommendations.

[0116] Embodiment 2 is a method according to embodiment 1, wherein providing one or more recommendations comprises: converting the corresponding predicted performance metrics of the job into the one or more recommendations based on a statistical analysis of the corresponding predicted performance metrics of the job.

[0117] Embodiment 3 is a method according to any one of embodiments 1 to 2, wherein the machine learning system includes a graph convolutional neural network having multiple layers and trained to generate embeddings based on features extracted from the computational graph.

[0118] Embodiment 4 is a method according to any one of embodiments 1 to 3, wherein the data representing the operation specifies a computational graph representing the operation, wherein the feature includes an embedding of the computational graph in an embedding space, and wherein generating the feature includes using a machine learning system to perform the operation represented by the computational graph, the machine learning system having been trained to generate embeddings in the embedding space such that the position of the embedding in the embedding space reflects the corresponding performance metric of the computational graph on each of the multiple different accelerator types according to the performance objective function.

[0119] Embodiment 5 is a method according to any one of embodiments 1-4, wherein the machine learning system includes a graph convolutional neural network having multiple layers and trained to generate embeddings based on features extracted from the computational graph.

[0120] Embodiment 6 is a method according to any one of embodiments 1 to 5, wherein the graph convolutional neural network is configured to, at each layer of a plurality of layers: generate a corresponding output of the node-by-node calculation for each node in the layer, aggregate the corresponding activation output of the corresponding activation function performed on the layer on each node output of each node in the layer, aggregate the corresponding activation output of the corresponding activation function performed on the layer on each node input of each node in the layer, and update the corresponding weight corresponding to the layer based on the calculated gradient of at least one of the following: (i) the corresponding output of the node-by-node calculation, (ii) the aggregated corresponding activation output of each node output of each node in the layer, and (iii) the aggregated corresponding activation output of each node input of each node in the layer.

[0121] Embodiment 7 is the method of any one of embodiments 1 to 6, wherein generating a respective predicted performance metric for the job for each of the plurality of different accelerator types from the features comprises performing collaborative filtering on the embeddings.

[0122] Embodiment 8 is a method according to any one of embodiments 1 to 7, wherein the machine learning system includes a neural collaborative filtering network, which is trained to receive embeddings from the graph convolutional neural network and perform collaborative filtering on the embeddings to generate the corresponding predicted performance metric for each of the multiple accelerator types according to the performance objective function.

[0123] Embodiment 9 is a method according to any one of embodiments 1 to 8, wherein the neural collaborative filtering network and the graph convolutional neural network are jointly trained.

[0124] Embodiment 10 is a method according to any one of embodiments 1 to 9, wherein generating the corresponding predicted performance metric of the job for each of the multiple different accelerator types includes: performing a simulation of performing the operation on the multiple different accelerator types through a simulator; and for each accelerator type and based on the performance objective function, measuring the performance of the simulation of the accelerator type as the corresponding performance metric.

[0125] Embodiment 11 is a method according to any one of embodiments 1 to 10, wherein the performance metric and the performance objective function are one of the following: a cost performance metric and a cost objective function, a runtime performance metric and a runtime objective function, or an energy performance metric and an energy objective function.

[0126] Embodiment 12 is a system comprising: one or more computers and one or more storage devices storing instructions, wherein the instructions, when executed by the one or more computers, are operable to cause the one or more computers to perform the method according to any one of embodiments 1 to 11.

[0127] Embodiment 13 is a computer storage medium encoded with a computer program, wherein the computer program includes instructions that, when executed by a data processing device, are operable to cause the data processing device to perform the method according to any one of embodiments 1 to 11.

Claims

1. A system comprising: One or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving data representing operations of a job to be performed to execute the job on a plurality of hardware accelerators of a plurality of different accelerator types, wherein the data representing the operations specifies a computation graph representing the operations; generating, for the job and from the computation graph representing the operation, an embedding representing a feature of the computation graph, wherein the generating comprises processing the computation graph as input using a machine learning system that has been trained to generate embeddings from the computation graph in an embedding space such that a position of each embedding in the embedding space reflects a corresponding performance metric of the computation graph executed on each of the plurality of different accelerator types according to a performance objective function, and wherein embeddings that are close to each other in the embedding space correspond to computation graphs that execute similarly according to the performance objective function; generating a respective predicted performance metric for the job from the embedding for each of the plurality of different accelerator types according to the performance objective function; and One or more recommendations determined based on the corresponding predicted performance metrics are provided to a scheduling system for scheduling the job to be executed on one or more recommended types of hardware accelerators, wherein the scheduling system is configured to schedule the operations of the job for execution (i) by one or more of the plurality of hardware accelerators and (ii) based on the one or more recommendations.

2. The system according to claim 1, wherein: Providing one or more recommendations determined based on the corresponding predicted performance metrics includes: The corresponding predicted performance metrics are converted into the one or more recommendations based on a statistical analysis of the corresponding predicted performance metrics for the jobs.

3. The system according to claim 2, wherein: The one or more recommendations include text, and wherein converting the corresponding predicted performance metric of the job includes adding predetermined text corresponding to a recommended type of hardware accelerator from the one or more recommended types of hardware accelerators.

4. The system according to claim 1, wherein: The machine learning system includes a graph convolutional neural network having a plurality of layers and trained to generate embeddings of a computational graph.

5. The system according to claim 4, wherein: The graph convolutional neural network is configured to, at each of the multiple layers: Generate the corresponding output of the node-by-node computation for each node in the layer, aggregating over each node output of each node in the layer the corresponding activation output of the corresponding activation function performed on the layer, aggregating the corresponding activation outputs of the corresponding activation functions performed on the layer over each node input of each node in the layer, and updating respective weights corresponding to the layers based on the calculated gradients of at least one of: (i) the corresponding output of the node-by-node calculation, (ii) the aggregated corresponding activation outputs of each node output of each node in the layer, and (iii) The aggregated corresponding activation output of each node input for each node in the layer.

6. The system according to any one of claims 4 to 5, wherein: Generating a respective predicted performance metric from the features for the job executed by each of the plurality of different accelerator types includes performing collaborative filtering on the embeddings.

7. The system according to claim 6, wherein: The machine learning system includes a neural collaborative filtering network trained to receive embeddings from the graph convolutional neural network and perform collaborative filtering on the embeddings to generate the respective predicted performance metric for each of the plurality of accelerator types executing the job according to the performance objective function.

8. The system according to claim 7, wherein: The neural collaborative filtering network and the graph convolutional neural network are trained jointly.

9. The system according to claim 1, wherein: Generating the respective predicted performance metrics for the job executed by each of the plurality of different accelerator types includes: performing, by a simulator, a simulation of executing the operations on the plurality of different accelerator types; and For each accelerator type and according to the performance objective function, the performance of the simulation for the accelerator type is measured as a corresponding performance metric.

10. The system according to claim 1, wherein: The performance metric and the performance objective function are one of the following: Cost performance measures and cost objective functions, Runtime performance metrics and runtime objective functions, or Energy performance metrics and energy objective functions.

11. A method comprising: receiving data representing operations of a job to be performed to execute the job on a plurality of hardware accelerators of a plurality of different accelerator types, wherein the data representing the operations specifies a computation graph representing the operations; generating, for the job and from the computation graph representing the operation, an embedding representing a feature of the computation graph, wherein the generating comprises processing the computation graph as input using a machine learning system that has been trained to generate embeddings from the computation graph in an embedding space such that a position of each embedding in the embedding space reflects a corresponding performance metric of the computation graph executed on each of the plurality of different accelerator types according to a performance objective function, and wherein embeddings that are close to each other in the embedding space correspond to computation graphs that execute similarly according to the performance objective function; generating a respective predicted performance metric for the job from the embedding for each of the plurality of different accelerator types according to the performance objective function; as well as One or more recommendations determined based on the corresponding predicted performance metrics are provided to a scheduling system for scheduling the job to be executed on one or more recommended types of hardware accelerators, wherein the scheduling system is configured to schedule the operations of the job for execution (i) by one or more of the plurality of hardware accelerators and (ii) based on the one or more recommendations.

12. The method according to claim 11, wherein Providing one or more recommendations determined based on the corresponding predicted performance metrics includes: The corresponding predicted performance metrics are converted into the one or more recommendations based on a statistical analysis of the corresponding predicted performance metrics for the jobs.

13. The method according to claim 12, wherein: The one or more recommendations include text, and wherein converting the corresponding predicted performance metric of the job includes adding predetermined text corresponding to a recommended type of hardware accelerator from the one or more recommended types of hardware accelerators.

14. The method according to claim 11, wherein The machine learning system includes a graph convolutional neural network having a plurality of layers and trained to generate embeddings of a computational graph.

15. The method according to claim 14, wherein The graph convolutional neural network is configured to, at each of the multiple layers: Generate the corresponding output of the node-by-node computation for each node in the layer, aggregating over each node output of each node in the layer the corresponding activation output of the corresponding activation function performed on the layer, aggregating the corresponding activation outputs of the corresponding activation functions performed on the layer over each node input of each node in the layer, and updating respective weights corresponding to the layers based on the calculated gradients of at least one of: (i) the corresponding output of the node-by-node calculation, (ii) the aggregated corresponding activation outputs of each node output of each node in the layer, and (iii) The aggregated corresponding activation output of each node input for each node in the layer.

16. The method according to any one of claims 14-15, wherein: Generating a respective predicted performance metric from the features for the job executed by each of the plurality of different accelerator types includes performing collaborative filtering on the embeddings.

17. The method according to claim 16, wherein: The machine learning system includes a neural collaborative filtering network trained to receive embeddings from the graph convolutional neural network and perform collaborative filtering on the embeddings to generate the respective predicted performance metric for each of the plurality of accelerator types executing the job according to the performance objective function.

18. The method according to claim 17, wherein The neural collaborative filtering network and the graph convolutional neural network are trained jointly.

19. The method according to claim 11, wherein Generating the respective predicted performance metrics for the job executed by each of the plurality of different accelerator types includes: performing, by a simulator, a simulation of executing the operations on the plurality of different accelerator types; and For each accelerator type and according to the performance objective function, the performance of the simulation for the accelerator type is measured as a corresponding performance metric.

20. The method according to claim 11, wherein The performance metric and the performance objective function are one of the following: Cost performance measures and cost objective functions, Runtime performance metrics and runtime objective functions, or Energy performance metrics and energy objective functions.

21. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving data representing operations of a job to be performed to execute the job on a plurality of hardware accelerators of a plurality of different accelerator types, wherein the data representing the operations specifies a computation graph representing the operations; generating, for the job and from the computation graph representing the operation, an embedding representing a feature of the computation graph, wherein the generating comprises processing the computation graph as input using a machine learning system that has been trained to generate embeddings from the computation graph in an embedding space such that a position of each embedding in the embedding space reflects a corresponding performance metric of the computation graph executed on each of the plurality of different accelerator types according to a performance objective function, and wherein embeddings that are close to each other in the embedding space correspond to computation graphs that execute similarly according to the performance objective function; generating a respective predicted performance metric for the job from the embedding for each of the plurality of different accelerator types according to the performance objective function; as well as One or more recommendations determined based on the corresponding predicted performance metrics are provided to a scheduling system for scheduling the job to be executed on one or more recommended types of hardware accelerators, wherein the scheduling system is configured to schedule the operations of the job for execution (i) by one or more of the plurality of hardware accelerators and (ii) based on the one or more recommendations.

22. The one or more non-transitory computer-readable storage media of claim 21 , wherein providing one or more recommendations determined based on the corresponding predicted performance metrics comprises: The corresponding predicted performance metrics are converted into the one or more recommendations based on a statistical analysis of the corresponding predicted performance metrics for the jobs.

23. One or more non-transitory computer-readable storage media according to claim 22, wherein the one or more recommendations include text, and wherein converting the corresponding predicted performance metric of the job includes adding predetermined text corresponding to a hardware accelerator of the recommended type in the one or more recommended types of hardware accelerators.

24. The one or more non-transitory computer-readable storage media of claim 21, wherein: The machine learning system includes a graph convolutional neural network having a plurality of layers and trained to generate embeddings of a computational graph.

25. The one or more non-transitory computer-readable storage media of claim 24, wherein: The graph convolutional neural network is configured to, at each of the multiple layers: Generate the corresponding output of the node-by-node computation for each node in the layer, aggregating over each node output of each node in the layer the corresponding activation output of the corresponding activation function performed on the layer, aggregating the corresponding activation outputs of the corresponding activation functions performed on the layer over each node input of each node in the layer, and updating respective weights corresponding to the layers based on the calculated gradients of at least one of: (i) the corresponding output of the node-by-node calculation, (ii) the aggregated corresponding activation outputs of each node output of each node in the layer, and (iii) The aggregated corresponding activation output of each node input for each node in the layer.

26. One or more non-transitory computer-readable storage media according to any one of claims 24-25, wherein: Generating a respective predicted performance metric from the features for the job executed by each of the plurality of different accelerator types includes performing collaborative filtering on the embeddings.

27. The one or more non-transitory computer-readable storage media of claim 26, wherein: The machine learning system includes a neural collaborative filtering network trained to receive embeddings from the graph convolutional neural network and perform collaborative filtering on the embeddings to generate the respective predicted performance metric for each of the plurality of accelerator types executing the job according to the performance objective function.

28. The one or more non-transitory computer-readable storage media of claim 27, wherein: The neural collaborative filtering network and the graph convolutional neural network are trained jointly.

29. The one or more non-transitory computer-readable storage media of claim 21, wherein: Generating the respective predicted performance metrics for the job executed by each of the plurality of different accelerator types includes: performing, by a simulator, a simulation of executing the operations on the plurality of different accelerator types; and For each accelerator type and according to the performance objective function, the performance of the simulation for the accelerator type is measured as a corresponding performance metric.

30. The one or more non-transitory computer-readable storage media of claim 21, wherein: The performance metric and the performance objective function are one of the following: Cost performance measures and cost objective functions, Runtime performance metrics and runtime objective functions, or Energy performance metrics and energy objective functions.

Images