Task Scheduling for Machine Learning Workloads

Abstract

To provide a method, a system, and a device for scheduling tasks of machine learning (ML) workloads.SOLUTION: A system including a plurality of hosts, each including a plurality of accelerators, is configured to: receive requests to execute workloads; determine resource requirements to execute the workloads; determine a quantity of hosts assigned to execute tasks of the workload based on the resource requirements and the accelerators for each host; and for each host in the quantity of hosts, generate task specifications based on a memory access topology of the host. The task specifications specify the task to be executed at the host using resources of the host including the plurality of accelerators. The system further provides the task specifications to the hosts and executes the workloads when each host executes the assigned task specified in the task specifications for the host.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] background This specification generally relates to scheduling tasks of a computational workload and allocating resources used to execute the tasks of a computational workload. [Background technology]

[0002] A distributed computing system typically includes a central processing unit (CPU), storage components, and various resources such as image / audio processing accelerators, video transcoding accelerators, or neural network processors (e.g., machine learning (ML) accelerators). These resources can interact to process tasks for computational workloads such as, for example, a workload for training an ML system, or an inference workload for classifying images or generating transcriptions for speech recognition. Summary of the Invention [Problem to be solved by the invention]

[0003] Existing schemes for processing workloads require periodic memory access and data communication between computing resources or groups of resources in a distributed system that are non-local (or remote) to each other. Such non-local memory access operations and data communication are often bandwidth intensive and can lead to performance bottlenecks in compute clusters when host bandwidth for cross-socket (e.g., remote) operations is limited. [Means for solving the problem]

[0004] overview This document describes techniques for improving scheduling and resource allocation when processing machine learning (ML) workloads by allocating tasks of the workload to respective groups of resources across multiple hosts in a large-scale distributed system. Using the techniques described herein, a distributed system can be configured to assign each task of the workload to a group of resources that exchange data communication over a shared or common hardware bus of the distributed system. This allocation scheme takes advantage of the non-uniform memory access (NUMA) of the resource groups. Workload processing times can be reduced by exploiting topology-based resource locality. In some examples, the described techniques can be used to accelerate neural network computations performed on individual tensor processing nodes of a distributed system by performing NUMA-aware scheduling across a fleet of hardware accelerators.

[0005] One aspect of the subject matter described herein may be embodied in a method of scheduling tasks and allocating resources for executing a machine learning workload using hardware accelerators, each of the hardware accelerators configured to execute a neural network including a plurality of neural network layers, the method including: receiving a request to execute a machine learning (ML) workload; and determining, based on the request, resource requirements for executing the ML workload in a distributed processing system including a plurality of hosts, each host of the plurality of hosts including a respective plurality of hardware accelerators, the method further including: determining a number of hosts, each assigned to execute a respective task from a set of tasks forming the ML workload, based on the resource requirements and the respective plurality of hardware accelerators of each host.

[0006] The method includes generating, for each host of the multiple hosts, a respective task specification that specifies the tasks assigned to be executed on the host using resources of the host including the respective plurality of hardware accelerators based on a memory access topology of the host; providing the respective task specifications to the host of the multiple hosts; and executing, by each host of the multiple hosts, the tasks specified in the respective task specification of the host.

[0007] Each of these and other implementations may optionally include one or more of the following features: For example, in some implementations, the memory access topology of each host includes a respective non-uniform memory access (NUMA) topology including a respective memory local to the host, the respective memory including a socket interface coupling the respective memory to a respective hardware accelerator of the respective plurality of hardware accelerators and to one or more other resources of the host.

[0008] In some implementations, executing the tasks specified in the respective task specifications includes performing a plurality of neural network computations to generate an output of each neural network layer of the plurality of neural network layers in response to allocating a respective portion of the plurality of neural network computations to a respective hardware accelerator of the respective plurality of hardware accelerators.

[0009] In some implementations, executing the ML workload includes processing instructions for the respective task specifications using resources of a control group of the host and based on data exchanged between the respective memories, the hardware accelerators, and respective processors included in the resources of the host.

[0010] In some implementations, executing the ML workload includes executing tasks specified in the respective task specifications in response to processing the instructions based on the data exchanged via a hardware socket linking each resource of the control group of the host, the hardware socket defining a local communication bus shared among multiple resources managed by the host.

[0011] In some implementations, the NUMA topology of each of the first hosts is based in part on i) a respective first memory in a respective configuration of resources that are local to the first host, and ii) a respective second different memory in a respective configuration of resources that are local to a second, different host but remote to the first host.

[0012] In some implementations, determining the number of hosts includes obtaining a system file describing a configuration of resources managed by each of the multiple hosts, and determining the number of hosts based on the configuration of the resources described in the system file for each of the multiple hosts. In some implementations, the method includes identifying one or more sockets that bind resources of the host based on a system file describing NUMA socket mapping for each of the multiple hosts, and determining the one or more sockets that bind resources of the host. and forming a control group of said hosts based on said sockets.

[0013] In some implementations, the method includes assigning an ML task of the task specification to the control group of the host based on one or more socket interfaces for accelerators in the control group, the socket interfaces being included in the mapping of the NUMA socket described in the system file, and the method further includes executing the ML task as a process under the control group using the accelerators in the control group.

[0014] Other implementations of this and other aspects include corresponding systems, devices, and computer programs configured to perform the actions of the methods, encoded on a non-transitory computer-readable storage device. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof installed on the system that, when operated, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0015] One aspect of the subject matter described herein may be embodied in a system that receives a request to execute a workload and determines, based on the request, resource requirements for executing the workload. The system includes a plurality of hosts, each host including a plurality of accelerators. The system determines a number of hosts to be assigned to execute tasks of the workload based on the resource requirements and the accelerators of each host. For each host of the number of hosts, the system generates a task specification based on a memory access topology of the host. The specification specifies the tasks to be executed on the host using resources of the host including the plurality of accelerators. The system provides the task specification to the hosts and executes the workload as each host executes an assigned task specified in the task specification of the host.

[0016] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages: The techniques described herein can alleviate performance bottlenecks in a system by reducing or preventing the occurrence of non-local memory access operations and data communication when hosts of the system execute tasks of a workload. Compared to previous approaches, the techniques described can be used to reduce the time required to process a workload by taking advantage of resource locality based on a non-uniform memory access (NUMA) topology of resources or groups of resources managed by each host of the system.

[0017] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram of an example of a computing system that schedules tasks to be executed to execute a machine learning workload. [Diagram 2] FIG. 2 is a block diagram of an example of resources managed by a host included in the computing system of FIG. 1. [Diagram 3] FIG. 1 illustrates an example of computational logic that may be executed to generate a task specification for executing a machine learning workload. [Figure 4] FIG. 1 illustrates an example process for scheduling tasks to be executed to execute a machine learning workload. [Diagram 5] FIG. 2 illustrates an example process for generating a task specification that is provided to a host in the computing system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Detailed Description Like reference numbers and designations in the various drawings indicate like elements.

[0020] A distributed system has multiple nodes (e.g., hosts) that include hardware devices used to execute computational workloads. The nodes may form individual hardware computation clusters that process data and execute workloads, or host devices within a cluster. Each node may include multiple resources. For example, a resource may be a processor or CPU, memory, or a peripheral component interconnect (PCI) device (e.g., hardware accelerator), and each host may include multiple resources that form a resource group.

[0021] Each resource may have specific hardware or socket connections that make some resources in a node remote or local to other resources. Approaches to processing workloads in distributed systems often require accessing memory and performing operations that involve moving data between resources that are non-local (or remote) to each other. As discussed above, such non-local memory access and data transfer operations can lead to performance bottlenecks in clusters or hosts that have limited bandwidth for cross-socket (e.g., remote) operations.

[0022] In this context, techniques are described that improve scheduling of tasks of a computational workload in a computational cluster in a distributed system and allocating resources of the system used to execute the tasks that form the computational workload. With respect to scheduling, the techniques include an improved process of allocating tasks of a workload (e.g., an ML workload) to respective groups of resources managed by individual hosts of a large-scale distributed system. For example, the system is configured to allocate specific tasks of the workload to specific groups of resources, where the individual resources in the groups exchange data communications over a shared or common hardware bus of the distributed system. The process of allocating specific tasks to specific resource groups of hosts is performed by exploiting resource locality within the computational cluster, where the locality is based on a non-uniform memory access (NUMA) topology of the resource groups.

[0023] Also described is a technique that provides an approach to perform NUMA-aware task scheduling and resource allocation across multiple hosts in separate compute clusters of a distributed system. For example, a controller of a compute cluster can pass a set of protocol bits that describe a job or task of a workload that requires NUMA locality. The technique exploits the NUMA locality of a cluster or of hosts within a cluster by assigning one or more tasks to a specific set of resources or devices, such as a resource group that includes CPUs, memory, and peripheral component interconnect (PCI) devices (e.g., hardware accelerators).

[0024] A host that manages multiple resource groups is operable to receive and process the protocol bits passed by the master controller of the cluster, for example, the host processes the protocol bits based on the determined socket topology of the resource group and allocates a particular set of resources from the same NUMA socket to execute tasks for a particular job or portion of a workload specified by the protocol bits. The host is operable to bind or assign a set of machine learning tasks of a task specification to resource groups (or control groups) constructed at the host. For example, the host may bind tasks to a given resource group based on information conveyed by a protocol bit to reduce or prevent the occurrence of non-local memory or data access operations that may degrade the performance or execution of computations for a given task.

[0025] The described approach for NUMA-aware task scheduling and resource allocation enables distributed systems to optimize the use of NUMA locality to reduce bandwidth requirements and improve computation time for executing certain workloads. For example, this can be achieved based on at least exploiting cluster NUMA locality, whereby certain types of tasks are assigned to specific sets of co-located devices or resource groups such that cross-socket or cross-node communication can be reduced, thus freeing up bandwidth for other computations.

[0026] 1 is a block diagram of an example distributed computing system 100 that schedules tasks to be executed to execute a computational workload. System 100 may be a large-scale distributed hardware computing system that includes multiple computational clusters 102, each of which includes multiple hosts 104, and each of which includes multiple computational resources 105.

[0027] One or more groups of resources may be managed by a host 104 of the distributed system 100, and each of the multiple computing clusters 102 may include multiple hosts 104. More specifically, each host 104 is configured to manage two or more individual resources 105 that form a group of resources. Alternatively, a group of resources managed by a host 104 may be referred to herein as a resource group. Thus, in some cases, a resource 105 may represent an individual resource, such as a processor or memory device, while in other cases, a resource 105 may represent multiple resources, such as two or more processors, two or more memory banks, two or more hardware accelerators, or a combination of each. Resource groups of a host 104 are described in more detail below with reference to FIG. 2.

[0028] In some implementations, the host 104 is a hardware computing device (e.g., a computer or a server). In some implementations, the host 104 is a virtual machine of a distributed system (or computing cluster), a software construct for managing a group of computing resources 105, or both. The system 100 may include M computing clusters 102, where each computing cluster 102 of the M computing clusters may include N hosts, where each of M and N is an integer greater than or equal to 1.

[0029] In some implementations, each of the computing clusters 102 includes a set of machines (e.g., hardware or virtual machines) that form the hosts 104 of the cluster 102. As shown in FIG. 1, a cluster 102 may include multiple controllers 108, each of which functions to allocate workload tasks to one or more of the hosts 104 in the cluster 102.

[0030] Each of the computing clusters 102 may include a scheduler 106 in communication with a master controller 108 ("controller 108") of the cluster 102, and link shards 110 accessible to the master controller 108. The controller 108 is responsible for generating task specifications and preparing and sending instructions and commands to the hosts 104, and for receiving instructions and commands from the hosts 104. and is responsible for updating the current processing state of the host 104 based on the response of the status logic 110. In some implementations, each controller 108 includes status logic 110 that manages communication with a subset of the hosts 104. For example, the status logic 110 is executed by the controller 108 to send commands to the subset of the hosts 104 to obtain information regarding the processing state of the hosts 104, and to receive responses from the hosts 104. For example, the status logic 110 is used to receive and process host reports indicating whether an assigned task is completed or in progress. If a host 104 does not provide a status report after a threshold number of attempts to obtain information regarding the host's processing state exceeds the status logic 110, the status logic 110 may determine that a task assigned to the host 104 has stalled. In some cases, the status logic 110 is operable to aggregate and compress processing state information reported by the hosts 104 to reduce the size of the update load received at the master controller 108.

[0031] As described in more detail below, the scheduler 106 and the controller 108 interact or communicate to schedule and assign tasks of a workload for execution on a particular host 104. Although the scheduler 106 is shown in FIG. 1 as being separate from the controller 108, it may also be integrated into the controller 108. In some implementations, the scheduler 106 is any processing element of the compute cluster 102, and its functionality may be integrated into the allocation and control functionality configured in the controller 108.

[0032] The controllers 108 are operable to assign tasks of a workload based at least on a request 112 to execute one or more workloads and a hardware configuration of resources managed by the hosts 104. For example, each of the controllers 108 may be a logically centralized controller that generates instructions based on parameters in the request 112 received at the cluster 102 and based on the hardware socket topology of the resources 105 (or resource groups) included in the hosts 104. In some implementations, each host 104 in the subset of hosts 104 is configured as a "slave" computing asset under a particular master controller 108. In this implementation, the master controller 108 generates instructions based on parameters in the request 112 and the hardware socket topology at each host 104 of the subset of hosts 104 that are "slave" under the particular master controller 108.

[0033] A host 104 may include multiple resources 105, e.g., hundreds or thousands of resources or devices, corresponding to machines or hardware devices. The resources 105 in a host 104 may be diverse or heterogeneous in many respects. For example, each group of resources 105 managed by a host 104 may differ in terms of processing units (e.g., CPU, RAM, disk, network), processor type, processing speed, performance, and capabilities such as external IP addresses or flash storage. More specifically, for each computational cluster 102, each of the multiple hosts 104 in the cluster 102 includes one or more dedicated hardware circuits that interact with the host's other resources to perform the tasks of a workload.

[0034] For example, the dedicated hardware circuitry may be a hardware accelerator, a graphics processing unit (GPU) hardware accelerator, or a neural network processor. In the example of Figure 1, the system 100 may include a first host 104-1, a second host 104-2, a third host 104-3, and N additional hosts 104-n. In some implementations, the dedicated circuitry and resources 105 of the first host 104-1 may differ (e.g., slightly or significantly) from the dedicated circuitry and resources 105 of the second host 104-2.

[0035] For example, the first host 104-1 may include 10 GPU hardware accelerators each configured to perform location-based and in-memory analytics or GPU-accelerated database queries, and the second host 104-2 may include 20 neural network processors each configured to perform convolutional neural network (CNN) or recurrent neural network (RNN) models. In some implementations, the 20 neural network processors may be configured to perform binary execution on trained inference models to accelerate the execution of floating-point based inference models, integer quantization inference models, or both.

[0036] The system 100 uses the controller 108 to generate instructions for allocating and controlling the execution of individual tasks, such as tasks executable on one or more machines of a particular host 104. The decision to allocate a particular task to a particular resource group of a host 104 is made with particular emphasis on exploiting resource locality within the host 104 of the computing cluster 102. The resource locality is based on the hardware topology of the resource groups at the host 104, and more specifically, on the non-uniform memory access (NUMA) topology of the resource groups, as described below.

[0037] The system 100 is configured to generate a system topology that includes a hardware topology of each computing cluster 102 and each host 104 of the computing cluster. The hardware topology is configured to identify i) the connectivity (e.g., socket connections and interfaces) of the hosts' multiple devices and resources, and ii) the local communication buses that enable data transfer between the resources 105 of the hosts 104.

[0038] The system 100 is configured to identify the location of each resource or peripheral device coupled to a connection point or component interface of a hardware socket in the host 104. For example, the host 104 may execute program code (e.g., firmware) associated with a system BIOS of a hardware computer managed by the host 104 to identify resource locations and types of resources 105 (e.g., processor and memory) coupled to the computer's motherboard. In some implementations, the operating system of the host 104 may use the chipset of the hardware computer to obtain a detailed list of information about connected data buses and peripheral devices in the computer managed by the host 104. For example, the list may be based on a common portable interconnect library (e.g., libpci) that represents the interconnect configuration space of an operating system running on the computer's processor.

[0039] The controller 108 is operable to send outstanding instructions to each host 104 to execute tasks, and to send commands to each host 104 to obtain information regarding the current processing state of a particular machine or resource group 105 managed by the host 104. In some implementations, the controller 108 dynamically sends commands to obtain information regarding the processing state. Alternatively, the controller 108 may send commands to obtain information regarding the processing state with reference to a predefined schedule (e.g., every few seconds), the schedule being based on the particular task being performed by the host 104. In general, each controller 108 is operable to control the respective communication rates between the various resources of the host and between different resource groups based on the instructions and commands the controller 108 sends to the host 104.

[0040] FIG. 2 illustrates a block diagram of an example resource group 200 managed by a host 104 of an example computing cluster 102. As described above, the host 104 A host 104 may contain hundreds or thousands of resources corresponding to applications or hardware devices. The resources 105 within a resource group of a host 104 may be diverse or heterogeneous in many respects. For example, each group of resources 200 managed by a host 104 may differ in terms of processing units (e.g., CPU, RAM, disk, network), processor type, processing speed, overall performance, and capabilities such as external IP addresses or flash storage.

[0041] 2, the memory access topology of each host 104 may include a non-uniform memory access (NUMA) topology or socket 202-1, 202-2 of each of one or more resource groups 200 managed by the host 104. The NUMA topology of a resource group 200 may include multiple processors (P) 204, or multiple processor cores (P), memory resources, e.g., random access memory (RAM), and one or more special-purpose circuitry, such as a hardware accelerator 208. Individual resources in the NUMA topology may form a local NUMA node corresponding to either the NUMA socket 202-1 or 202-2.

[0042] For example, a local NUMA node may be formed based on resources in a group that exchange data communications over a shared or common hardware bus 210. Each resource in a local NUMA node may be local to another resource if the resources are connected to a common socket in this node through an interface (or socket) connection. In some implementations, each hardware accelerator 208 connects to other resources of the NUMA node through a PCI or PCI-e socket connection.

[0043] As used herein, NUMA refers to a computer memory design used in distributed multiprocessing systems, where memory access time is determined by memory location relative to a processor (P) 204 or processor core. Under NUMA, a processor 204 can access its own local memory 206-1 faster than non-local memory, such as memory 206-2 that is local to another processor or memory shared between processors.

[0044] An example resource group 202 may include multiple interconnect locations 212. For example, each of interconnect locations 212-1 and 212-2 may correspond to a respective component interface for establishing a data connection between resources 105 of the host 104, such as between memory 206-1 and hardware accelerator 208 of the host 104. In some implementations, resources 105 of resource group 200 communicate data communications via hardware sockets that link local resources of NUMA socket 202-1, which define a local communication bus 210 shared among multiple resources managed by the host 104.

[0045] In some implementations, the NUMA topology of each of the first NUMA sockets 202-1 is based in part on i) a respective first memory 206-1 in a respective configuration of resources that are local to the NUMA socket 202-1, and ii) a respective second different memory 206-2 in a respective configuration of resources that are local to a second different NUMA socket 202-2 but remote to the first NUMA socket 202-1.

[0046] 3 illustrates an example task specification 300 based on computational logic 302 executed by system 100 to execute a computational workload. As shown in FIG. 3, logic 302 defines multiple computational blocks, each of which includes instructions (e.g., programmed code / instructions). The instructions may be executed in the system 100 using a processing unit in the controller 108, a processing unit in the host 104 and other resources 105, or a combination of each.

[0047] The computation logic 302 may be a programmatic representation of an example task specification 300 for scheduling tasks and allocating resources for executing an ML workload on the system 100. In some implementations, the ML workload is executed using hardware accelerators, each configured to execute a neural network that includes multiple neural network layers. Instructions may be stored in one or more non-transitory machine-readable storage media of the system 100 and are executable by one or more processors of the system 100 to perform operations to execute the tasks of the workload.

[0048] For example, operations may be performed to generate instructions and protocol bits (e.g., for a task specification) that are provided to a particular host 104 to execute a task of the ML workload. In some implementations, the protocol bits are represented by coded signals, binary-valued data words, or other related parameters or data values. The coded signals and binary values ​​are received and processed (or interpreted) at the host 104 to determine the assignment of tasks of the task specification. Generally, each host 104 is configured to run or execute one or more tasks of a workload that includes tasks of multiple different workloads that may be assigned to the host 104. When a request 112 is received by a computational cluster 102, a scheduler 106 and a controller 108 of the cluster 102 interact to scan for the request 112.

[0049] For example, the controller 108 may scan the request 112 to identify parameters (e.g., protocol bits) in the request 112 that specify the CPU, memory, and accelerator requirements of various tasks in the workload (304). The controller 108 may determine, based on the parameters and values ​​in the request 112, that an example workload includes 16 tasks, each of which requires a total resource allocation of 96 CPUs and 4 dedicated circuits (e.g., hardware accelerators). For example, the request 112 may include a scalar resource parameter that specifies the number (4) of hardware accelerators to be used to execute each of the 16 tasks. In some implementations, the scalar resource parameter may include a subtype that specifies the type of hardware accelerator to be used to process the workload. For example, the subtype may specify that each of the four hardware accelerators is a neural net processor configured to accelerate the execution of a model trained for feature recognition.

[0050] The package field of the computation logic specifies a task binary for performing each of the 16 tasks (306). For example, the task binary may be a particular type of neural network or inference model that should be executed or run on a hardware accelerator to perform computations to perform a particular one of the 16 tasks. In some cases, the task binary is derived from a scalar resource subtype that specifies the type of hardware accelerator to be used to process the tasks of the workload.

[0051] The controller 108 is operable, in response to identifying the parameters in the request 112, to determine an allocation scheme for scheduling and assigning tasks to the hosts 104 in the cluster 102 based on the parameters of the request 112 and based on the hardware socket topology of the resources 105 (or resource groups 200) in the hosts 104. The controller 108 determines an allocation scheme for scheduling and assigning tasks to the hosts 104 in the cluster 102 based on the parameters of the request 112 and based on the hardware socket topology of the resources 105 (or resource groups 200) in the hosts 104. The request 112 generates respective task specifications based on the allocation scheme to be assigned. For example, each of the parameters and corresponding parameter values ​​in the request 112 may represent a scheduling constraint for the scheduler 108. In some implementations, the request 112 may assign a priority to each of the parameters to further constrain the scheduler 108 and the controller 108. For example, a priority assigned to an accelerator subtype or a CPU core may constrain the controller 108 to a particular host 104 that has a particular type of hardware accelerator or a particular number of available CPUs.

[0052] The controller 108 determines task allocation and generates task specifications by analyzing parameters of the requests 112 against details of the hardware socket topology of each resource group managed by one or more of the hosts 104 that are "slaves" under the controller 108. For example, the controller 108 is operable to scan the hardware socket topology of each resource group of the host 104 to determine the locality of the resource 105, determine whether a resource or type of resource satisfies the constraints of the request 112, and determine resource availability. In some examples, the controller 108 determines resource availability from among resources that are local to a particular NUMA node and that satisfy one or more of the constraints of the request 112. In some implementations, the hardware socket topology of each host 104 is based on the respective memory access topology of each resource group 200 within the host 104.

[0053] In a NUMA system, there are multiple NUMA nodes, each of which is made up of a set of processors and memory. As described above, accesses by a processor 204 in the same NUMA node 202-1 to memory 206-1 are local, whereas accesses by a processor 204 in a NUMA node 202-1 to memory 206-2 in another NUMA node 202-2 are remote. In some implementations, remote accesses may involve multi-hop operations, so that remote accesses may take multiple cycles compared to local accesses. Due to this asymmetric memory access latency, performance of distributed processing can be improved by keeping memory accesses local or maximizing memory locality. In some implementations, CPU load balancing across NUMA nodes, in conjunction with exploiting NUMA locality, can lead to further performance improvements.

[0054] The master controller 108 is configured to encode 308 one or more constraints in the task specification. For example, the task specification 300 may include scheduling constraints derived from parameters and values ​​in the request 112. For example, the controller 108 may translate parameters in the request 112 into task constraints that instruct the host 104 to load data used to perform computations for the task on host machines located in a particular cloud zone. For example, a cloud zone may be a particular physical or geographic location of a data center that contains a particular set of hardware accelerator resources required to perform data computations for a given task of a workload.

[0055] The compute cluster 102 is configured to determine an allocation scheme that schedules and assigns tasks to particular resource groups 200 of hosts 104 in a manner that exploits resource locality across multiple hosts 104 in the compute cluster 102. An example task specification 300 shown in Figure 3 provides a simplified task specification that represents a compute cluster 102 receiving requests to execute 16 tasks. Each of the tasks occupies a host 104 that includes two NUMA nodes. In this example, each of the tasks requires a total resource allocation of 96 CPUs and 4 dedicated circuitry (e.g., hardware accelerators).

[0056] The task specification 300 includes parameters that define the host 104's resource allocation, such as how a particular host 104 should allocate its CPU cores from a particular NUMA node (310). In the example of Figure 3, a balanced CPU allocation is shown, where 48 processor cores are allocated from each NUMA node to satisfy a total resource allocation of 96 CPUs for each task. In other examples, the controller 108 may generate a task specification that specifies an unbalanced allocation, such as 36 CPUs from a first NUMA node and 60 CPUs from a second NUMA node.

[0057] 4 illustrates an example process 400 for scheduling tasks to be performed to execute a machine learning workload. Process 400 may be implemented or performed using system 100 described above. Thus, the description of process 400 may refer to the above-described computational resources of system 100, as well as other components described herein. In general, the computational steps or process flows in the description of process 400 may be grouped or configured to occur in different orders and are not limited to the numerical order described herein.

[0058] Now referring to process 400, system 100 receives 402 a request to execute a workload using one or more of its compute clusters. In some implementations, process 400 corresponds to a method of scheduling tasks and allocating resources to execute the workload using hardware accelerators and other resources of host 104. In some examples, the workload is a training or inference workload associated with a particular machine learning operation, such as video transcoding, image processing, voice processing, autonomous vehicle navigation, or image recognition.

[0059] The request 112 may be a request to perform an ML workload, such as an inference workload for detecting objects in an image or for recognizing words from an audio utterance. In this context, one or more of the hardware accelerators may be configured to execute a neural network that includes multiple neural network layers, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). The received request 112 may include parameters that specify a particular type of neural network configuration (e.g., CNN or RNN) to be used to perform the tasks of the workload.

[0060] The received request 112 may also be followed by a second request 112 to deploy the particular neural network on the cluster 102, for example, using a resource group of the host 104. The second request may be followed by an instruction or command that causes the controller 108 (or the host 104) to obtain parameters of a set of weights for a particular neural network layer. For example, the set of weights may be obtained from a memory location managed by the host 104 based on a location address specified in the instruction. In some implementations, the memory that stores the weights obtained by the host 104 is one of multiple local resources of a NUMA node that defines a resource group at the host 104. Similarly, the instruction may cause the controller 108 (or the host 104) to access other memory locations to fetch inputs for processing through the neural network layer and generate outputs of the neural network layer using local resources 105 of the NUMA node in the host 104. In some implementations, processing a particular portion of the input identified in the request 112 through a neural network layer to generate a layer output may represent the execution of one or more tasks of a larger workload that may be processed across multiple hosts 104 of a single compute cluster 102 or across multiple compute clusters 102.

[0061] The system 100 determines resource requirements based on the request (404). The resource requirements may indicate specific details about the resources of the system 100 associated with the workload request 112, such as the type and amount of computational resources needed to execute a set of tasks representing the ML workload. For example, the resource requirements may specify a particular processor or processor type, processing power or speed, amount of memory or memory size, number of hardware accelerators, or a measure of resource locality for resources in a distributed system.

[0062] The system 100 determines (406) a number of hosts 104 to be assigned to execute respective tasks of the ML workload based on the resource requirements and the number of hardware accelerators of each host. The system 100 generates (408) for each host 104 of the number of hosts 104 a respective task specification based on the host's memory access topology. The host's memory access topology may be based on one of a number of respective NUMA topologies of each resource group 200 of the host 104. In some implementations, the particular NUMA topology of a resource group 200 is specific to a local NUMA node and includes a respective memory (M) that is local to other resources of the group. Each memory (M) may include a socket interface that locally couples the memory to at least one hardware accelerator and one or more other resources of the NUMA node.

[0063] The controller 108 generates a task specification in response to scanning the parameters of the request 112 and cross-referencing the respective hardware topologies for each resource group 200 in the set of hosts 104 that are assigned to the controller 108 as slave assets. The system 100 provides the respective task specifications to the hosts (410). For example, the controller 108 can provide multiple respective task specifications to different hosts 104, and the system 100 executes the ML workload by executing the tasks specified in the respective task specifications on each host (412).

[0064] 5 illustrates an example process for generating a task specification to be provided to a host of system 100. Like process 400, process 500 may be implemented or performed using system 100, and the description of process 500 may refer to resources of system 100, including other components described herein. In general, the computational steps or process flows in the description of process 500 may be grouped or configured to occur in different orders and are not limited to the numerical order described herein.

[0065] Now referring to process 500, system 100 is configured to identify one or more sockets for resources of a host using a system file that describes the NUMA socket mapping of each host (502). For example, controller 108 is configured to determine the NUMA socket mapping based on the hardware topology of resource groups at host 104. The NUMA socket mapping is used to indicate the NUMA topology of each resource group managed by host 104.

[0066] The controller 108 builds 504 control groups for the host 104 using the sockets for resources described in the host's NUMA socket mappings. In some implementations, the controller 108 passes one or more protocol bits to the host 104, and the host's scheduler uses the protocol bits (e.g., in the request or in the task specification) to build control groups based on the locality of resources 105 among various resource groups at the host 104. For example, the controller 108, or the host's 104 scheduler, may use the NUMA topology of each resource group and the received and the types of resources in the hosts 104 that satisfy some (or all) of the constraints in the requirements 112 .

[0067] The controller 108 cooperates with the host 104 to bind or assign ML tasks of the task specification to control groups established by the host 104. In some implementations, protocol bits passed by the controller 108 to the host 104 are used by the scheduler of the host 104 to bind or assign ML tasks of the task specification to control groups established by the host 104. The protocol bits may be included in the task specification to indicate one or more constraints or requirements of the task specification. The protocol bits may be represented by one or more coded signals, binary valued data words, or other related parameters or data values. The coded signals and binary values ​​may be received and processed or otherwise interpreted by the scheduler of the host to determine the assignment of tasks of the task specification. In some implementations, the protocol bits may be associated with the task specification, but may be provided separately from the task specification.

[0068] The host 104 is operable to bind the task-specific ML task to the control group based at least on the socket interface for the particular type of accelerator in the control group, including the availability of resources in the control group (506). The host 104 executes the task-specific ML task as a process under the control group using memory resources and hardware accelerators in the control group (508). In some implementations, the host 104 can execute the ML task using one or more other resources of the control group, including local memory. In some cases, non-local memory may be utilized, but when utilizing such resources, the non-local memory is balanced against the required computational bandwidth or any performance impact that may result.

[0069] The embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed herein and their structural equivalents), or in a combination of one or more of them.Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing apparatus.

[0070] Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical or electromagnetic signal) that is generated to encode and transmit information to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0071] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. These processes and logic flows may also be performed by, or an apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit)).

[0072] A computer suitable for executing a computer program may include or be based on, by way of example, a general-purpose or special-purpose microprocessor, or both, or other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operatively coupled to the mass storage devices for receiving data from, transferring data to, or both. However, a computer need not have such devices.

[0073] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, and magnetic disks, such as internal hard disks or removable disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0074] Although the specification contains many specific implementation details, these should not be construed as limiting the scope of the invention or the claims, but as describing features that may be specific to certain embodiments of a particular invention. Certain features described in the specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination and may be initially described as such, one or more features in the described combination may in some cases be excluded from this combination, and the described combination may be directed to a subcombination or a variation of the subcombination.

[0075] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all of the operations shown must be performed to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0076] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method of scheduling tasks and allocating resources for executing a machine learning ("ML") workload using hardware accelerators, each of the hardware accelerators configured to execute a neural network including a plurality of neural network layers, the method comprising: determining resource requirements for executing the ML workload using a plurality of hosts based on requests to execute the ML workload; For each host of the plurality of hosts, assigning tasks to be executed on the host using multiple hardware accelerators of the host based on a non-uniform memory access (NUMA) topology of NUMA nodes in the host; executing the ML workload by executing the tasks assigned to the hosts; the NUMA node includes a memory local to the host, the memory having a socket interface coupling the memory to each of the plurality of hardware accelerators of the host; The NUMA topology of a first host of the plurality of hosts comprises, in part: i) a first memory of a configuration of resources local to the first host; ii) a second different memory in a configuration of resources that are local to a second different host but remote to the first host.

2. The tasks are specified in respective task specifications that are provided to the host, and executing the tasks comprises: allocating respective portions of a plurality of neural network computations to respective hardware accelerators of the plurality of hardware accelerators; performing respective portions of the plurality of neural network calculations using the plurality of hardware accelerators; and generating an output for each neural network layer of the plurality of neural network layers based on the plurality of neural network calculations.

3. Executing the ML workload comprises:

3. The method of claim 1, further comprising: processing instructions for a respective task specification using each resource of a control group of the first host and based on data exchanged between the first memory, a hardware accelerator of the first host, and each processor of the resources of the first host.

4. Executing the ML workload comprises:

4. The method of claim 3, further comprising: executing a task specified in the respective task specification in response to processing the instruction based on the data exchanged over a hardware socket linking each resource of the control group of the first host, the hardware socket defining a local communications bus shared among multiple resources managed by the first host.

5. obtaining a system file describing a configuration of resources managed by each host of the plurality of hosts; The method of any one of claims 1 to 4, further comprising: determining a number of hosts on which to execute the ML workload based on the resource configuration described in the system file of each host among the plurality of hosts.

6. For each host among the plurality of hosts, further comprising: allocating ML tasks of a task specification to the control group of the host based on one or more socket interfaces for hardware accelerators in the control group, the socket interfaces being included in a NUMA socket mapping described in a system file describing a NUMA socket mapping of each host of the plurality of hosts; The method of any one of claims 1 to 5, further comprising: executing the ML task as a process under the control group using the hardware accelerator in the control group.

7. A system configured to schedule tasks and allocate resources for executing machine learning ("ML") workloads using hardware accelerators, each of the hardware accelerators configured to execute a neural network including multiple neural network layers, the system comprising: one or more processing devices; and one or more non-transitory machine-readable storage devices storing instructions executable by the one or more processing devices to cause the one or more processing devices to perform operations, the operations including: determining resource requirements for executing the ML workload using a plurality of hosts based on requests to execute the ML workload; For each host of the plurality of hosts, assigning tasks to be executed on the host using multiple hardware accelerators of the host based on a non-uniform memory access (NUMA) topology of NUMA nodes in the host; executing the ML workload by executing the tasks assigned to the hosts; the NUMA node includes a memory local to the host, the memory having a socket interface coupling the memory to each of the plurality of hardware accelerators of the host; The NUMA topology of a first host of the plurality of hosts comprises, in part: i) a first memory of a configuration of resources local to the first host; ii) a second distinct memory for configuration of resources that are local to a second distinct host but remote to said first host.

8. The tasks are specified in respective task specifications that are provided to the host, and executing the tasks comprises: allocating respective portions of a plurality of neural network computations to respective hardware accelerators of the plurality of hardware accelerators; performing respective portions of the plurality of neural network calculations using the plurality of hardware accelerators; and generating an output for each neural network layer of the plurality of neural network layers based on the plurality of neural network calculations.

9. Executing the ML workload comprises:

9. The system of claim 7 or 8, further comprising: processing instructions for a respective task specification using each resource of a control group of the first host and based on data exchanged between the first memory, a hardware accelerator of the first host, and each processor of the resources of the first host.

10. Executing the ML workload comprises:

10. The system of claim 9, further comprising: executing a task specified in each task specification in response to processing the instruction based on the data exchanged via a hardware socket linking each resource of the first host, the hardware socket defining a local communication bus shared among multiple resources managed by the first host.

11. The operation further comprises: obtaining a system file describing a configuration of resources managed by each host of the plurality of hosts; and determining a number of hosts to execute the ML workload based on the resource configuration described in the system file of each host among the plurality of hosts.

12. For each host among the plurality of hosts: assigning ML tasks of a task specification to the control group of the host based on one or more socket interfaces for hardware accelerators in the control group, the socket interfaces included in a NUMA socket mapping described in a system file describing NUMA socket mappings for each host of the plurality of hosts; The system of any one of claims 7 to 11, comprising executing the ML task as a process under the control group using the hardware accelerator in the control group.

13. A program for scheduling tasks and allocating resources for executing machine learning ("ML") workloads using hardware accelerators, each of the hardware accelerators configured to execute a neural network including multiple neural network layers, the program causing one or more processors to perform a method as recited in any one of claims 1 to 6.

Images