Resource allocation system and method applied to GPU cluster, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a resource allocation system and method applied to a GPU cluster, electronic equipment, a storage medium and a computer program product. The system comprises a load scheduler, a topology scheduler and a topology manager deployed in each GPU node in the GPU cluster, the topology manager deployed in each GPU node is used for determining an equipment topological graph of the GPU node; the load scheduler is used for determining at least one candidate GPU node meeting a resource request in the GPU cluster according to the resource request of the target load; and the topology scheduler is used for determining a target GPU node according to the equipment topological graph of each candidate GPU node on the basis of a shortest topological distance principle, and determining at least one target GPU-network interface equipment pair meeting the resource request in the target GPU node. According to the embodiment of the invention, the optimal equipment resource with the shortest topological distance can be allocated to the target load, the data transmission delay is effectively reduced, and the network data transmission performance is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more particularly to a resource allocation system and method, electronic device, storage medium and computer program product applied to GPU clusters. Background Technology

[0002] A Graphics Processing Unit (GPU) cluster is a computer cluster in which each GPU node is equipped with a GPU device. Leveraging the computing power of modern GPUs, GPU clusters can perform very fast computations. In scenarios such as high-performance computing, deep learning, and big data analytics, network performance is a critical issue in current GPU clusters (e.g., multi-GPU clusters). Allocating the GPU devices and network interface devices (e.g., NIC devices) on GPU nodes to workloads (e.g., containers or virtual machines) that utilize the GPU cluster allows the workloads to directly interact with the GPU devices and network interface devices, improving data transmission performance. Therefore, there is an urgent need for a resource allocation system for GPU clusters to optimize the allocation of GPU devices and network interface devices and improve network data transmission performance. Summary of the Invention

[0003] In view of this, this disclosure presents a resource allocation system and method, electronic device, storage medium and computer program product for GPU clusters.

[0004] According to one aspect of this disclosure, a resource allocation system for a GPU cluster is provided, comprising: a load scheduler, a topology scheduler, and a topology manager deployed within each GPU node in the GPU cluster; the topology manager deployed within each GPU node is used to determine a device topology map of the GPU node, wherein the device topology map of the GPU node is used to indicate the topological relationship between GPU devices and network interface devices included in the GPU node; the load scheduler is used to determine at least one candidate GPU node in the GPU cluster that satisfies the resource request of a target load, wherein the resource request is used to indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices; the topology scheduler is used to determine a target GPU node based on the shortest topological distance principle according to the device topology map of each candidate GPU node, and to determine at least one target GPU-network interface device pair that satisfies the resource request in the target GPU node.

[0005] In one possible implementation, a topology manager deployed within each GPU node is used to generate and report a topology configuration file of the GPU node to the topology scheduler after determining the device topology map of the GPU node; the topology scheduler is used to obtain the device topology map of each candidate GPU node by reading the topology configuration file of each candidate GPU node.

[0006] In one possible implementation, the topology scheduler is configured to: for any candidate GPU node, identify at least one GPU-network interface device pair in the candidate GPU node; and determine the score of each GPU-network interface device pair according to the device topology map of the candidate GPU node and a preset scoring principle.

[0007] In one possible implementation, for any GPU-network interface device pair, the higher the score of the GPU-network interface device pair, the shorter the topological distance between the GPU device and the network interface device in the GPU-network interface device pair.

[0008] In one possible implementation, the preset scoring principles include: when the GPU device and network interface device in a GPU-network interface device pair pass through a switch, the GPU-network interface device pair is scored as a first score; when the GPU device and network interface device in a GPU-network interface device pair pass through multiple switches, the GPU-network interface device pair is scored as a second score, wherein the second score is less than the first score; when the GPU device and network interface device in a GPU-network interface device pair pass through a CPU, the GPU-network interface device pair is scored as a third score, wherein the third score is less than the second score; and when the GPU device and network interface device in a GPU-network interface device pair pass through a non-uniform memory access node, the GPU-network interface device pair is scored as a fourth score, wherein the fourth score is less than the third score.

[0009] In one possible implementation, the topology scheduler is configured to: for any candidate GPU node, determine the score of the candidate GPU node based on the score of each GPU-network interface device pair in the candidate GPU node; determine the candidate GPU node with the highest score as the target GPU node; and determine the target GPU-network interface device pair in the target GPU node based on the score of each GPU-network interface device pair in the target GPU node.

[0010] In one possible implementation, the topology scheduler is configured to: schedule the target load to the target GPU node; and generate a resource allocation tag corresponding to the target load, wherein the resource allocation tag is used to indicate the target GPU-network interface device pair.

[0011] In one possible implementation, the system further includes: a node controller, a GPU device management plugin, and a network interface device management plugin deployed within each GPU node in the GPU cluster; the node controller deployed within the target GPU node is configured to parse the resource allocation tag after detecting that the target load is scheduled to the target GPU node, and to send the resource allocation tag to the GPU device management plugin and the network interface device management plugin deployed within the target GPU node; the GPU device management plugin deployed within the target GPU node is configured to allocate the GPU devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag; the network interface device management plugin deployed within the target GPU node is configured to allocate the network interface devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag.

[0012] According to another aspect of this disclosure, a resource allocation method for a GPU cluster is provided, comprising: determining a device topology map of the GPU node using a topology manager deployed within each GPU node in the GPU cluster, wherein the device topology map of the GPU node is used to indicate the topological relationship between GPU devices and network interface devices included in the GPU node; determining at least one candidate GPU node in the GPU cluster that satisfies the resource request based on a resource request of a target load using a load scheduler deployed in the GPU cluster, wherein the resource request is used to indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices; determining a target GPU node based on the device topology map of each candidate GPU node, using a topology scheduler deployed in the GPU cluster, based on the principle of shortest topological distance, and determining at least one target GPU-network interface device pair that satisfies the resource request in the target GPU node.

[0013] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0016] The resource allocation system applied to GPU clusters according to embodiments of this disclosure includes a topology manager deployed within each GPU node that determines the device topology map of the GPU node to indicate the topological relationships between GPU devices and network interface devices included in the GPU node. First, a load scheduler determines at least one candidate GPU node in the GPU cluster that satisfies the resource request of the target load. Then, the topology scheduler can determine the target GPU node based on the shortest topological distance principle according to the device topology map of each candidate GPU node, and determine at least one target GPU-network interface device pair including a first number of target GPU devices and a second number of target network interface devices in the target GPU node. This enables the allocation of optimal device resources with the shortest topological distance to the target load, effectively reducing data transmission latency and improving network data transmission performance.

[0017] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0019] Figure 1 A block diagram is shown of a resource allocation system applied to a GPU cluster according to an embodiment of the present disclosure.

[0020] Figure 2 A schematic diagram of a resource allocation system applied to a GPU cluster according to an embodiment of the present disclosure is shown.

[0021] Figure 3 This diagram illustrates a target GPU node allocating resources to a target load according to an embodiment of the present disclosure.

[0022] Figure 4 A flowchart illustrating a resource allocation method applied to a GPU cluster according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0025] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0026] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0027] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0028] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0029] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0030] Cloud computing is a delivery method that enables the on-demand provision of infrastructure, services, platforms, and applications across networks, rapidly replacing the traditional method of resource sharing via hardwired connections. Cloud-native is a software approach to building, deploying, and managing modern applications in a cloud computing environment. Modern enterprises want to build highly scalable, flexible, and resilient applications that can be rapidly updated to meet customer needs. To do this, they use modern tools and technologies that inherently support application development on cloud infrastructure. These cloud-native technologies enable rapid and frequent application changes without impacting service delivery, thus providing adopters with an innovative competitive advantage.

[0031] A GPU cluster is a computer cluster in which each GPU node is equipped with a GPU device. Leveraging the computing power of modern GPUs, GPU clusters can perform very fast computations.

[0032] In current high-performance computing (HPC) GPU clusters, network performance is a critical issue, especially in high-performance computing, deep learning, and big data analytics scenarios, impacting both performance and resource utilization. Data transmission latency: Distributed training and parallel processing require rapid data transmission from GPUs; network congestion significantly increases latency. Reduced throughput: Limited network bandwidth leads to packet loss and retransmission, reducing overall throughput. For workloads requiring large data exchange (e.g., deep learning model training), this results in longer training times. Decreased GPU utilization: Network congestion reduces data transmission rates and throughput, directly increasing idle time for peer GPUs, leading to decreased utilization and computational efficiency. Increased computational costs: A mismatch between network data transmission capacity and GPU computing power results in wasted computational resources and increased overall costs.

[0033] Currently, existing technologies for network performance optimization of GPU clusters include network-level optimizations.

[0034] Network-level optimizations include: building lossless networks and network decoupling.

[0035] Building lossless networks. InfiniBand (IB) networking technology is a computer network communication standard for high-performance computing, featuring extremely high throughput and extremely low latency for direct data interconnection between computers. RoCE (RDMA over Converged Ethernet) is a remote direct memory access protocol based on Ethernet, operating on Ethernet networks for efficient data transmission. Employing high-performance network technologies such as IB or RoCE provides higher bandwidth and lower latency, reduces network congestion, and enables lossless data transmission even when network congestion occurs.

[0036] Network separation. The network of a large-scale GPU cluster is divided into a front-end network and a back-end network. The front-end network is used for tasks such as checkpointing, logging, and data acquisition, while the back-end network focuses on computational tasks such as inference and training. Separating front-end and back-end traffic can reduce network congestion and improve data transmission efficiency.

[0037] Resource allocation in a GPU cluster includes device allocation. Device allocation refers to passing GPU devices and network interface devices (e.g., NIC devices) on GPU nodes directly to workloads (e.g., containers or virtual machines). Workloads can directly operate GPU devices and network interface devices, improving data transfer performance.

[0038] In existing technology, the device allocation process is as follows: 1. The user creates a workload, including: a resource request for the workload, indicating the number of GPU devices and network interface devices the workload needs to request. For example, the resource request for the workload is: aa.com / gpu:1, aa.com / netdevice:1, indicating that the workload needs to request 1 GPU device and 1 network interface device. 2. The workload's configuration information is applied to the GPU cluster. 3. Node scheduling. The scheduler selects GPU nodes in the cluster that meet the resource requests of the workload to create the workload. 4. Node proxy. After the workload is scheduled to a specific GPU node, the proxy service running on the GPU node creates the workload entity (e.g., a container or virtual machine) and calls the device plugin to allocate GPU devices and network interface devices to the workload.

[0039] Different devices have their own device plugins to manage device resource allocation. The GPU device management plugin is responsible for checking the available GPU devices within the GPU node and allocating them to workloads; the network interface device management plugin similarly allocates the available network interface devices within the GPU node to workloads.

[0040] PCI is a computer bus standard used to connect peripheral devices on a computer motherboard. It allows various devices (such as graphics cards, network interface devices, and storage controllers) to communicate with the motherboard through the same interface. The PCI bus uses a set of standard interfaces and protocols to achieve communication between devices, providing a flexible way to expand the functionality of a computer. PCI topology refers to the structural layout of all PCI devices (e.g., GPU devices, network interface devices, etc.) and their connections in a computer system.

[0041] PCI bus topology impacts data transmission performance; the topological distance between GPU devices and network interface devices on the PCI bus directly affects data transmission efficiency. In current technologies, resource allocation on GPU nodes within a GPU cluster is handled independently by GPU device management plugins and network interface device management plugins, failing to consider the topological distribution of GPU devices and network interface devices on the PCI bus. For example, if GPU devices and network interface devices are on different main bridges, or even communicate across Non-Uniform Memory Access (NUMA) nodes, the data link length increases, impacting transmission efficiency.

[0042] This disclosure provides a resource allocation system for GPU clusters. During the resource allocation process, it optimizes the data transmission path between GPU devices and network interface devices on GPU nodes, allocating the optimal device resources with the shortest topology distance to the workload. This effectively reduces data transmission latency and improves network data transmission performance. The resource allocation system for GPU clusters provided in this disclosure is described in detail below.

[0043] Figure 1 A block diagram of a resource allocation system applied to a GPU cluster according to an embodiment of the present disclosure is shown. Figure 1 As shown, the resource allocation system applied to the GPU cluster includes: a load scheduler, a topology scheduler, and a topology manager deployed within each GPU node in the GPU cluster.

[0044] Each GPU node has a deployed topology manager for determining the device topology of that GPU node, which indicates the topological relationships between the GPU devices and network interface devices included in the GPU node. A load scheduler is used to determine at least one candidate GPU node in the GPU cluster that satisfies the resource requests of the target load, where the resource requests of the target load indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices. The topology scheduler is used to determine the target GPU node based on the shortest topological distance principle, according to the device topology of each candidate GPU node, and to determine at least one target GPU-network interface device pair within the target GPU node that satisfies the resource requests of the target load.

[0045] A target GPU-network interface device pair includes a GPU device and a network interface device.

[0046] At least one target GPU-network interface device pair that satisfies the resource request of the target load includes: a first number of GPU devices and a second number of network interface devices.

[0047] The resource allocation system applied to GPU clusters according to embodiments of this disclosure includes a topology manager deployed within each GPU node that determines the device topology map of the GPU node to indicate the topological relationships between GPU devices and network interface devices included in the GPU node. First, a load scheduler determines at least one candidate GPU node in the GPU cluster that satisfies the resource request of the target load. Then, the topology scheduler can determine the target GPU node based on the shortest topological distance principle according to the device topology map of each candidate GPU node, and determine at least one target GPU-network interface device pair including a first number of target GPU devices and a second number of target network interface devices in the target GPU node. This enables the allocation of optimal device resources with the shortest topological distance to the target load, effectively reducing data transmission latency and improving network data transmission performance.

[0048] In one example, the network interface device can be configured as a Network Interface Card (NIC) device, or it can be configured as other types of network interface devices depending on the actual application scenario. This disclosure does not make any specific limitations in this regard.

[0049] Figure 2 A schematic diagram is shown of a resource allocation system applied to a GPU cluster according to an embodiment of the present disclosure. For example... Figure 2 As shown, after the GPU cluster administrator enables the function of optimizing device resource allocation based on PCI topology, a topology scheduler is created, and a topology manager is created within each GPU node in the GPU cluster.

[0050] In one possible implementation, a topology manager deployed within each GPU node is used to generate and report the topology configuration file of the GPU node to the topology scheduler after determining the device topology map of the GPU node; the topology scheduler is used to obtain the device topology map of each candidate GPU node by reading the topology configuration file of each candidate GPU node.

[0051] The topology manager can collect the topology relationships between PCI devices. Since both GPU devices and network interface devices are PCI devices, the topology manager within a GPU node can construct a device topology map of the GPU node by collecting the topology relationships between GPU devices and network interface devices within the GPU node.

[0052] Furthermore, since the topology manager operates at the GPU node level, while the topology scheduler operates at the GPU cluster level, in order for the topology scheduler to effectively obtain the device topology map of the GPU node, the topology manager within the GPU node, after generating the device topology map of the GPU node, generates the topology configuration file (e.g., Kubernetes configmap) of the GPU node based on the device topology map of the GPU node, and then reports the topology configuration file of the GPU node to the topology scheduler.

[0053] Each GPU node corresponds to a topology configuration file. By reading the topology configuration file of each GPU node, the topology scheduler can obtain the device topology map of that GPU node, and thus perceive the topological relationships between the GPU devices and network interface devices included in each GPU node.

[0054] like Figure 2 As shown, a user in the GPU cluster creates a target workload. The target workload includes resource requests indicating the number of GPU devices and the number of network interface devices required.

[0055] In one example, the target workload can be deep learning model training or any task that needs to be performed using a GPU cluster. This disclosure does not limit the specific form of the target workload or the specific values ​​of the first quantity and the second quantity.

[0056] like Figure 2 As shown, the resource allocation system applied to GPU clusters includes a load scheduler.

[0057] The load scheduler first determines the number of available GPU devices and available network interface devices in each GPU node of the GPU cluster. Then, based on the resource requests of the target load, it selects at least one candidate GPU node in the GPU cluster that meets the resource requests of the target load. The candidate GPU nodes include at least a first number of available GPU devices and at least a second number of network interface devices.

[0058] For example, a GPU cluster includes three GPU nodes: GPU node 1 to GPU node 3. First, the load scheduler determines that GPU node 1 includes two available GPU devices (GPU devices 101-102) and three available network interface devices (network interface devices 111-113), GPU node 2 includes one available GPU device (GPU device 201) and one available network interface device (network interface device 211), and GPU node 3 includes two available GPU devices (GPU devices 301-302) and two available network interface devices (network interface devices 311-312).

[0059] When the target load's resource request indicates that the target load needs to request 2 GPU devices and 2 network interface devices, the load scheduler selects GPU node 1 and GPU node 3 as candidate GPU nodes.

[0060] Furthermore, such as Figure 2 As shown, the topology scheduler can determine the device topology of candidate GPU node 1 and candidate GPU node 3 by reading their topology configuration files.

[0061] In one possible implementation, the topology scheduler is configured to: for any candidate GPU node, identify at least one GPU-network interface device pair in that candidate GPU node; and determine the score for each GPU-network interface device pair based on the device topology of the candidate GPU node and a preset scoring principle.

[0062] Take the aforementioned candidate GPU node 1 and candidate GPU node 3 as examples.

[0063] Candidate GPU node 1 includes 6 GPU-network interface device pairs: GPU device 101-network interface device 111, GPU device 101-network interface device 112, GPU device 102-network interface device 111, GPU device 102-network interface device 112, GPU device 101-network interface device 113, and GPU device 102-network interface device 113.

[0064] Candidate GPU node 3 includes four GPU-network interface device pairs: GPU device 301-network interface device 311, GPU device 301-network interface device 312, GPU device 302-network interface device 311, and GPU device 302-network interface device 312.

[0065] For any candidate GPU node, based on the device topology map of the candidate GPU node, the topological distance between the GPU device and the network interface device in each GPU-network interface device pair included in the candidate GPU node can be determined. Then, according to the preset scoring principle, the score of each GPU-network interface device pair in the candidate GPU node can be determined.

[0066] In one possible implementation, for any GPU-network interface device pair, the higher the score of the GPU-network interface device pair, the shorter the topological distance between the GPU device and the network interface device in the GPU-network interface device pair.

[0067] By comparing the scores of different GPU-network interface device pairs, the topological distance between the GPU device and the network interface device in different GPU-network interface device pairs can be effectively and quickly compared.

[0068] In one possible implementation, the preset scoring principles include: if the GPU device and network interface device in a GPU-network interface device pair pass through a switch, the GPU-network interface device pair is scored as a first score; if the GPU device and network interface device in a GPU-network interface device pair pass through multiple switches, the GPU-network interface device pair is scored as a second score, where the second score is less than the first score; if the GPU device and network interface device in a GPU-network interface device pair pass through a CPU, the GPU-network interface device pair is scored as a third score, where the third score is less than the second score; and if the GPU device and network interface device in a GPU-network interface device pair pass through a non-uniform memory access node, the GPU-network interface device pair is scored as a fourth score, where the fourth score is less than the third score.

[0069] The preset scoring principles decrease gradually in the following order:

[0070] In a GPU-network interface device pair, the GPU device and the network interface device are connected via a switch.

[0071] In a GPU-network interface device pair, the GPU device and the network interface device are connected via multiple switches;

[0072] In a GPU-Network Interface Device pair, the GPU device and the network interface device communicate via the CPU.

[0073] The GPU device and network interface device in the GPU-Network Interface Device pair are connected via NUM nodes.

[0074] The first score, the second score, the third score, and the fourth score can be a specific value or a range of values; this disclosure does not impose any specific limitations on them.

[0075] In addition to the rules mentioned above, the preset scoring principles can also include other rules that can be set according to the actual application scenario, as long as the shorter the topological distance, the higher the score. This disclosure does not limit the specific rules.

[0076] In one possible implementation, the topology scheduler is configured to: for any candidate GPU node, determine the score of the candidate GPU node based on the score of each GPU-network interface device pair in the candidate GPU node; determine the candidate GPU node with the highest score as the target GPU node; and determine the target GPU-network interface device pair in the target GPU node based on the score of each GPU-network interface device pair in the target GPU node.

[0077] For any candidate GPU node, the score of the candidate GPU node is determined based on the score of each GPU-network interface device pair in the candidate GPU node.

[0078] In one example, the score of a candidate GPU node is determined by summing the scores of all GPU-network interface device pairs in the candidate GPU node.

[0079] In one example, since the target load requires a first number of GPU devices and a second number of network interface devices, at least one GPU-network interface device pair that includes the first number of GPU devices and the second number of network interface devices and has the highest score can be selected from the candidate GPU nodes. Then, the scores of the selected GPU-network interface device pairs are summed to determine the score of the candidate GPU node.

[0080] The specific method for determining the score of a candidate GPU node based on the score of each GPU-network interface device pair in the candidate GPU node may include the methods mentioned above, or other determination methods may be set according to the actual application scenario. This disclosure does not make specific limitations on this.

[0081] After determining the score of each candidate GPU node, the candidate GPU node with the highest score is selected as the target GPU node.

[0082] Then, at least one target GPU-network interface device pair is selected in descending order of the GPU-network interface device pairs in the target GPU node. The selected target GPU-network interface device pairs include a first number of GPU devices and a second number of network interface devices.

[0083] Take the aforementioned candidate GPU node 1 and candidate GPU node 3 as examples.

[0084] The score of candidate GPU node 1 is determined based on the score of each GPU-network interface device pair in candidate GPU node 1; the score of candidate GPU node 3 is determined based on the score of each GPU-network interface device pair in candidate GPU node 3; if the score of candidate GPU node 1 is higher than the score of candidate GPU node 3, candidate GPU node 1 is determined as the target GPU node.

[0085] If the target workload requires the application of 2 GPU devices and 2 network interface devices, select the 2 target GPU-network interface device pairs with the highest scores in the order of high to low scores for each GPU-network interface device pair in target GPU node 1: GPU device 101-network interface device 111 and GPU device 102-network interface device 112. The selected 2 target GPU-network interface device pairs include 2 available GPU devices (GPU device 101 and GPU device 102) and 2 available network interface devices (network interface device 111 and network interface device 112).

[0086] In one possible implementation, the topology scheduler is used to: schedule the target load to the target GPU node; and generate a resource allocation tag corresponding to the target load, wherein the resource allocation tag is used to indicate the target GPU-network interface device pair.

[0087] like Figure 2 As shown, after the topology scheduler determines the target GPU node, it schedules the target load to the target GPU node.

[0088] Taking the aforementioned target GPU node 1 as an example, after the topology scheduler schedules the target load to target GPU node 1, it will also generate and add a resource allocation tag to the target load. The resource allocation tag is used to indicate two target GPU-network interface device pairs: GPU device 101-network interface device 111 and GPU device 102-network interface device 112.

[0089] In one possible implementation, the resource allocation system applied to the GPU cluster further includes: a node controller, a GPU device management plugin, and a network interface device management plugin deployed within each GPU node in the GPU cluster; a node controller deployed within the target GPU node, used to parse a resource allocation tag after detecting that the target load is scheduled to the target GPU node, and to send the resource allocation tag to the GPU device management plugin and the network interface device management plugin deployed within the target GPU node; a GPU device management plugin deployed within the target GPU node, used to allocate the GPU devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag; and a network interface device management plugin deployed within the target GPU node, used to allocate the network interface devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag.

[0090] Each GPU node has a node controller deployed within it. Therefore, after the node controller deployed within the target GPU node detects that the target load has been scheduled to the target GPU node, it parses the resource allocation tag of the target load.

[0091] Since different devices have their own device plugins to manage device resource allocation, each GPU node has a GPU device management plugin for managing GPU device allocation and a network interface device management plugin for managing network interface device allocation.

[0092] Figure 3 This diagram illustrates how a target GPU node allocates resources to a target load according to an embodiment of the present disclosure. Figure 3 As shown, the node controller sends the parsed resource allocation tag of the target load to the GPU device management plugin and the network interface device management plugin.

[0093] like Figure 2 and Figure 3 As shown, the GPU device management plugin parses the resource allocation tag and assigns the GPU devices included in the target GPU-network interface device indicated by the resource allocation tag to the target load; the network interface device management plugin parses the resource allocation tag and assigns the network interface devices included in the target GPU-network interface device indicated by the resource allocation tag to the target load.

[0094] Taking the aforementioned target GPU node 1 as an example, the device management plugin in target GPU node 1 assigns GPU device 101 and GPU device 102 from the two target GPU-network interface device pairs to the target load; the network interface device management plugin assigns network interface device 111 and network interface device 112 from the two target GPU-network interface device pairs to the target load.

[0095] like Figure 2 and Figure 3 As shown, the node controller in the target GPU node creates an entity (e.g., a container or virtual machine) for the target workload. This entity can directly manipulate the GPU devices and network interface devices assigned to the target workload.

[0096] The resource allocation system applied to GPU clusters according to embodiments of this disclosure includes a topology manager deployed within each GPU node that determines the device topology map of the GPU node to indicate the topological relationships between GPU devices and network interface devices included in the GPU node. First, a load scheduler determines at least one candidate GPU node in the GPU cluster that satisfies the resource request of the target load. Then, the topology scheduler can determine the target GPU node based on the shortest topological distance principle according to the device topology map of each candidate GPU node, and determine at least one target GPU-network interface device pair including a first number of target GPU devices and a second number of target network interface devices in the target GPU node. This enables the allocation of optimal device resources with the shortest topological distance to the target load, effectively reducing data transmission latency and improving network data transmission performance.

[0097] In existing technologies, the actual PCI topology is not considered when allocating device resources for workloads. The drawback of this is that the PCI topology on the GPU node cannot be effectively utilized to optimize data transmission efficiency, resulting in the inability to achieve optimal performance when running services such as real-time inference, online learning, and high-performance computing.

[0098] In this embodiment, the target GPU-network interface device pair with the shortest topological distance is allocated to the target load. This allows the GPU device and network interface device to exchange data over a shorter PCI bus in scenarios where the target load requires a large amount of data exchange, achieving higher bandwidth and transmission rates while avoiding bandwidth contention and bottlenecks. This, in turn, improves the execution efficiency of the target load's application. When the target load is a training task, it effectively reduces the number of training interruptions; when the target load is a real-time inference or online learning task, it shortens response time. In situations where there are numerous continuous data processing and computation tasks within the GPU cluster, optimizing the allocation of device resources between the GPU device and the network interface device increases the efficiency of the GPU cluster in processing requests, thereby effectively reducing latency and performance degradation caused by resource contention.

[0099] Furthermore, by sensing the hardware topology between GPU devices and network interface devices within a GPU node, hardware resources can be quickly reallocated in the event of a hardware failure, reducing service interruption time.

[0100] When allocating hardware devices to a target load, by sensing the hardware topology between GPU devices and network interface devices within a GPU node, data paths can be optimized, unnecessary jumps can be reduced, thereby reducing energy consumption and improving overall performance.

[0101] Figure 4 A flowchart illustrating a resource allocation method applied to a GPU cluster according to an embodiment of this disclosure is shown. Figure 4 As shown, the method includes:

[0102] In step S41, the device topology map of the GPU node is determined by using the topology manager deployed in each GPU node in the GPU cluster. The device topology map of the GPU node is used to indicate the topological relationship between the GPU devices and network interface devices included in the GPU node.

[0103] In step S42, based on the resource request of the target load, at least one candidate GPU node that meets the resource request is determined in the GPU cluster using the load scheduler deployed in the GPU cluster. The resource request is used to indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices.

[0104] In step S43, based on the device topology map of each candidate GPU node, the topology scheduler deployed in the GPU cluster is used to determine the target GPU node based on the principle of shortest topology distance, and at least one target GPU-network interface device pair that satisfies the resource request is determined in the target GPU node.

[0105] In one possible implementation, the method further includes:

[0106] After the topology manager deployed within each GPU node determines the device topology map of that GPU node, it generates and reports the topology configuration file of that GPU node to the topology scheduler based on the device topology map of that GPU node.

[0107] The topology scheduler is used to read the topology configuration file of each candidate GPU node to obtain the device topology map of that candidate GPU node.

[0108] In one possible implementation, the method further includes:

[0109] For any candidate GPU node, identify at least one GPU-network interface device pair within that candidate GPU node;

[0110] Based on the device topology of the candidate GPU node and the preset scoring principles, the score of each GPU-network interface device pair is determined.

[0111] In one possible implementation, for any GPU-network interface device pair, the higher the score of the GPU-network interface device pair, the shorter the topological distance between the GPU device and the network interface device in the GPU-network interface device pair.

[0112] In one possible implementation, the pre-defined scoring principles include:

[0113] When a switch passes between the GPU device and the network interface device in a GPU-network interface device pair, the GPU-network interface device pair is scored as the first score.

[0114] When the GPU device and network interface device in a GPU-network interface device pair pass through multiple switches, the GPU-network interface device pair is scored as a second score, where the second score is less than the first score.

[0115] When the GPU device and network interface device in a GPU-network interface device pair pass through the CPU, the GPU-network interface device pair is scored as a third score, where the third score is less than the second score.

[0116] When the GPU device and network interface device in a GPU-network interface device pair pass through a non-uniform memory access node, the GPU-network interface device pair is scored as the fourth score, which is lower than the third score.

[0117] In one possible implementation, based on the device topology map of each candidate GPU node, a topology scheduler deployed in the GPU cluster is used to determine the target GPU node based on the principle of shortest topological distance, and at least one target GPU-network interface device pair that satisfies the resource request is determined within the target GPU node, including:

[0118] For any candidate GPU node, the score of the candidate GPU node is determined based on the score of each GPU-network interface device pair in the candidate GPU node;

[0119] The candidate GPU node with the highest score is selected as the target GPU node;

[0120] The target GPU-network interface device pair is determined in the target GPU node based on the score of each GPU-network interface device pair in the target GPU node.

[0121] In one possible implementation, the method further includes:

[0122] Dispatch the target workload to the target GPU node;

[0123] Generate a resource allocation tag corresponding to the target workload, wherein the resource allocation tag is used to indicate the target GPU-network interface device pair.

[0124] In one possible implementation, the method further includes:

[0125] After detecting that the target load is scheduled to the target GPU node, the resource allocation tag is obtained by parsing the node controller deployed in the target GPU node, and the resource allocation tag is sent to the GPU device management plugin and network interface device management plugin deployed in the target GPU node.

[0126] Using the GPU device management plugin deployed within the target GPU node, the resource allocation tag is parsed, and the GPU devices included in the target GPU-network interface device indicated by the resource allocation tag are allocated to the target load;

[0127] By utilizing the network interface device management plugin deployed within the target GPU node, the resource allocation tag is parsed, and the network interface devices included in the target GPU-network interface devices indicated by the resource allocation tag are allocated to the target load.

[0128] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0129] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0130] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0131] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0132] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 5 Device 1900 can be provided as a server or terminal device. (See reference...) Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0133] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0135] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0136] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0137] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0138] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0142] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A resource allocation system for GPU clusters, characterized in that, include: Load scheduler, topology scheduler, and topology manager deployed within each GPU node in the GPU cluster; The topology manager deployed within each GPU node is used to determine the device topology map of that GPU node, where the device topology map of the GPU node is used to indicate the topological relationship between the GPU devices and network interface devices included in that GPU node; The load scheduler is configured to determine at least one candidate GPU node in the GPU cluster that satisfies the resource request of the target load, wherein the resource request is used to indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices. The topology scheduler is used to determine the target GPU node based on the device topology map of each candidate GPU node and the principle of shortest topology distance, and to determine at least one target GPU-network interface device pair that satisfies the resource request among the target GPU nodes.

2. The system according to claim 1, characterized in that, The topology manager deployed within each GPU node is used to generate and report the topology configuration file of the GPU node to the topology scheduler after determining the device topology map of the GPU node. The topology scheduler is used to obtain the device topology map of each candidate GPU node by reading the topology configuration file of each candidate GPU node.

3. The system according to claim 1, characterized in that, The topology scheduler is used for: For any candidate GPU node, identify at least one GPU-network interface device pair within that candidate GPU node; Based on the device topology of the candidate GPU node and the preset scoring principles, the score of each GPU-network interface device pair is determined.

4. The system according to claim 3, characterized in that, For any GPU-network interface device pair, the higher the score of the GPU-network interface device pair, the shorter the topological distance between the GPU device and the network interface device in the GPU-network interface device pair.

5. The system according to claim 3 or 4, characterized in that, The preset scoring principles include: When a switch passes between the GPU device and the network interface device in a GPU-network interface device pair, the GPU-network interface device pair is scored as the first score. When the GPU device and network interface device in a GPU-network interface device pair pass through multiple switches, the GPU-network interface device pair is scored as a second score, wherein the second score is less than the first score; When the GPU device and network interface device in a GPU-network interface device pair pass through the CPU, the GPU-network interface device pair is scored as a third score, wherein the third score is less than the second score; When the GPU device and network interface device in a GPU-network interface device pair pass through a non-uniform memory access node, the GPU-network interface device pair is scored as a fourth score, which is less than the third score.

6. The system according to claim 4, characterized in that, The topology scheduler is used for: For any candidate GPU node, the score of the candidate GPU node is determined based on the score of each GPU-network interface device pair in the candidate GPU node; The candidate GPU node with the highest score is determined as the target GPU node; The target GPU-network interface device pair is determined in the target GPU node based on the score of each GPU-network interface device pair in the target GPU node.

7. The system according to claim 1, characterized in that, The topology scheduler is used for: The target load is scheduled to the target GPU node; Generate a resource allocation tag corresponding to the target load, wherein the resource allocation tag is used to indicate the target GPU-network interface device pair.

8. The system according to claim 7, characterized in that, The system also includes: a node controller, a GPU device management plugin, and a network interface device management plugin deployed within each GPU node in the GPU cluster; The node controller deployed within the target GPU node is used to parse the resource allocation tag after detecting that the target load has been scheduled to the target GPU node, and to send the resource allocation tag to the GPU device management plugin and the network interface device management plugin deployed within the target GPU node. The GPU device management plugin deployed within the target GPU node is used to allocate the GPU devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag. The network interface device management plugin deployed within the target GPU node is used to allocate the network interface devices included in the target GPU-network interface devices indicated by the resource allocation tag to the target load by parsing the resource allocation tag.

9. A resource allocation method applied to GPU clusters, characterized in that, include: The device topology map of each GPU node is determined by using the topology manager deployed within each GPU node in the GPU cluster. The device topology map of the GPU node is used to indicate the topological relationship between the GPU devices and network interface devices included in the GPU node. Based on the resource request of the target load, at least one candidate GPU node in the GPU cluster is determined using the load scheduler deployed in the GPU cluster to satisfy the resource request, wherein the resource request is used to indicate that the target load needs to request a first number of GPU devices and a second number of network interface devices. Based on the device topology map of each candidate GPU node, the topology scheduler deployed in the GPU cluster determines the target GPU node based on the principle of shortest topology distance, and determines at least one target GPU-network interface device pair that satisfies the resource request in the target GPU node.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 9.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 9.

12. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 9.

Images