Resource management method and computing device

Abstract

Disclosed in the present application are a resource management method and a computing device. The method is applied to a management node of a server cluster, wherein the server cluster is used for processing a distributed task, the distributed task comprises a plurality of sub-tasks, and a plurality of target computing units in the server cluster are used for separately training the sub-tasks. The method comprises: during the processing for a distributed task, determining whether a heterogeneous computing resource overflow event occurs when each target computing unit trains a sub-task; and if the heterogeneous computing resource overflow event occurs in a first target computing unit among a plurality of target computing units, performing a scaling-up operation on resources for training the distributed task. In this way, when any target computing unit trains a sub-task, once this event occurs and resources required for training the sub-task are insufficient to support the completion of the training, a scaling-up operation is performed on resources for training a distributed task, thereby ensuring the sufficiency of the resources for the continuation of distributed training. Therefore, the training efficiency of the distributed training is improved.

Description

Resource management methods and computing devices

[0001] This application claims priority to Chinese Patent Application No. 202410790793.5, filed on June 18, 2024, entitled “Resource Management Method and Computing Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computing device technology, and in particular to a resource management method and a computing device. Background Technology

[0003] Currently, when processing distributed tasks such as distributed training of artificial intelligence (AI) models using server clusters, the values ​​of the AI ​​model parameters have a crucial impact on the model training performance. For example, batch size refers to the amount of training data used in each iteration of training the AI ​​model. A large batch size can improve training efficiency, but it may cause GPU memory overflow due to the amount of training data exceeding the GPU's processing capacity, leading to the interruption of distributed task training. In related technologies, when distributed task training is interrupted, parameters are usually adjusted manually based on experience before training resumes, resulting in low training efficiency for distributed tasks. Summary of the Invention

[0004] This application provides a resource management method and computing device that can improve the training efficiency of distributed training and save training time.

[0005] Firstly, a resource management method is provided, applied to the management node of a server cluster. The server cluster is used to process distributed tasks, each including multiple subtasks. Multiple target computing units within the server cluster are used to train the subtasks respectively. This method includes: during the processing of the distributed task, determining whether a heterogeneous computing resource overflow event occurs when each target computing unit trains a subtask. The heterogeneous computing resource overflow event indicates that the resources required for training the subtask in the target computing unit exceed the actual resources allocated to the subtask. If a heterogeneous computing resource overflow event occurs in the first target computing unit among the multiple target computing units, then an expansion operation is performed on the resources used to train the distributed task. The expansion operation increases the number of target computing units used to train the distributed task.

[0006] The method provided in this application, during the processing of distributed tasks, indicates that if any target computing unit experiences a heterogeneous computing resource overflow event while training a subtask, it means that the resources required for training the subtask have exceeded the actual resources allocated to the subtask. If the resources required for training the subtask are insufficient to support training completion, an expansion operation is performed on the resources of the distributed training task to ensure sufficient resources for the continuation of distributed training. This improves the training efficiency of distributed tasks and saves training time.

[0007] In one possible implementation, determining the occurrence of heterogeneous computing resource overflow events during the training of subtasks for each target computing unit includes: obtaining their respective heterogeneous computing resource overflow events from each target computing unit.

[0008] Each target computing unit corresponds to training each subtask. The management node can obtain information from each target computing unit to determine whether they have experienced heterogeneous computing resource overflow events, thereby identifying which target computing units face resource overflow issues when training subtasks, so that subsequent resource expansion operations can be performed on demand for training distributed tasks.

[0009] In another possible implementation, a scaling operation is performed on the resources of the training distributed task, including: obtaining the resource utilization rate of each target computing unit when training subtasks; and determining the number of target computing units to be added based on the resource utilization rate.

[0010] In another possible implementation, the number of target computing units to be added is determined based on resource utilization, including: determining the number of target computing units to be added based on the range of resource utilization and a preset coefficient.

[0011] By flexibly allocating resources based on resource requirements, responding promptly to resource overflow events and adding target computing units, and timely performing resource expansion, sufficient resources can be ensured to facilitate the continuation of distributed training. This can reduce the failure of training tasks due to insufficient resources, improve the training efficiency of distributed tasks, and save training time.

[0012] In another possible implementation, the target computing unit is a server, and the first target computing unit is a first server; if a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, then an expansion operation is performed on the resources for training the distributed task, including: if a heterogeneous computing resource overflow event occurs in the first server among multiple servers, then the number of servers used for training the distributed task is increased.

[0013] or;

[0014] The target computing unit is a container on the server, and the first target computing unit is the first container. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, then an expansion operation is performed on the resources for training the distributed task, including: if a heterogeneous computing resource overflow event occurs in the first container among multiple containers, then the number of containers used for training the distributed task is increased.

[0015] Both servers and containers can perform resource scaling based on the methods provided in the embodiments of this application. By increasing the number of servers or containers, the resources used for training distributed tasks are increased, which helps to prevent training overload and make fuller use of resources.

[0016] In another possible implementation, before determining whether a heterogeneous computing resource overflow event occurs when each target computing unit trains a subtask, the method further includes: obtaining the configuration information of the distributed task, which includes the number of target computing units and configuration options for expansion operations; if a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, then an expansion operation is performed on the resources of the training distributed task, including: collecting the resource utilization rate of each target computing unit when training a subtask within a preset collection period; if no expansion operation request is obtained within the preset collection period, and a heterogeneous computing resource overflow event is obtained in the first target computing unit, then an expansion operation is performed on the resources of the training distributed task based on the resource utilization rate of the first target computing unit when training a subtask.

[0017] In this way, even if no scaling request is received within the preset collection period, but a heterogeneous computing resource overflow event occurs when a training subtask is acquired, a scaling operation can be performed on the resources of the distributed training task in a timely manner. This ensures that sufficient resources are used to execute the distributed task training and prevents training interruption. Thus, the training efficiency of the distributed task is improved, and training time is saved.

[0018] In another possible implementation, resource utilization includes GPU utilization or NPU utilization.

[0019] Secondly, a resource management method is provided, which is applied to computing nodes in a server cluster. The server cluster is used to process distributed tasks, each including multiple subtasks. Multiple target computing units in the server cluster are used to train the subtasks respectively. Target computing units are deployed on the computing nodes, or the computing nodes themselves are target computing units. The method includes: during the training of subtasks, detecting whether a heterogeneous computing resource overflow event occurs. A heterogeneous computing resource overflow event indicates that the resources required by the target computing unit to train the subtask exceed the actual resources allocated to the subtask; when a heterogeneous computing resource overflow event occurs, reporting the event to the management node of the server cluster.

[0020] During the training subtasks, each computing node can detect whether a heterogeneous computing resource overflow event occurs, so as to promptly notify the management node when such an event occurs. This resource monitoring and reporting mechanism enables the management node to respond quickly to resource overflow events and take appropriate measures.

[0021] In one possible implementation, detecting whether a heterogeneous computing resource overflow event has occurred includes periodically checking the training logs of the training subtasks for records of heterogeneous computing resource overflow events.

[0022] In another possible implementation, detecting whether a heterogeneous computing resource overflow event has occurred includes: determining whether a heterogeneous computing resource overflow event has occurred based on the heterogeneous computing resource overflow event perceived by the resource overflow perception component.

[0023] By periodically checking training logs or using resource overflow awareness components, computing nodes can understand resource usage during training subtasks and promptly detect heterogeneous computing resource overflow events.

[0024] Thirdly, a resource management device is provided, which is applied to the management node of a server cluster. The server cluster is used to process distributed tasks, each of which includes multiple subtasks. Multiple target computing units in the server cluster are used to train the subtasks respectively. The device includes an acquisition module and a scaling module.

[0025] The acquisition module is used to determine whether a heterogeneous computing resource overflow event occurs when training subtasks in each target computing unit during the processing of distributed tasks. A heterogeneous computing resource overflow event indicates that the resources required for training subtasks in a target computing unit exceed the actual resources allocated to the subtasks. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, the expansion module performs an expansion operation on the resources used for training the distributed task. The expansion operation increases the number of target computing units used for training the distributed task.

[0026] Fourthly, a resource management device is provided, which is applied to computing nodes in a server cluster. The server cluster is used to process distributed tasks, each of which includes multiple subtasks. Multiple target computing units in the server cluster are used to train the subtasks respectively. Target computing units are deployed on the computing nodes, or the computing nodes themselves are target computing units. The device includes a detection module and a communication module.

[0027] The detection module detects heterogeneous computing resource overflow events during the training subtask process. These events indicate that the resources required by the training subtask in the target computing unit exceed the actual resources allocated to the subtask. The communication module reports the heterogeneous computing resource overflow event to the management node of the server cluster when it occurs.

[0028] Fifthly, a computing device is provided, comprising: a memory and a processor, the memory storing a set of computer execution instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, it executes the resource management method in the first aspect or any possible implementation of the first aspect. Alternatively, when the processor executes the set of computer instructions as an execution device in the second aspect or any possible implementation of the second aspect, it executes the resource management method in the second aspect or any possible implementation of the second aspect.

[0029] Sixthly, a chip is provided, comprising: an interface circuit and a processor; the interface circuit is configured to receive code instructions and transmit them to the processor; the processor is configured to execute the code instructions to perform the resource management method of the first aspect or any possible implementation thereof. Alternatively, the processor is configured to execute code instructions to perform the resource management method of the second aspect or any possible implementation thereof.

[0030] A seventh aspect provides a computer-readable storage medium storing computer-executable instructions that, when executed on a computing device, cause the computing device to perform the resource management method of the first aspect or any possible implementation thereof. Alternatively, cause the computing device to perform the resource management method of the second aspect or any possible implementation thereof.

[0031] Eighthly, a computer program product is provided, comprising computer-executable instructions that, when executed on a computing device, cause the computing device to perform the resource management method of the first aspect or any possible implementation thereof. Alternatively, the computing device is caused to perform the resource management method of the second aspect or any possible implementation thereof.

[0032] The technical effects of any of the implementation methods in aspects three through eight can be found in the technical effects of different implementation methods in aspects one or two, and will not be repeated here.

[0033] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0034] Figure 1 is a schematic diagram of the architecture of a resource management system provided in an embodiment of this application;

[0035] Figure 2 is a flowchart illustrating a resource management method provided in an embodiment of this application;

[0036] Figure 3 is a schematic diagram of a periodic comparison provided in an embodiment of this application;

[0037] Figure 4 is a schematic diagram of a capacity expansion operation based on the K8S platform provided in an embodiment of this application;

[0038] Figure 5 is a schematic diagram of the structure of a resource management device provided in an embodiment of this application;

[0039] Figure 6 is a schematic diagram of another resource management device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0041] In the description of this application, unless otherwise stated, "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0042] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0043] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0044] To ensure clarity and conciseness in the description of the following embodiments, the terminology used in this application will be briefly introduced first.

[0045] A server cluster is a group of loosely or tightly connected servers that work together, typically used to execute large jobs. Clustered servers are generally more computationally efficient than a single server with comparable speed or availability. By using multiple servers in parallel and utilizing various computing resources simultaneously to solve problems, the computational and processing speeds of the cluster system can be improved, thus enhancing the overall performance of the cluster system. The servers in a cluster are interconnected via a network, and each server runs its own operating system.

[0046] Containers allow different applications to run in relatively isolated and secure environments, achieving isolation between applications and between applications and the external environment. Containers are a lightweight virtualization technology with advantages such as fast startup, easy deployment and migration, good security, and scalability, and are widely used in fields such as AI and cloud computing.

[0047] Kubernetes (K8S) is an open-source container management platform. By using the K8S platform, the tedious manual orchestration of containers can be eliminated, making container configuration, deployment, and scaling simple and efficient.

[0048] A distributed task refers to a large computational task that is broken down into multiple subtasks. These subtasks are distributed across various computing units (which can be servers or containers deployed on servers) within a server cluster. Each computing unit executes its assigned subtask independently, and the units can exchange data and collaborate over a network. A typical example of a distributed task is distributed training of AI models. The training dataset can be divided into multiple batches, and each computing unit trains each batch of the dataset in parallel, thereby improving training efficiency.

[0049] To address the low training efficiency of distributed training, this application provides a resource management method. This method is applied to the management node of a server cluster, which processes distributed tasks, each including multiple subtasks. Multiple target computing units within the server cluster are used to train the subtasks respectively. The method includes: during the processing of the distributed task, determining whether a heterogeneous computing resource overflow event occurs when each target computing unit trains a subtask. The heterogeneous computing resource overflow event indicates that the resources required for training the subtask in the target computing unit exceed the actual resources allocated to the subtask. If a heterogeneous computing resource overflow event occurs in the first target computing unit among the multiple target computing units, then an expansion operation is performed on the resources used to train the distributed task. The expansion operation increases the number of target computing units used to train the distributed task.

[0050] Compared to manually adjusting parameters based on experience after training is interrupted and then resuming training, which results in lower training efficiency for distributed tasks, the method provided in this application addresses this issue. During distributed task processing, if any target computing unit experiences a heterogeneous computing resource overflow event while training a subtask, it indicates that the resources required for the training subtask have exceeded the actual resources allocated to it. Insufficient resources for the training subtask prevent training from completing, and a resource expansion operation is performed on the distributed training resources to ensure sufficient resources for the continuation of distributed training. This improves the training efficiency of distributed tasks and saves training time.

[0051] The method provided in this application can be applied to scenarios where distributed tasks are processed based on multiple computing nodes in a server cluster, or to scenarios where distributed tasks are processed based on containers deployed on multiple computing nodes in a server cluster. Containers can use the virtualization resources provided by the computing unit to perform task processing.

[0052] The following uses the example of containers deployed on multiple computing nodes to process distributed tasks, and describes in detail the implementation of the embodiments of this application with reference to the accompanying drawings.

[0053] Figure 1 is a schematic diagram of the architecture of a resource management system provided in an embodiment of this application. As shown in Figure 1, the resource management system 100 includes a management node 110 and a computing node cluster 120 in a server cluster.

[0054] A container management platform (such as a Kubernetes platform) can run on management node 110. Based on the Kubernetes platform, containers deployed on each compute node in the compute node cluster 120 can be managed. In the Kubernetes platform, containers are usually managed in units of pods. A pod can encapsulate a container, and a container can also be called a pod.

[0055] The Kubernetes (K8S) platform provides functionality for creating, deploying, scaling, orchestrating, and scheduling containers. To coordinate these functions, the K8S platform can include multiple components, such as the horizontal pod autoscaler (HPA) component, the deployment component, the metrics server component, the Kubelet proxy component, the custom metrics server component, and the Prometheus monitoring component. The functions of each component are briefly described below.

[0056] The HPA component is used to periodically obtain resource usage metrics for each container and increase or decrease the number of container replicas based on these metrics to dynamically adjust resources according to the actual load of the containers and prevent resource waste.

[0057] The deployment component is used to manage the number of container replicas based on the replica set controller (RSC). If a container crashes due to an error, the RSC can automatically create a replacement container.

[0058] The resource metrics server component is used to collect resource usage metrics for each container (such as container CPU utilization and container memory utilization) and send the resource usage metrics to the HPA component through the corresponding interface (metrics server API).

[0059] The Kubelet agent component is a core functional component provided by the Kubernetes (K8S) platform. Acting as an agent for resources on compute nodes, it is responsible for maintaining and managing tasks such as container creation, startup, and shutdown on each compute node in the compute cluster. It also maintains real-time communication with K8S to ensure the normal operation of containers. The Kubelet agent component can manage the resources used by containers based on control group (cgroup) processes. cgroups can control the upper limits of compute, storage, and network resources that containers can use, allowing multiple containers to share resources. Through the execution of cgroup processes, the Kubelet agent component can obtain resource usage metrics for containers and report them to the resource metrics server component.

[0060] The custom resource metrics server component is used to collect resource usage metrics customized by users according to their usage needs. Custom resource usage metrics can be heterogeneous computing resource metrics, such as the NPU utilization rate and GPU utilization rate of containers, and send the custom resource usage metrics to the HPA component through the corresponding interface (custom metrics server API).

[0061] The Prometheus monitoring component is used to monitor the resource usage of containers deployed on each compute node in the compute node cluster of 120. The component has a built-in time-series database for storing the collected monitoring data.

[0062] The compute node cluster 120 includes multiple compute nodes, such as compute node 1, compute node 2, ..., and compute node N. Each compute node can deploy at least one pod, such as pod 1 on compute node 1, pod 2 on compute node 2, and pod N on compute node N. Because the pods deployed on the compute node cluster are managed using a Kubernetes (K8S) platform running on management node 110, the compute node cluster 120 can also be referred to as a K8S cluster.

[0063] When using containers deployed across multiple compute nodes in a Kubernetes cluster to process distributed AI model training tasks, the training dataset is typically large, requiring multiple iterations to train the AI ​​model. Therefore, the compute nodes need strong computing power to run neural networks and perform computations such as training them. The neural network in this application's embodiments can be a deep neural network or a convolutional neural network; the type of neural network is not limited.

[0064] Different types of AI models implement different functions, and computing nodes can be different types of servers. Examples include central processing unit (CPU) servers, graphics processing unit (GPU) servers, and neural network processing unit (NPU) servers. CPU servers are more versatile and suitable for deep learning fields such as pattern recognition, object detection, and image processing. GPU servers can be used specifically for image-related computing tasks, accelerating graphics rendering. NPU servers can be used to accelerate the training and inference computations of artificial neural networks, enabling efficient large-scale neural network computations.

[0065] Figure 1 uses an NPU server as an example of a compute node. An NPU server can include multiple NPU accelerator cards, such as NPU1 to NPU8. Each NPU card contains a storage chip used to cache training data during AI model training. The compute node can allocate the computing and storage resources provided by the NPU cards to containers, allowing containers to use all or part of the NPU resources to perform AI model training tasks according to their training needs.

[0066] To improve training efficiency and save training time, the method provided in this application embodiment, based on the K8S platform running on the management node 110, can obtain in real time whether a heterogeneous computing resource overflow event occurs during the training of each container subtask. If such an event occurs, an expansion operation is performed on the NPU resources for training the distributed task to ensure that sufficient NPU resources can be used to train the AI ​​model. The specific implementation of the expansion operation can be found in the following method steps.

[0067] As shown in the expansion example in Figure 1, before expansion, AI models can be trained in a distributed manner based on pod 1 deployed on compute node 1 and pod 2 deployed on compute node 2. When management node 110 receives a heterogeneous computing resource overflow event, it executes an expansion operation, designating compute node N as the expansion node and deploying pod N on compute node N, allocating the computing resources provided by the NPU card to pod N. After expansion, AI models can be trained in a distributed manner based on pod 1 deployed on compute node 1, pod 2 deployed on compute node 2, and pod N deployed on compute node N.

[0068] The components shown in Figure 1 above do not constitute a limitation on the computing node. A computing node may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, a computing node may also include memory, a baseboard management controller (BMC), an AI card, a redundant array of independent disks (RAID), and a CPU and GPU, etc.

[0069] Furthermore, the system architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0070] Next, with reference to Figures 2 to 4, the resource management method provided in this application will be described by way of example. Figure 2 is a schematic flowchart of a resource management method provided in an embodiment of this application. The method is applied to the management node of a server cluster. As shown in Figure 2, the resource management method may include the following steps.

[0071] In some embodiments, prior to S201, the management node can obtain configuration information for the distributed task. This configuration information includes the number of target computing units and configuration options for scaling operations. For example, through the resource management interface provided by the management node, a distributed task training job is distributed to the server cluster. The number of target computing units for executing the distributed task is selected in the interface. If the resources required for the training subtasks of each target computing unit are greater than or equal to the actual resources allocated to the subtasks, then an automatic scaling operation is performed.

[0072] S201. In the process of processing distributed tasks, determine whether a heterogeneous computing resource overflow event occurs when each target computing unit is training a subtask.

[0073] Server clusters can be used to process distributed tasks, such as distributed training of AI models. Distributed tasks include multiple subtasks. Computing nodes in a server cluster that have computing power and can independently process subtasks can be called target computing units, or target computing units can be deployed on computing nodes. Multiple target computing units in a server cluster are used to train subtasks separately, meaning that any target computing unit can be used to train any subtask.

[0074] Resources used for training subtasks in the target computing unit can include heterogeneous computing resources, such as NPU or GPU resources; they also include CPU, memory, and network resources. During the distributed task processing in the server cluster, the management node can determine whether a heterogeneous computing resource overflow event occurs when each target computing unit is training a subtask. This event indicates that the resources required for training a subtask in the target computing unit exceed the actual resources allocated to the subtask. If a heterogeneous computing resource overflow event occurs, the resources required for training a subtask in the target computing unit are insufficient to support training completion.

[0075] In some embodiments, during the training of subtasks, each computing node can detect whether a heterogeneous computing resource overflow event has occurred. When a heterogeneous computing resource overflow event occurs, it reports the event to the management node. In this way, the management node can obtain information from each target computing unit regarding whether it has experienced a heterogeneous computing resource overflow event, thereby identifying which target computing units are facing resource overflow problems. This allows the management node to subsequently perform resource scaling operations on demand for training distributed tasks, ensuring resource sufficiency.

[0076] During the training subtasks, each computing node can detect whether a heterogeneous computing resource overflow event occurs, so as to promptly notify the management node when such an event occurs. This resource monitoring and reporting mechanism enables the management node to respond quickly to resource overflow events and take appropriate measures.

[0077] In one possible implementation, the compute node can periodically check the training logs of the training subtasks for records of heterogeneous computing resource overflow events. The AI ​​model's training log serves as a file recording various key information during model training. For example, during the training of a subtask, if the resources required by the subtask exceed the actual resources allocated to it, the training log will record the keyword "out of resource (OOR)," which the compute node can read. Thus, the compute node detects the heterogeneous computing resource overflow event from the training logs and reports it to the management node.

[0078] In another possible implementation, a resource overflow awareness component can run on the compute nodes. This component monitors resource usage during training subtasks in real time, sets the actual resources allocated to the subtasks as a resource usage threshold, and detects a heterogeneous computing resource overflow event when the resources required for a training subtask exceed this threshold. Based on the heterogeneous computing resource overflow event detected by the awareness component, the compute node can determine that a heterogeneous computing resource overflow event has occurred and report it to the management node.

[0079] The aforementioned resource overflow detection component can send event notifications to the training log, reporting heterogeneous computing resource overflow events that occurred during the training subtasks of the compute nodes. This allows the compute nodes to detect records of heterogeneous computing resource overflow events from the training logs, ensuring the timeliness of event information. The acquisition module in the compute node can also intercept event notifications to improve the monitoring and response capabilities to resource overflow events. By periodically checking the training logs or using the resource overflow detection component, compute nodes can better understand resource usage during training subtasks and promptly detect heterogeneous computing resource overflow events.

[0080] S202. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, then an expansion operation is performed on the resources of the training distributed task.

[0081] The first target computing unit can be any one of multiple target computing units. If a heterogeneous computing resource overflow event occurs in the first target computing unit, the management node needs to handle the resource overflow event in a timely manner. It can perform an expansion operation on the resources used to train the distributed task. The expansion operation can be to increase the number of target computing units used to train the distributed task, thereby expanding the resources in a timely manner to ensure that sufficient resources are used to train the distributed task.

[0082] As distributed training iterates, resources are continuously consumed during the training process. For example, the amount of resources used by each target computing unit to train its subtasks will increase, gradually reaching or even exceeding the resource allocation for the subtasks. Furthermore, when performing a scaling operation, the management node can obtain the resource utilization rate of each target computing unit during its training subtasks and determine the number of target computing units to be added based on this rate.

[0083] In one possible implementation, the management node can collect the resource usage of each target computing unit during its training subtask within a preset collection period. The resource utilization rate is determined based on the ratio of resource usage to allocated resources, thus obtaining the resource utilization rate for each target computing unit's training subtask. To prevent training interruptions, if a scaling request for any target computing unit is received within the preset collection period, a scaling operation is periodically executed. For example, the expected resource utilization rate can be preset based on the user's actual resource usage needs. When the resource utilization rate of a target computing unit during its training subtask exceeds the expected resource utilization rate, the management node can then receive a scaling operation request.

[0084] However, the collection cycle for the above-mentioned resource usage is relatively long, and some AI models are lightweight models, such as the LLaMa 2 large language model, where the training time for one iteration is about 30 seconds. Within this training time, the complete collection cycle has not yet been completed. If a heterogeneous computing resource overflow event has occurred, but expansion operations are not performed in time, it will still lead to training interruption.

[0085] Figure 3 is a schematic diagram of a period comparison provided in an embodiment of this application. As shown in Figure 3, the iteration period of training the AI ​​model is different from the resource usage collection period. The iteration period of training the AI ​​model (between t1 and t2) is shorter than the resource usage collection period T1. If any target computing unit has experienced a heterogeneous computing resource overflow event during the time period t1 to t2, but no expansion operation request has been obtained during this time period, the management node cannot trigger the expansion operation in a timely manner.

[0086] In one implementation provided by this application, when no expansion operation request is obtained within a preset collection period, and a heterogeneous computing resource overflow event is obtained for the first target computing unit, the management node performs an expansion operation on the resources of the training distributed task based on the resource utilization rate of the first target computing unit when training subtasks, that is, determines the number of target computing units to be added.

[0087] In this way, even if no scaling request is received within the preset collection period, but a heterogeneous computing resource overflow event occurs when a training subtask is acquired, the resources of the distributed training task can be scaled up in a timely manner once the heterogeneous computing resource overflow event occurs, thereby ensuring that sufficient resources are used to execute the distributed task training. This improves the training efficiency of the distributed task and saves training time.

[0088] In one possible implementation, if a heterogeneous computing resource overflow event occurs in the first target computing unit, the resources required for the training subtask have exceeded the actual resources allocated to the subtask. The management node can determine the resource utilization rate based on the proportion by which the resource usage of the training subtask exceeds the resource allocation. Based on the range of the resource utilization rate and a preset coefficient, the number of target computing units to be added is determined.

[0089] This application does not limit the range of values ​​or the value of the preset coefficient. For example, a resource utilization rate below 30% is divided into a first interval, with a preset coefficient of 1, and the number of target computing units to be added is 1. A resource utilization rate between 30% and 80% is divided into a second interval, with a preset coefficient of 2, and the number of target computing units to be added is 2. A resource utilization rate above 80% is divided into a third interval, with a preset coefficient of 3, and the number of target computing units to be added is 3.

[0090] By flexibly allocating resources based on resource requirements, responding promptly to resource overflow events and adding target computing units, and timely performing resource expansion, sufficient resources can be ensured to facilitate the continuation of distributed training. This can reduce the failure of training tasks due to insufficient resources, improve the training efficiency of distributed tasks, and save training time.

[0091] In some embodiments, the target computing unit can be a server in a server cluster, and the first target computing unit is a first server. If the first server among multiple servers experiences a heterogeneous computing resource overflow event, the scaling operation is to increase the number of servers used for training distributed tasks.

[0092] In other embodiments, the method provided in this application can be applied to scenarios where distributed task training is performed based on containers. The target computing unit can be a container deployed on a server, and the first target computing unit is a first container. If a heterogeneous computing resource overflow event occurs in the first container among multiple containers, the scaling operation is to increase the number of containers used for training distributed tasks.

[0093] For example, the Kubernetes platform running on the management node can manage the resources used by each container training subtask. The management node can obtain the resource utilization rate of each container training subtask from the Kubernetes platform's resource metrics server (such as metrics server or custom metrics server) to determine the number of containers to be added based on the resource utilization rate.

[0094] When the management node deploys containers based on the number of containers to be added, it can be done using the deployment component in the Kubernetes platform. The replica set controller in the deployment component updates the number of replicas it stores according to the number of containers to be added. When the number of replicas changes, the deployment component can deploy containers on the compute nodes of the server cluster based on the number of containers to be added.

[0095] In one possible implementation, the first compute node has already deployed the first container. When the virtualization resources provided by the first compute node are sufficient to deploy the number of containers to be increased, the management node uses the first compute node as a scaling node and deploys the number of containers to be increased on the first compute node, thereby realizing the scaling operation.

[0096] In another possible implementation, the management node selects a second compute node in the server cluster, uses the second compute node as a scaling node, and deploys the containers on the second compute node to increase the number of containers, thereby achieving the scaling operation.

[0097] After the expansion operation is completed, distributed training tasks are performed based on the containers before and after the expansion operation. This ensures that the resources used for distributed training are sufficient, helps prevent training interruptions, improves the training efficiency of distributed tasks, and saves training time.

[0098] To facilitate understanding of the resource management method described in the above steps, an exemplary description is provided below with reference to the accompanying drawings. Figure 4 is a schematic diagram of a capacity expansion operation based on the K8S platform provided in an embodiment of this application. As shown in Figure 4, a resource overflow event monitoring component and a resource overflow event handling component are added to the original components in the K8S platform. Through the cooperation between the various components, the capacity expansion operation is realized.

[0099] The management node runs the K8S platform shown in Figure 4. The K8S platform can manage the tasks of the pods deployed on the compute nodes in the server cluster, such as sending AI model training tasks to the pods deployed on the compute nodes and selecting the number of compute nodes in the server cluster used for distributed training of AI models. Starting the HPA component in the K8S platform can support automatic scaling during AI model training.

[0100] The HPA component can obtain the resource utilization of heterogeneous computing resources used during training subtasks of pods deployed on compute nodes. For example, it can obtain the NPU or GPU utilization of each container within each pod from the custom resource metrics server component. Specifically, during container training subtasks, the GPU / NPU metric detector can detect the GPU / NPU utilization, first reporting it to the Promethues monitoring component, which then reports it to the custom resource metrics server component. Furthermore, the HPA component can also obtain the computing resource utilization (e.g., CPU utilization) and storage resource utilization (e.g., memory utilization) of each container from the resource metrics server component. Specifically, the Kubelet proxy component can detect the container's running status to obtain these utilization rates and report them to the resource metrics server component.

[0101] In the process of distributed training of AI models based on containers deployed on multiple compute nodes, each compute node can detect whether heterogeneous computing resource overflow events occur during container training subtasks. For example, each compute node can periodically check whether there are records of heterogeneous computing resource overflow events in the training logs of the training subtasks. If a record of a heterogeneous computing resource overflow event is detected, the notification of the resource overflow event is reported to the management node. The resource overflow event monitoring component in the K8S platform receives the event notification and sends the event notification to the resource overflow event handling component in HPA.

[0102] After receiving a resource overflow event notification, the resource overflow event handling component in HPA processes the event, increases the resources used by the container training distributed task, determines the number of containers to be added based on the resource utilization range and preset coefficients, and sends the number of containers to be added to the deployment component.

[0103] After receiving the number of containers to be added, the replica set controller within the deployment component updates the stored number of replicas according to the number of containers to be added. When the number of replicas changes, the deployment component can deploy that number of containers on the compute nodes based on the number of containers to be added, thus achieving scaling. As shown in Figure 4, taking the second compute node in the compute cluster as the scaling node as an example, the containers to be added are deployed on the second compute node, and these containers are encapsulated by pod N. In this way, the AI ​​model is trained based on pod 1 and pod 2 deployed on the first compute node and pod N deployed on the second compute node.

[0104] The resource management method described above, compared to manually adjusting parameters based on experience after training interruption and then resuming training, results in lower training efficiency for distributed tasks. The method provided in this application, during the processing of distributed tasks, indicates that if any target computing unit experiences a heterogeneous computing resource overflow event while training a subtask, it means that the resources required for the training subtask have exceeded the actual resources allocated to it. Insufficient resources for the training subtask prevent training from completing, so an expansion operation is performed on the resources of the distributed training task to ensure sufficient resources for the continuation of distributed training. This improves the training efficiency of distributed tasks and saves training time.

[0105] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the resource management device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] This application embodiment can, according to the above method, exemplarily divide the resource management device into functional modules. For example, the resource management device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0107] For example, Figure 5 is a schematic diagram of a resource management device provided in an embodiment of this application. The resource management device 500 is applied to the management node of a server cluster. The server cluster is used to process distributed tasks, and the distributed tasks include multiple sub-tasks. Multiple target computing units in the server cluster are used to train the sub-tasks respectively. The resource management device 500 includes an acquisition module 510 and an expansion module 520.

[0108] The acquisition module 510 is used to determine whether a heterogeneous computing resource overflow event occurs when each target computing unit trains a subtask during the processing of distributed tasks. A heterogeneous computing resource overflow event indicates that the resources required for training a subtask in a target computing unit exceed the actual resources allocated to the subtask. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, the expansion module 520 performs an expansion operation on the resources used to train the distributed task. The expansion operation increases the number of target computing units used for training the distributed task.

[0109] Optionally, when the acquisition module 510 determines that a heterogeneous computing resource overflow event occurs during the training subtask of each target computing unit, it is specifically used to: acquire the heterogeneous computing resource overflow event of each target computing unit.

[0110] Optionally, when the expansion module 520 performs an expansion operation on the resources of the training distributed task, it is specifically used to: obtain the resource utilization rate of each target computing unit during the training subtask; and determine the number of target computing units to be added based on the resource utilization rate.

[0111] Optionally, when the expansion module 520 determines the number of target computing units to be added based on the resource utilization rate, it is specifically used to: determine the number of target computing units to be added based on the range of resource utilization rate and a preset coefficient.

[0112] Optionally, the target computing unit is a server, and the first target computing unit is a first server; if a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, the expansion module 520 performs an expansion operation on the resources for training distributed tasks, specifically for: if a heterogeneous computing resource overflow event occurs in the first server among multiple servers, then the number of servers used for training distributed tasks is increased.

[0113] or;

[0114] The target computing unit is a container on the server, and the first target computing unit is the first container. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, the expansion module 520 performs an expansion operation on the resources for training distributed tasks. Specifically, if a heterogeneous computing resource overflow event occurs in the first container among multiple containers, the number of containers used for training distributed tasks is increased.

[0115] Optionally, before the acquisition module 510 determines whether a heterogeneous computing resource overflow event occurs during the training subtask of each target computing unit, it is also used to acquire the configuration information of the distributed task. The configuration information includes the number of target computing units and the configuration options for the expansion operation. If a heterogeneous computing resource overflow event occurs in the first target computing unit among multiple target computing units, the expansion module 520 performs an expansion operation on the resources of the training distributed task, specifically by: acquiring the resource utilization rate of each target computing unit during the training subtask within a preset acquisition period; if no expansion operation request is acquired within the preset acquisition period, and a heterogeneous computing resource overflow event is acquired in the first target computing unit, then an expansion operation is performed on the resources of the training distributed task according to the resource utilization rate of the first target computing unit during the training subtask.

[0116] Optionally, the resource utilization mentioned above includes GPU utilization or NPU utilization.

[0117] Figure 6 is a schematic diagram of a resource management device provided in an embodiment of this application. The resource management device 600 is applied to a computing node in a server cluster. The server cluster is used to process distributed tasks, which include multiple sub-tasks. Multiple target computing units in the server cluster are used to train the sub-tasks respectively. Target computing units are deployed on the computing nodes, or the computing nodes are target computing units. The device includes a detection module 610 and a communication module 620.

[0118] The detection module 610 is used to detect whether a heterogeneous computing resource overflow event occurs during the training subtask. A heterogeneous computing resource overflow event indicates that the resources required by the training subtask in the target computing unit exceed the actual resources allocated to the subtask. The communication module 620 is used to report the heterogeneous computing resource overflow event to the management node of the server cluster when it occurs.

[0119] Optionally, when the detection module 610 detects whether a heterogeneous computing resource overflow event has occurred, it is specifically used to periodically check whether there are records of heterogeneous computing resource overflow events in the training logs of the training subtasks.

[0120] Optionally, when the detection module 610 detects whether a heterogeneous computing resource overflow event has occurred, it is specifically used to: determine that a heterogeneous computing resource overflow event has occurred based on the heterogeneous computing resource overflow event perceived by the resource overflow perception component.

[0121] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanations and descriptions of the beneficial effects of any of the resource management devices 500 and 600 provided above can be found in the corresponding method embodiments above, and will not be repeated here.

[0122] This application also provides a computing device, which includes a memory and a processor. The memory stores computer execution instructions, and the processor executes the computer execution instructions to implement the resource management method in the above embodiments.

[0123] This application does not impose any restrictions on the specific form of the computing device. For example, the computing device can be a single physical or logical computing device, or it can consist of two or more physical or logical computing devices that perform different responsibilities and work together to achieve the various functions of the computing device.

[0124] This application also provides a computer-readable storage medium storing a computer program that, when run on a computing device, causes the computing device to perform any of the methods described above.

[0125] For explanations of the relevant content and descriptions of the beneficial effects in any of the computer-readable storage media provided above, please refer to the corresponding embodiments described above, which will not be repeated here.

[0126] This application also provides a chip. This chip integrates a control circuit for implementing the functions of the aforementioned computing device and one or more ports. Optionally, the functions supported by this chip can be referred to above, and will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, random access memory, etc. The aforementioned processing unit or processor can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0127] This application also provides a computer program product containing instructions that, when executed on a computing device, cause the computing device to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computing device, all or part of the flow or function according to the embodiments of this application is generated.

[0128] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computing device, all or part of the flow or function according to the embodiments of this application is generated. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device including one or more computing devices, data centers, etc., that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0130] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0131] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A resource management method, characterized in that, The method is applied to the management node of a server cluster, the server cluster being used to process distributed tasks, the distributed tasks comprising multiple subtasks, and multiple target computing units in the server cluster being used to train the subtasks respectively; the method includes: During the processing of the distributed task, it is determined whether a heterogeneous computing resource overflow event occurs when each target computing unit trains the subtask. The heterogeneous computing resource overflow event is used to indicate that the resources required by the target computing unit to train the subtask exceed the actual resources allocated to the subtask. If the heterogeneous computing resource overflow event occurs in the first target computing unit among the plurality of target computing units, then an expansion operation is performed on the resources used to train the distributed task, wherein the expansion operation is to increase the number of target computing units used to train the distributed task.

2. The method according to claim 1, characterized in that, The determination of a heterogeneous computing resource overflow event occurring when each of the target computing units trains the subtask includes: Obtain each of the target computing units their respective heterogeneous computing resource overflow events.

3. The method according to claim 1 or 2, characterized in that, The scaling operation on the resources used to train the distributed task includes: Obtain the resource utilization rate of each target computing unit when training the subtask; The number of target computing units to be added is determined based on the resource utilization rate.

4. The method according to claim 3, characterized in that, Determining the number of target computing units to be added based on the resource utilization rate includes: The number of target computing units to be added is determined based on the range of resource utilization rates and preset coefficients.

5. The method according to any one of claims 1-4, characterized in that, The target computing unit is a server, and the first target computing unit is a first server; If the heterogeneous computing resource overflow event occurs in the first target computing unit among the plurality of target computing units, then an expansion operation is performed on the resources used to train the distributed task, including: If the heterogeneous computing resource overflow event occurs on the first server among the plurality of servers, the number of servers used to train the distributed task is increased. or; The target computing unit is a container on a server, and the first target computing unit is a first container; If the heterogeneous computing resource overflow event occurs in the first target computing unit among the plurality of target computing units, then an expansion operation is performed on the resources used to train the distributed task, including: If the heterogeneous computing resource overflow event occurs in the first container among the plurality of containers, the number of containers used for training the distributed task is increased.

6. The method according to any one of claims 1-5, characterized in that, Before determining whether a heterogeneous computing resource overflow event occurs when each of the target computing units trains the subtask, the method further includes: Obtain the configuration information of the distributed task, the configuration information including the number of the target computing units and the configuration options for the scaling operation; If the heterogeneous computing resource overflow event occurs in the first target computing unit among the plurality of target computing units, then an expansion operation is performed on the resources used to train the distributed task, including: Within a preset collection period, the resource usage of each target computing unit during the training of the subtask is collected; If no expansion operation request is obtained within the preset collection period, and the heterogeneous computing resource overflow event occurs in the first target computing unit, then an expansion operation is performed on the resources used to train the distributed task based on the resource utilization rate of the first target computing unit when training the subtask.

7. The method according to any one of claims 4-6, characterized in that, The resource utilization rate includes the utilization rate of the graphics processing unit (GPU) or the neural network processing unit (NPU).

8. A resource management method, characterized in that, The method is applied to computing nodes in a server cluster, the server cluster being used to process distributed tasks, the distributed tasks including multiple subtasks, and multiple target computing units in the server cluster being used to train the subtasks respectively; The target computing unit is deployed on the computing node, or the computing node is the target computing unit; the method includes: During the training of the subtask, it is detected whether a heterogeneous computing resource overflow event occurs. The heterogeneous computing resource overflow event is used to indicate that the resources required for training the subtask in the target computing unit exceed the actual resources allocated to the subtask. When the heterogeneous computing resource overflow event occurs, the heterogeneous computing resource overflow event is reported to the management node of the server cluster.

9. The method according to claim 8, characterized in that, The detection of whether a heterogeneous computing resource overflow event has occurred includes: Periodically check the training logs of the subtasks for records of heterogeneous computing resource overflow events.

10. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store a set of computer instructions, which, when executed by the processor, perform the method of any one of claims 1-7 or any one of claims 8-9.

Images