Distributed AI training task processing method and system

Abstract

The invention discloses a distributed AI training task processing method and system, and relates to the technical field of cloud computing. The distributed AI training task processing method comprises the steps that labels are set for hardware resources of nodes of each cluster of each cloud platform, and resource information of each hardware resource is acquired; and intercepting a CRD configuration file containing a hardware resource quantity request through a pre-deployed webhook service in the cluster, and creating an AI training task carrying the hardware resource quantity request after modifying resource information in the CRD configuration file into a parameter format adaptive to the hardware environment of the cluster according to the tag. According to the method and the device, hardware resources in multi-cloud platform heterogeneous resources can be scheduled and used in a unified manner without manual intervention, dynamic parameter adaptation and automatic injection of hardware resource configuration can be completed through the webhook service deployed in the cluster, and seamless resource scheduling and adaptation across cloud platforms and across hardware architectures are realized.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and specifically to a method and system for processing distributed AI training tasks. Background Art

[0002] With breakthroughs in artificial intelligence (AI) technology, the number of large-scale pre-trained models has exploded, with model parameter counts exceeding trillions (e.g., GPT-4), and the scale of training data is also expanding exponentially. Currently, single-machine training is limited by computing resources and storage bottlenecks, making the distributed training paradigm a key technical approach to supporting the implementation of large AI models. By parallelizing training tasks across heterogeneous compute nodes (e.g., GPU / TPU clusters), combined with data parallelism, model parallelism, and pipeline parallelism, training throughput can be significantly improved. However, the frequent gradient synchronization and parameter update operations (e.g., AllReduce communication) in distributed training lead to inter-node communication efficiency, a core bottleneck limiting system performance.

[0003] In related technologies, mainstream domestic cloud service providers provide infrastructure support for distributed training by offering heterogeneous computing resources (such as NVIDIA A100 / H100 GPU clusters) and high-speed network facilities (such as InfiniBand RDMA). To avoid the failure of AI distributed training due to the failure of a single cloud platform, using cloud computing resources from multiple cloud service providers is a simple and convenient coping strategy.

[0004] However, when deploying distributed training tasks in a multi-cloud hybrid deployment model, the hardware configuration differences between different cloud platforms (such as GPU architecture and RDMA network card model) are incompatible with the Kubernetes (K8S) cluster management strategy, resulting in the need to manually adapt resource request parameters (such as CUDA version and RDMA driver) for cross-cloud tasks, which increases the complexity of operation and maintenance and reduces resource scheduling efficiency. Summary of the Invention

[0005] This application provides a method and system for processing distributed AI training tasks, which can solve the technical problems of complex operation and maintenance and low resource scheduling efficiency in the existing technology.

[0006] In a first aspect, the present application provides a method for processing a distributed AI training task, the method comprising: Set labels for the hardware resources of each node in each cluster of each cloud platform and obtain resource information for each hardware resource; The webhook service pre-deployed in the cluster intercepts the CRD configuration file containing the hardware resource quantity request. After modifying the resource information in the CRD configuration file to the parameter format adapted to the cluster hardware environment based on the above tags, an AI training task carrying the hardware resource quantity request is created.

[0007] In conjunction with the first aspect, in one embodiment, before intercepting the CRD configuration file containing the hardware resource quantity request, the method further includes: Receive AI training task requests, determine the target node and target cluster to be scheduled, add the scheduling results to the CRD configuration file, and then issue the creation request for the above AI training task to the above target cluster; When the AI ​​training task request carries a hardware resource quantity request, the scheduling result includes the hardware resource quantity request, the label corresponding to the target node, and resource information.

[0008] In conjunction with the first aspect, in one embodiment, when the AI ​​training task request does not carry a hardware resource quantity request, the scheduling result includes a label and resource information corresponding to the target node; After sending the AI ​​training task creation request to the target cluster, the target cluster creates the AI ​​training task.

[0009] In combination with the first aspect, in one implementation, the hardware resources are GPU resources, or GPU resources and RDMA resources.

[0010] In conjunction with the first aspect, in one embodiment, obtaining resource information of each hardware resource specifically includes: When the above hardware resources are GPU resources, use Prometheus to monitor and obtain the GPU utilization and task queue status of each node in real time; When the above hardware resources are GPU resources and RDMA resources, Prometheus is used to monitor and obtain the GPU utilization, RDMA bandwidth and task queue status of each node in real time.

[0011] In conjunction with the first aspect, in one embodiment, determining the target node to be scheduled and the target cluster to which it belongs specifically includes: According to the resource information of the hardware resources of each node, the target node to be scheduled and the target cluster to which it belongs are determined through the scheduling strategy.

[0012] In conjunction with the first aspect, in one embodiment, before the cluster executes the AI ​​training task carrying the hardware resource quantity request, the further steps include: When the above-mentioned hardware resources are GPU resources and RDMA resources, the above-mentioned webhook service is used to detect the RDMA link status in the cluster in real time. When the link is abnormal, the backup communication protocol and gradient quantization compression configuration are automatically injected into the above-mentioned AI training task.

[0013] In conjunction with the first aspect, in one embodiment, after modifying the resource information in the CRD configuration file into a parameter format adapted to the cluster hardware environment, the method further includes: Inject the Sidecar container into the AI ​​training task through the webhook service. When executing the AI ​​training task, the Sidecar container monitors the hardware resources in real time. When the Sidecar container detects hardware resource anomalies or link failures, it dynamically adjusts the training parameters or communication protocols of the main container.

[0014] In conjunction with the first aspect, in one implementation, deploying a webhook service specifically includes: Develop webhook service program to build container image; Deploy the preceding container image to the cluster so that the webhook service can run in the cluster.

[0015] In a second aspect, the present application provides a distributed AI training task processing system, the system comprising: A setting module is used to set labels for the hardware resources of the nodes of each cluster of each cloud platform and obtain resource information of each hardware resource; The deployment execution module is used to intercept the CRD configuration file containing the hardware resource quantity request through the webhook service pre-deployed in the cluster, modify the resource information in the above CRD configuration file into a parameter format adapted to the cluster hardware environment based on the above tags, and then create an AI training task carrying the hardware resource quantity request.

[0016] The beneficial effects of the technical solution provided by this application include: Since labels are set for the hardware resources of the nodes of each cluster of each cloud platform, and the resource information of each hardware resource is obtained, the CRD configuration file containing the request for the number of hardware resources is intercepted through the webhook service pre-deployed in the cluster, and the resource information in the above CRD configuration file is modified to the parameter format adapted to the cluster hardware environment based on the above labels, and an AI training task carrying the request for the number of hardware resources is created. Therefore, not only can the hardware resources in the heterogeneous resources of multi-cloud platforms be uniformly scheduled and used without human intervention, but the dynamic parameter adaptation and automatic injection of hardware resource configuration can be completed through the webhook service deployed in the cluster, realizing seamless resource scheduling and adaptation across cloud platforms and hardware architectures, solving the technical problems of complex operation and maintenance and low resource scheduling efficiency in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1This is a flowchart of an embodiment of a method for processing distributed AI training tasks in this application; Figure 2 This is a flowchart of another embodiment of the method for processing distributed AI training tasks of this application; Figure 3 This is a schematic diagram of the architecture of an embodiment of a system for processing distributed AI training tasks in this application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0020] In a first aspect, an embodiment of the present application provides a method for processing distributed AI training tasks.

[0021] In one embodiment, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the method for processing distributed AI training tasks of this application. The above-mentioned method for processing distributed AI training tasks includes: S1. Label the hardware resources of each node in each cluster on each cloud platform and obtain resource information for each hardware resource. S2. Use the pre-deployed webhook service in the cluster to intercept the CRD (Custom Resource Definition) configuration file containing the hardware resource quantity request. Based on the tags, the resource information in the CRD configuration file is modified to a parameter format suitable for the cluster's hardware environment. Then, an AI training task is created that carries the hardware resource quantity request.

[0022] In this embodiment, labels are set for the hardware resources of the nodes of each cluster of each cloud platform, and the resource information of each hardware resource is obtained; then, through the webhook service pre-deployed in the cluster, the CRD configuration file containing the request for the number of hardware resources is intercepted, and the resource information in the above CRD configuration file is modified to the parameter format adapted to the hardware environment of the cluster according to the above label, and then an AI training task carrying the request for the number of hardware resources is created. Therefore, not only can the hardware resources in the heterogeneous resources of the multi-cloud platform be uniformly scheduled and used without manual intervention, but also the dynamic parameter adaptation and automatic injection of the hardware resource configuration can be completed through the webhook service deployed in the cluster, realizing seamless resource scheduling and adaptation across cloud platforms and hardware architectures, solving the technical problems of complex operation and maintenance and low resource scheduling efficiency in related technologies.

[0023] Furthermore, in one embodiment, in the above step S2, before intercepting the CRD configuration file containing the hardware resource quantity request, the following is further included: Receive AI training task requests, determine the target node and target cluster to be scheduled, add the scheduling results to the CRD configuration file, and then issue the creation request for the above AI training task to the above target cluster.

[0024] When the AI ​​training task request carries a hardware resource quantity request, the scheduling result includes the hardware resource quantity request, the label corresponding to the target node, and resource information.

[0025] In this embodiment, by adding a hardware resource quantity request field to the CRD configuration file of the above-mentioned creation request, the webhook service pre-deployed in the cluster automatically intercepts the CRD configuration file and implements dynamic parameter adaptation, thereby solving the problems of poor resource scheduling flexibility and inability to adapt to multi-cloud environments in related technologies.

[0026] Based on the above embodiment, in this embodiment, when the above AI training task request does not carry a request for the quantity of hardware resources, the above scheduling result includes the label and resource information corresponding to the target node.

[0027] At this point, after sending the AI ​​training task creation request to the target cluster, the target cluster can execute the AI ​​training task. In other words, the pre-deployed webhook service in the cluster has no effect on the AI ​​training task, and the task can be scheduled to the cluster node through the adaptive scheduling algorithm.

[0028] Furthermore, in this embodiment, the hardware resources are GPU (Graphics Processing Unit) resources, or GPU resources and RDMA (Remote Direct Memory Access) resources.

[0029] Preferably, the above hardware resources are GPU resources and RDMA resources.

[0030] In other embodiments, the aforementioned hardware resources are GPU resources.

[0031] In some embodiments, in the training of large AI models, the coordinated utilization of GPU and RDMA resources directly determines the task throughput. Due to the expansion of model parameters (such as GPT-4 reaching 1.8 trillion) and the surge in the scale of training data (PB level), distributed training requires frequent exchange of gradients and parameters between multiple nodes (single-task AllReduce operations exceed 10,000 times per second). Through RDMA technology (end-to-end zero copy, bandwidth utilization >90%), the multi-layer replication overhead of the traditional TCP / IP protocol stack can be circumvented, reducing communication latency from milliseconds to microseconds. However, from the perspective of platform scheduling, GPU and RDMA resources in multi-cloud environments cannot be efficiently coordinated, resulting in high resource costs and an inability to support more user tasks. In addition, static scheduling strategies are prone to resource fragmentation problems because they cannot perceive dynamic indicators such as GPU memory occupancy and RDMA bandwidth utilization in real time, thereby reducing the overall throughput of the cluster.

[0032] In this embodiment, a Kubernetes Mutating Admission Webhook-based service program is developed to target the heterogeneous hardware resources of different cloud platform Kubernetes clusters in a multi-cloud environment. When the CRD for an AI training task is submitted to the Kubernetes API server, it automatically intercepts and parses the task description, identifying the GPU and RDMA resource request fields. The deployed Webhook service then dynamically converts the generic GPU and RDMA parameters in the user task request into the specific parameter format required by the target cluster based on the actual hardware tags and resource status of the cluster nodes. For example, different cloud service providers may have different GPU models, RDMA network card types, and driver versions. The Webhook automatically injects or modifies fields such as environment variables, resource requests, and mounted volumes based on node tags and real-time monitoring information, ensuring that the task seamlessly adapts to the hardware environment of the target cluster. This enables adaptive cloud platform Kubernetes cluster RDMA and GPU resource scheduling, optimizes the collaborative utilization efficiency of GPU and RDMA resources for distributed AI training tasks, and solves the problem of efficient collaborative utilization of GPU and RDMA resources for distributed AI training tasks in a multi-cloud environment.

[0033] In this embodiment, in the above step S1, obtaining resource information of each hardware resource specifically includes: When the above hardware resources are GPU resources, use Prometheus to monitor and obtain the GPU utilization and task queue status of each node in real time; When the above hardware resources are GPU resources and RDMA resources, Prometheus is used to monitor and obtain the GPU utilization, RDMA bandwidth and task queue status of each node in real time as the resource information of the hardware resources.

[0034] Furthermore, the above-mentioned determination of the target node to be scheduled and the target cluster to which it belongs specifically includes: According to the resource information of the hardware resources of each node, the target node to be scheduled and the target cluster to which it belongs are determined through the scheduling strategy.

[0035] Optionally, when the hardware resources are GPU resources, the target node to be scheduled and the target cluster to which it belongs are determined by a multi-strategy fusion adaptive scheduling strategy according to the GPU utilization rate of each node and the task queue status.

[0036] In another embodiment, when the hardware resources are GPU resources, the target node to be scheduled and the target cluster to which it belongs are determined through a multi-strategy fusion adaptive scheduling strategy based on the GPU utilization, RDMA bandwidth and task queue status of each node.

[0037] In this embodiment, the above-mentioned scheduling strategy can be a binpack strategy, a priority label strategy for distributed AI training tasks, etc., to determine the cluster and node where the distributed AI training task is scheduled, and add the information to the corresponding YAML related fields.

[0038] Furthermore, in one embodiment, before the cluster executes the AI ​​training task carrying the hardware resource quantity request, the cluster further includes: When the above-mentioned hardware resources are GPU resources and RDMA resources, the above-mentioned webhook service is used to detect the link status of RDMA in the cluster in real time. When the link is abnormal, the backup communication protocol and gradient quantization compression configuration are automatically injected into the above-mentioned AI training task.

[0039] In this embodiment, the Webhook service supports real-time monitoring of intra-cluster RDMA link status (such as RTT and packet loss rate). When an RDMA link anomaly is detected (such as RTT > 500μs or link interruption), the Webhook automatically injects configuration parameters such as an alternative communication protocol (such as TCP / IP) and gradient quantization compression into the task. This enables automatic switching of communication modes without interrupting training tasks, improving system robustness and continuity.

[0040] Furthermore, in one embodiment, after modifying the resource information in the CRD configuration file into a parameter format adapted to the cluster hardware environment, the following steps are further included: Inject the Sidecar container into the AI ​​training task through the webhook service. When executing the AI ​​training task, the Sidecar container monitors the hardware resources in real time. When the Sidecar container detects hardware resource anomalies or link failures, it dynamically adjusts the training parameters or communication protocols of the main container.

[0041] In this embodiment, the Webhook service automatically injects the Sidecar container into the AI ​​training task. The Sidecar container is responsible for real-time monitoring of the GPU and RDMA resource utilization, link status, training progress, etc. of the main container, and supports log collection, configuration hot update and other functions. When a resource anomaly or link failure is detected, the Sidecar can dynamically adjust the training parameters or communication protocol of the main container through shared volumes, environment variables or APIs to achieve adaptive adjustment of the task and uninterrupted operation. During the entire monitoring process, the sidecar remains decoupled from the main container, supports independent upgrades and hot restarts, and improves the maintainability and elasticity of the system.

[0042] It is understandable that for newly created tasks, the backup communication protocols (such as TCP / IP) and configuration parameters such as gradient quantization compression automatically injected by the Webhook service for the task will be directly written into the task's Pod definition; for tasks that are already running, it is necessary to rely on the Sidecar container to implement runtime configuration hot updates and protocol switching to ensure that the AI ​​training task completes the communication mode switch without interruption. The switching of the communication protocol is bidirectional. When RDMA is restored, it can automatically switch back to the high-performance RDMA communication protocol, thereby further improving the communication efficiency and system elasticity of the AI ​​training task.

[0043] Furthermore, in one embodiment, deploying a webhook service specifically includes: First, develop a service program based on Kubernetes Mutating Admission Webhook to build the container image.

[0044] Then, deploy the above container image to the cluster so that the above webhook service can run in the cluster.

[0045] Therefore, the processing method of this embodiment has the following technical effects: (1) This embodiment addresses the differences in heterogeneous resources such as GPUs and RDMA in a multi-cloud environment and proposes a unified label, dynamic parameter adaptation, and automatic injection mechanism, achieving seamless resource scheduling and adaptation across cloud platforms and hardware architectures, greatly reducing manual intervention and operation and maintenance complexity. (2) The Webhook service of this embodiment has real-time monitoring and intelligent fault tolerance capabilities. It can automatically switch communication protocols (such as RDMA and TCP / IP) according to the RDMA link status and dynamically inject configurations such as gradient quantization compression to ensure the continuity and high availability of distributed training tasks.

[0046] (3) This embodiment innovatively links the Sidecar container with the Webhook to achieve real-time monitoring, log collection, and hot configuration updates of GPU and RDMA resources, support dynamic adjustments during task execution, and further improve resource utilization and system elasticity.

[0047] Specifically, if Figure 2 As shown, the processing method of the above distributed AI training task specifically includes: A1. Unified access and identification of multi-cloud platform resources: A unified tagging system is used to identify GPU and RDMA hardware resources across multiple cloud platforms, ensuring accurate identification and scheduling across platforms. Furthermore, monitoring tools such as Prometheus are used to enable real-time monitoring of resource information such as GPU utilization, RDMA bandwidth, and task queue status on each node, providing data support for subsequent scheduling. A2. Standardized user task requests: When users submit distributed AI training tasks through the frontend, the tasks include requests for GPU and RDMA resource quantities. Task descriptions use a unified format to facilitate automatic parsing and processing by the backend system. A3. Backend Intelligent Scheduling Decisions: After receiving an AI training task request, the backend system automatically identifies the task's requested GPU and RDMA resource quantities, obtains real-time GPU and RDMA resource information for Kubernetes cluster nodes on different cloud platforms, and uses a multi-strategy adaptive scheduling algorithm to determine the cluster and dispatch the training task. This multi-strategy adaptive scheduling algorithm (such as binpack and priority tags) intelligently selects the optimal cluster and node based on current resource utilization and task queue status. The scheduling results are dynamically written into the task's YAML configuration, enabling automated resource allocation and efficient utilization, significantly reducing manual intervention. A4. Dynamic Adaptation and Injection of Webhook Services: All cloud platform Kubernetes clusters deploy a Webhook service. This service automatically intercepts task CRDs submitted to the Kubernetes API server and dynamically rewrites and injects GPU and RDMA-related parameters, environment variables, and mounted volumes based on the target cluster's hardware tags and real-time status, ensuring seamless task adaptation to heterogeneous hardware environments. Tasks can then be submitted to the Kubernetes cluster and begin processing. Webhook also supports features such as automatic sidecar injection, communication protocol switching, and hot policy updates, further enhancing system flexibility and robustness.

[0048] Among them, the Webhook service is built in the form of a container image and is automatically deployed and elastically scaled in various cloud platform clusters through K8S Deployment. It supports high-availability deployment and grayscale upgrades, facilitating subsequent maintenance and functional expansion.

[0049] The processing method implemented in this paper dynamically perceives the status of RDMA network and GPU resources, combines adaptive scheduling strategies with Webhook configuration injection technology, and realizes the automated, intelligent, seamless adaptation and efficient scheduling of GPU and RDMA resources for distributed AI training tasks in multi-cloud K8S clusters, greatly reducing manual operation and maintenance costs and improving resource utilization and system robustness.

[0050] Secondly, an embodiment of the present application also provides a processing system for distributed AI training tasks.

[0051] In one embodiment, referring to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of a distributed AI training task processing system of the present application. The above-mentioned distributed AI training task processing system includes a setting module and a deployment execution module.

[0052] The above setting module is set in the back-end system, and is used to set labels for the hardware resources of the nodes of each cluster of each cloud platform and obtain resource information of each hardware resource.

[0053] The above-mentioned deployment execution module is set in the cluster and is used to intercept the CRD configuration file containing the request for the number of hardware resources through the webhook service pre-deployed in the cluster, and modify the resource information in the above-mentioned CRD configuration file into a parameter format adapted to the cluster hardware environment according to the above-mentioned label, and then create an AI training task carrying the request for the number of hardware resources.

[0054] Furthermore, in one embodiment, the processing system further includes a scheduling module provided in the backend system, the scheduling module being configured to receive AI training task requests submitted by users, determine the target node to be scheduled and the target cluster to which it belongs, and, after adding the scheduling result to the task's CRD configuration file, issue a request to create the AI ​​training task to the target cluster. When the AI ​​training task request carries a hardware resource quantity request, the scheduling result includes the hardware resource quantity request, the label corresponding to the target node, and resource information.

[0055] Furthermore, in one embodiment, when the AI ​​training task request does not carry a request for the quantity of hardware resources, the scheduling result includes a label and resource information corresponding to the target node.

[0056] After the scheduling module sends the creation request of the AI ​​training task to the target cluster, the deployment execution module is also used to create the AI ​​training task.

[0057] Furthermore, in one embodiment, the above-mentioned hardware resources are GPU resources, or GPU resources and RDMA resources.

[0058] Furthermore, in one embodiment, the setting module is used to: When the above hardware resources are GPU resources, use Prometheus to monitor and obtain the GPU utilization and task queue status of each node in real time; When the above hardware resources are GPU resources and RDMA resources, Prometheus is used to monitor and obtain the GPU utilization, RDMA bandwidth and task queue status of each node in real time.

[0059] Furthermore, in one embodiment, the scheduling module is further configured to determine the target node to be scheduled and the target cluster to which it belongs based on the resource information of the hardware resources of each node and through a scheduling policy.

[0060] Furthermore, in one embodiment, the deployment execution module is further configured to: When the above-mentioned hardware resources are GPU resources and RDMA resources, the above-mentioned webhook service is used to detect the link status of RDMA in the cluster in real time. When the link is abnormal, the backup communication protocol and gradient quantization compression configuration are automatically injected into the above-mentioned AI training task.

[0061] Furthermore, in one embodiment, the deployment execution module is further configured to: The sidecar container is injected into the AI ​​training task through the webhook service. When the AI ​​training task is executed, the sidecar container monitors the hardware resources in real time.

[0062] When the Sidecar container detects hardware resource anomalies or link failures, it dynamically adjusts the training parameters or communication protocols of the main container.

[0063] Furthermore, in one embodiment, the deployment execution module is further configured to: Develop webhook service program to build container image; Deploy the preceding container image to the cluster so that the webhook service can run in the cluster.

[0064] Among them, the functional implementation of each module in the above-mentioned distributed AI training task processing system corresponds to the various steps in the above-mentioned distributed AI training task processing method embodiment, and its functions and implementation process will not be repeated here one by one.

[0065] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0066] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0067] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0068] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.

[0070] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing distributed AI training tasks, characterized in that: The method comprises: Set labels for the hardware resources of each node in each cluster of each cloud platform and obtain resource information for each hardware resource; Through the webhook service pre-deployed in the cluster, the CRD configuration file containing the hardware resource quantity request is intercepted, and the resource information in the CRD configuration file is modified to the parameter format adapted to the cluster hardware environment according to the label, and then an AI training task carrying the hardware resource quantity request is created.

2. The method for processing a distributed AI training task according to claim 1, wherein: Before intercepting the CRD configuration file containing the hardware resource quantity request, it also includes: Receive AI training task requests, determine the target node and target cluster to be scheduled, add the scheduling results to the CRD configuration file, and then issue the AI ​​training task creation request to the target cluster; When the AI ​​training task request carries a hardware resource quantity request, the scheduling result includes the hardware resource quantity request, the label corresponding to the target node, and resource information.

3. The method for processing a distributed AI training task according to claim 2, wherein: When the AI ​​training task request does not carry a hardware resource quantity request, the scheduling result includes a label and resource information corresponding to the target node; After sending the AI ​​training task creation request to the target cluster, the target cluster creates the AI ​​training task.

4. The method for processing a distributed AI training task according to claim 2, wherein: The hardware resources are GPU resources, or GPU resources and RDMA resources.

5. The method for processing a distributed AI training task according to claim 4, wherein: Get resource information for each hardware resource, including: When the hardware resource is a GPU resource, prometheus is used to monitor and obtain the GPU utilization and task queue status of each node in real time; When the hardware resources are GPU resources and RDMA resources, Prometheus is used to monitor and obtain the GPU utilization, RDMA bandwidth and task queue status of each node in real time.

6. The method for processing a distributed AI training task according to claim 5, wherein: Determine the target node and the target cluster to be scheduled, including: According to the resource information of the hardware resources of each node, the target node to be scheduled and the target cluster to which it belongs are determined through the scheduling strategy.

7. The method for processing a distributed AI training task according to claim 4, wherein: Before the cluster executes the AI ​​training task that carries the hardware resource quantity request, it also includes: When the hardware resources are GPU resources and RDMA resources, the webhook service is used to detect the RDMA link status in the cluster in real time, and when the link is abnormal, the backup communication protocol and gradient quantization compression configuration are automatically injected into the AI ​​training task.

8. The method for processing distributed AI training tasks according to claim 1, wherein: After modifying the resource information in the CRD configuration file to a parameter format adapted to the cluster hardware environment, the method further includes: Injecting a Sidecar container into the AI ​​training task through the webhook service, and monitoring the hardware resources in real time through the Sidecar container when executing the AI ​​training task; When the Sidecar container detects hardware resource anomalies or link failures, it dynamically adjusts the training parameters or communication protocols of the main container.

9. The method for processing distributed AI training tasks according to claim 1, wherein: Deploy the webhook service, including: Develop webhook service program to build container image; Deploy the container image to a cluster so that the webhook service runs in the cluster.

10. A distributed AI training task processing system, characterized in that: The system comprises: A setting module is used to set labels for the hardware resources of the nodes of each cluster of each cloud platform and obtain resource information of each hardware resource; A deployment execution module is used to intercept a CRD configuration file containing a request for the number of hardware resources through a webhook service pre-deployed in the cluster, and after modifying the resource information in the CRD configuration file to a parameter format adapted to the cluster hardware environment based on the label, create an AI training task carrying the request for the number of hardware resources.

Images