Sublimation NPU-based distributed training system and method

Abstract

The invention relates to a distributed training system and method based on mercuric chloride NPU, and the method comprises the steps: a batch job scheduler is configured to receive a batch job configuration list, and creates and manages a head node Pod and a plurality of work nodes Pod in batches on nodes with idle mercuric chloride NPU in a cluster managed by a container arrangement platform, the head node Pod and the working node Pod are automatically networked based on a unified distributed computing framework to form a distributed computing cluster special for tasks; the main control service module is further configured to monitor the networking state of the distributed computing cluster, and submit a stand-alone training starting command to the head node Pod after confirming that networking is completed; in addition, the head node Pod is further configured to take the received starting command as a distributed task, and the distributed task is scheduled to each working node Pod for parallel execution through a unified distributed computing framework, so that the use threshold is greatly reduced, and the development efficiency and the system reliability are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a distributed training system and method based on Ascend NPU. Background Technology

[0002] Currently, mainstream distributed training technologies based on Ascend NPUs are primarily implemented using algorithm frameworks and Kubernetes. This requires users to use Kubernetes to schedule hardware resources, create container environments for algorithm execution, and then call various algorithm frameworks to implement distributed training programming within these containers. Finally, execution scripts are written on each node to call the algorithm frameworks and perform distributed computation on each node. This process requires algorithm developers to know the IP addresses of each training node and to write and execute the relevant training scripts on each node.

[0003] However, existing technologies have the following drawbacks: High complexity of use: Algorithm developers need to have a deep understanding of the configuration details of Kubernetes and Volcano, as well as the underlying distributed communication principles (such as HCCL). The technology stack is lengthy, and the learning cost and operating threshold are extremely high. Summary of the Invention

[0004] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a distributed training system and method based on Ascend NPU, which solves the technical problem of high usage complexity in the prior art.

[0005] (II) Technical Solution To achieve the above objectives, the main technical solutions adopted by the present invention include: In a first aspect, embodiments of the present invention provide a distributed training system based on Ascend NPU, comprising: a main control service module, deployed in a cluster managed by a container orchestration platform, configured to receive training task requests submitted by users through a user-facing task submission interface, parse the training task requests and automatically generate corresponding batch job configuration lists, and submit the batch job configuration lists by calling the container orchestration platform management interface; the training task request includes a training code image, a single-machine training start command, and the required number of worker nodes; The batch job scheduler runs in the cluster managed by the container orchestration platform and is configured with the Ascend device management plugin. It is configured to receive batch job configuration lists and create and manage one head node Pod and multiple worker node Pods in batches on nodes with idle Ascend NPUs in the cluster managed by the container orchestration platform. The head node Pod and worker node Pods automatically network based on a unified distributed computing framework to form a task-specific distributed computing cluster. In the distributed computing cluster, the head node Pod is configured as the cluster coordinator, and the worker node Pods are automatically registered to the head node Pod through service discovery information injected by the container orchestration platform. The main control service module is also configured to monitor the network status of the distributed computing cluster and, after confirming that the network is complete, submit the single-machine training start command to the head node Pod. The head node Pod is also configured to treat the received startup command as a distributed task, which is then scheduled to be executed in parallel on various worker node Pods through a unified distributed computing framework. During the creation of worker node Pods, the container orchestration platform automatically injects the environment variables required for distributed training based on the batch job configuration list. This enables the training code pointed to in the single-machine training startup command to automatically call the Ascend NPU and the collection communication library when running in the worker node Pods, thus completing the training in a distributed data parallel mode.

[0006] In one possible embodiment, the main control service module is specifically configured as follows when generating the batch job configuration list: Based on the required number of worker nodes, determine the total number of Pods to be created and the resource requests of each Pod to the Ascend NPU; Define the Pod's startup command in the configuration manifest. The startup command is used to make the Pod automatically run the processes of the unified distributed computing framework after startup.

[0007] In one possible embodiment, the environment variables automatically injected by the container orchestration platform include at least a global ranking for identifying the global uniqueness of a process, a local ranking for identifying the device index within a node, and a global total number of processes for identifying the total number of processes.

[0008] In one possible embodiment, the main control service module is further configured to perform elastic scaling management, specifically configured to: continuously monitor the performance metrics and resource utilization of training tasks during the operation of the distributed computing cluster; dynamically determine whether the number of worker node Pods needs to be adjusted based on the performance metrics and resource utilization, and generate corresponding scaling-up or scaling-down decisions; automatically update the number of worker nodes in the batch job configuration list according to the decision, and trigger the batch job scheduler to execute the corresponding Pod instance creation or deletion operations; wherein, the batch job scheduler is configured to create new worker node Pods or delete specified worker node Pods in batches in response to the trigger in a group scheduling manner; the newly created worker node Pods are automatically added to the existing distributed computing cluster based on a unified distributed computing framework, and the computing tasks on the deleted worker node Pods are automatically rescheduled to other worker node Pods in the cluster.

[0009] In one possible embodiment, during the parallel execution of training tasks by worker Pods, dynamic communication optimization is collaboratively performed by the main control service module, the unified distributed computing framework, and the worker Pods. Specifically, the main control service module and / or the unified distributed computing framework are configured to select the optimal strategy from multiple communication optimization strategies during the gradient synchronization phase, based on real-time collected cluster communication status. The communication optimization strategies include one or more combinations of gradient sharding, gradient compression, and mixed-precision training. The worker Pods are configured to process local gradient data according to the selected optimal strategy and perform gradient aggregation and synchronization operations by calling the underlying aggregated communication library. The main control service module and / or the unified distributed computing framework are also configured to continuously monitor the performance indicators of gradient synchronization and dynamically adjust the adopted communication optimization strategy based on performance feedback, forming a closed-loop optimization.

[0010] Secondly, embodiments of the present invention provide a distributed training method based on Ascend NPU, comprising: The main control service receives training task requests submitted by users through the user-facing task submission interface, parses the training task requests and automatically generates the corresponding batch job configuration list, and submits the batch job configuration list by calling the container orchestration platform management interface. The training task request includes the training code image, the single-machine training start command and the required number of worker nodes. The batch job configuration list is received through the batch job scheduler, which runs in the cluster managed by the container orchestration platform and is configured with the Ascend device management plugin. The batch job scheduler creates and manages a head node Pod and multiple worker node Pods on nodes with idle Ascend NPUs within the cluster managed by the container orchestration platform. The head node Pod and worker node Pods automatically network based on a unified distributed computing framework to form a task-specific distributed computing cluster. In the distributed computing cluster, the head node Pod is configured as the cluster coordinator, and the worker node Pods are automatically registered to the head node Pod through service discovery information injected by the container orchestration platform. The master control service monitors the network status of the distributed computing cluster and submits the single-machine training start command to the head node Pod after confirming that the network is in place. The head node Pod receives the startup command as a distributed task and schedules it to various worker node Pods for parallel execution using a unified distributed computing framework. During the creation of worker node Pods, the container orchestration platform automatically injects the environment variables required for distributed training based on the batch job configuration list. This enables the training code pointed to in the single-machine training startup command to automatically call the Ascend NPU and the collection communication library when running in the worker node Pods, thus completing the training in a distributed data parallel mode.

[0011] In one possible embodiment, the steps for generating the batch job configuration manifest include: determining the total number of Pods to be created and the request for Ascend NPU resources for each Pod based on the required number of worker nodes; defining the Pod startup command in the configuration manifest, which is used to enable the Pod to automatically run the process of a unified distributed computing framework after startup.

[0012] In one possible embodiment, the environment variables automatically injected by the container orchestration platform include at least a global ranking for identifying the global uniqueness of a process, a local ranking for identifying the device index within a node, and a global total number of processes for identifying the total number of processes.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including: a memory for storing a computer program; and a processor for calling the computer program to cause the electronic device to execute the above-described distributed training method based on Ascend NPU.

[0014] Fourthly, this application provides a computer storage medium including computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned distributed training method based on Ascend NPU.

[0015] (III) Beneficial Effects The beneficial effects of this invention are: This application proposes a distributed training system and method based on Ascend NPU, which aims to completely encapsulate the underlying hardware resources (Ascend NPU), resource scheduling (Kubernetes and Volcano), and distributed communication details, making them transparent to algorithm developers. As a result, users do not need to write Kubernetes configuration and distributed training scripts, but only need to focus on the core algorithm logic and complete distributed training tasks automatically and in one stop through simple API calls, which greatly reduces the threshold for use and improves development efficiency and system reliability.

[0016] To make the above-mentioned objectives, features and advantages to be achieved by the embodiments of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This paper illustrates a structural block diagram of a distributed training system based on Ascend NPU provided in an embodiment of this application; Figure 2 This illustration shows a timing diagram of ResNet-50 distributed training provided in an embodiment of this application; Figure 3 A flowchart illustrating an elastic scaling method in large-scale training provided in an embodiment of this application is shown; Figure 4 This document illustrates a flowchart of gradient synchronization optimization during training, as proposed in an embodiment of this application. Figure 5 A flowchart of a distributed training method based on Ascend NPU provided in an embodiment of this application is shown. Detailed Implementation

[0019] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Besides the high complexity of use, existing technologies also have the following problems: Low flexibility and development efficiency: Training tasks are tightly coupled with the underlying hardware and scheduling system. Any change to resources (such as adding nodes) requires users to manually modify configurations and scripts, making dynamic elastic scaling impossible and the development iteration process cumbersome. Error-prone: Users need to manage complex processes such as node discovery and communication establishment themselves, which can easily lead to training task failure due to configuration errors (such as incorrect IP address or port), making operation and maintenance difficult.

[0021] Based on this, this application proposes a distributed training system and method based on Ascend NPU, which aims to completely encapsulate the underlying hardware resources (Ascend NPU), resource scheduling (Kubernetes and Volcano), and distributed communication details, making them transparent to algorithm developers. As a result, users do not need to write Kubernetes configuration and distributed training scripts, but only need to focus on the core algorithm logic and complete the distributed training task automatically and in one stop through simple API calls, which greatly reduces the threshold for use and improves development efficiency and system reliability.

[0022] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0023] To facilitate understanding of the embodiments of this application, the terms involved are explained below: Ascend NPU: A neural network processor.

[0024] Distributed training: A technique that breaks down a large AI model training task into multiple computing nodes for parallel execution.

[0025] Kubernetes: An open-source container orchestration system for automating the deployment, scaling, and management of containerized applications.

[0026] Volcano: A batch and high-performance workload scheduler based on Kubernetes.

[0027] Ascend-Volcano-Plugin: A plugin for Volcano that supports the scheduling and management of Ascend NPU devices.

[0028] Ray: A unified distributed computing framework for easily scaling Python applications.

[0029] Example 1

[0030] Please see Figure 1 , Figure 1 This diagram illustrates a structural block diagram of a distributed training system based on the Ascend NPU, as provided in an embodiment of this application. Figure 1 The distributed training system shown includes: The Master Service module, deployed within the cluster managed by the container orchestration platform (e.g., a Kubernetes cluster if the platform is Kubernetes), serves as the control center of the entire system. It receives training task requests from users through a client-side task submission interface. These requests employ declarative configuration, requiring only basic information such as the training code image, the single-machine training start command, and the desired number of worker nodes. Internally, the Master Service module may include components such as a task parsing engine, a configuration generator, and a status monitor. It can automatically parse user requests and generate corresponding batch job configuration lists (Volcano container orchestration platform Job configurations), then submit these lists by calling the container orchestration platform's management interface (Kubernetes container orchestration platform API). The task parsing engine is responsible for performing syntactic and semantic analysis on user-submitted training task requests, extracting key parameters and verifying their legality; the configuration generator has a built-in intelligent template system that can dynamically generate optimized Volcano Job configurations based on task characteristics and cluster status, accurately defining Pod specifications, resource requirements and service configurations; and the status monitor continuously tracks the task lifecycle status, from resource application and cluster setup to task execution and resource reclamation.

[0031] The Volcano Scheduler runs on clusters managed by the container orchestration platform and is configured with a dedicated plugin (Ascend-Volcano-Plugin) for Ascend NPU device management, providing device management capabilities for Ascend NPUs. After receiving the batch job configuration list submitted by the main control service module, the scheduler scans the cluster managed by the container orchestration platform for compute nodes with idle Ascend NPU resources. It then uses a group scheduling strategy (Gang container orchestration platform Scheduling) to create and manage one Ray Head Pod and multiple Ray Worker Pods in batches. During scheduling, the system comprehensively considers factors such as the interconnection topology between NPUs within a node, cross-node network latency, and device model consistency to achieve optimal resource allocation.

[0032] For example, the batch job scheduler first performs resource discovery, scanning the available Ascend NPU resources in the cluster. Then, it selects the optimal computing node based on strategies such as node affinity, resource balancing, and topology awareness. Finally, it creates Pods in batches using group scheduling to ensure that all components of the distributed task can start collaboratively, avoiding resource deadlock and task skew.

[0033] Furthermore, head node Pods and worker node Pods automatically network based on a unified distributed computing framework (Ray) to form a task-specific distributed computing cluster (Ray Cluster). During the formation of the distributed computing cluster, the head node Pod automatically starts the Ray master process (Ray Head) as the cluster coordinator, and worker node Pods automatically discover and register with the head node Pod through service discovery information injected by the container orchestration platform, forming a complete distributed computing environment. For example, the head node Pod will first start and automatically initialize as the head node of the Ray cluster, and the worker node Pods will automatically discover and connect to the head node Pod through pre-injected environment variables, forming a complete Ray computing cluster.

[0034] In other words, the distributed computing cluster is built on the Ray framework and includes two roles: head node Pods and worker node Pods. The head node Pod runs the Ray master process, responsible for distributed computing cluster management, metadata storage, and task scheduling. The worker node Pods run Ray worker processes and automatically connect to the head node Pod through service discovery information injected by the system, forming a unified distributed execution environment. Furthermore, during the setup of the distributed computing cluster, the system automatically handles complex processes such as network configuration, authentication, and resource registration, completely transparent to the user.

[0035] Additionally, the main control service module continuously monitors the network status of the distributed computing cluster. After confirming that all worker node Pods are ready, it submits the user-submitted single-machine training start command to the head node Pod. The head node Pod treats the received start command as a distributed task and uses Ray's core scheduler to decompose and schedule the task to be executed in parallel on each worker node Pod.

[0036] During the creation of worker node Pods, the container orchestration platform automatically injects the environment variables required for distributed training based on the batch job configuration list. This enables the training code pointed to by the single-machine training startup command to automatically call the Ascend NPU and the collection communication library when running in the worker node Pods, completing the training in a distributed data parallel mode. These environment variables include at least the global rank (RANK) for identifying the global uniqueness of a process, the local rank (LOCAL_RANK) for identifying the device index within the node, and the global process count (WORLD_SIZE) for identifying the total number of processes.

[0037] Furthermore, when the training code runs in the worker node Pod, it reads these environment variables, and the underlying deep learning framework (such as PyTorch) automatically initializes the distributed process group, adapts the data loader to distributed sampling mode, and wraps the model in distributed data parallel mode. This allows the training code, originally designed for a single machine, to automatically call the Ascend NPU and HCCL communication library to complete training in distributed data parallel mode. After training is complete, the model file is written to shared storage, the main control service module updates the task status and returns the results, and finally automatically cleans up all computing resources. For details, please refer to [link to documentation]. Figure 2 This will not be described in detail here.

[0038] In addition, when generating the batch job configuration list, the main control service module is specifically configured to: determine the total number of Pods to be created and the request of each Pod for Ascend NPU resources based on the required number of worker nodes; define the Pod startup command in the configuration list, which is used to enable the Pod to automatically run the process of the unified distributed computing framework after startup.

[0039] With the support of the aforementioned infrastructure, this application achieves intelligent elastic scaling management of distributed training clusters. For example... Figure 3 As shown, the system can dynamically adjust computing resources according to changes in training load. Its working mechanism is explained in detail below.

[0040] Specifically, such as Figure 3 As shown, Figure 3 A flowchart illustrating an elastic scaling method in large-scale training according to an embodiment of this application is shown, and its specific implementation process is as follows: During the initial deployment phase, a gradual resource allocation strategy is adopted, first creating a minimum available cluster containing one head node Pod and eight worker node Pods. This design is based on multiple considerations: a small cluster can complete the resource scheduling and startup process more quickly, effectively avoiding excessive allocation of resources in the early stages, while providing a test platform for verifying the normality of the underlying environment.

[0041] Subsequently, the monitoring and decision-making phase is led by the main control service module, which continuously collects multi-dimensional runtime metrics. System monitoring resource metrics include NPU utilization, GPU memory usage, network bandwidth, and storage I / O performance; performance metrics cover training throughput, iteration speed, gradient synchronization time, and computational bottleneck analysis; business metrics focus on loss function convergence trends, training progress, and model accuracy changes. Monitoring data is stored in a time-series database and trend analysis is performed using a sliding window algorithm. When the system detects that NPU utilization remains above 85% for more than 5 minutes, or the time of a single iteration cycle exceeds a preset threshold, or GPU memory usage remains above 90%, a capacity expansion assessment process is triggered.

[0042] Furthermore, after the elastic scaling phase begins, the system executes a complete automated scaling process. First, it analyzes the current cluster status and resource requirements to generate an optimized scaling plan. The master control service module automatically updates the replica count specification in the Volcano Job configuration, increasing the number of worker Pods from 8 to 32. Upon receiving the updated configuration, the Volcano scheduler creates 24 new worker Pods in batches using group scheduling. During resource allocation, the scheduler prioritizes nodes with lower network latency to existing Pods and uses fine-grained affinity and anti-affinity settings to ensure that new Pods are in the optimal network topology with existing Pods. Simultaneously, the system reserves a certain amount of resource buffer to avoid scheduling failures due to resource contention.

[0043] Furthermore, the cluster reorganization phase begins immediately after the new worker node Pod starts. The new Pod automatically discovers the network endpoint of the head node Pod through Kubernetes' Service mechanism, establishes a secure connection based on TLS certificates, and obtains its distributed identity from the head node Pod. After authentication, the new Pod registers its available computing resources (including the number of NPUs, memory size, etc.) with the head node Pod. During this period, the Ray framework performs intelligent load balancing, dynamically reallocating computing tasks based on the computing capabilities of each node, ensuring that newly added nodes can quickly integrate into the computing cluster and begin assuming workloads. The system also re-shards the training dataset to ensure balanced data distribution. During the stable training phase, when the 32 worker node Pods work collaboratively, the system implements comprehensive performance optimization measures. Gradient accumulation and sharding techniques optimize memory usage, pipelined parallelism improves computational efficiency, and a periodic checkpoint mechanism prevents training interruptions caused by node failures. The monitoring system continuously tracks the cluster's health status, including node liveness detection, network connectivity verification, and performance metric collection, ensuring stable operation of large-scale training.

[0044] Furthermore, in this scenario, users need to launch 32 Workers for large-scale parameter fine-tuning. The process is exactly the same as the architecture described above, reflecting the consistent design of the system. Users still only need to write standard single-machine training scripts and submit tasks through the same simple API interface; the only difference is to modify the `num_workers` parameter to 32 and specify the corresponding computing resources.

[0045] Furthermore, the system's internal processing flow demonstrates powerful scalability: the Master Service generates a Volcano Job configuration requiring 32 Workers, and the Volcano scheduler efficiently performs batch scheduling, creating Pod instances on multiple physical servers. The Ray framework perfectly handles the networking of large-scale clusters, automatically building a large computing cluster across nodes, containing 32 Ascend NPU cards.

[0046] In other words, during the operation of the distributed computing cluster, the performance metrics and resource utilization of the training tasks are continuously monitored; based on the performance metrics and resource utilization, it is dynamically determined whether the number of worker node Pods needs to be adjusted, and corresponding scaling-up or scaling-down decisions are generated; according to the decisions, the number of worker nodes in the batch job configuration list is automatically updated, and the batch job scheduler is triggered to execute the corresponding Pod instance creation or deletion operations; the batch job scheduler is configured to create new worker node Pods or delete specified worker node Pods in batches in response to triggering, using group scheduling; the newly created worker node Pods are automatically added to the existing distributed computing cluster based on a unified distributed computing framework, and the computing tasks on the deleted worker node Pods are automatically rescheduled to other worker node Pods in the cluster.

[0047] It should be noted here that, although Figure 3 This is merely an exemplary description, and those skilled in the art can adjust the relevant data according to actual needs.

[0048] While ensuring a flexible supply of resources, this application further improves training efficiency through an intelligent gradient synchronization optimization mechanism. For example... Figure 4 As shown, the system implements a multi-level, adaptive optimization strategy in the gradient synchronization stage. The technical details are described below.

[0049] like Figure 4 As shown, Figure 4 The flowchart of gradient synchronization optimization during training, as proposed in an embodiment of this application, is shown below. The process begins with each worker node (Pod) independently completing forward propagation and calculating its local gradient. Subsequently, the system enters the gradient synchronization optimization phase: first, gradient collection and preparation are performed, followed by the selection of the optimal communication optimization strategy based on real-time cluster status data via an intelligent decision engine. This strategy is implemented at the optimization strategy layer, which collaboratively utilizes three core technologies: gradient sharding, gradient compression, and mixed-precision training, to preprocess the gradient data to be synchronized. Gradient sharding is an optimization technique that achieves communication load balancing by dividing the gradient tensor dimensions. Its core idea is to divide a large-scale gradient tensor into multiple fragments, reducing the communication load and peak memory usage of a single node through parallel communication and processing. Gradient compression is a communication optimization method that reduces the amount of gradient data transmitted using lossy or lossless encoding techniques, minimizing communication bandwidth requirements while ensuring model convergence. Mixed-precision training is a computational method that improves training efficiency and reduces resource consumption by collaboratively using data types with different numerical precisions. Its core objective is to fully utilize the high-performance advantages of low-precision computation while maintaining model convergence accuracy.

[0050] The optimized gradient data is then processed by calling the underlying HCCL library to perform an All-Reduce operation, achieving efficient gradient aggregation and synchronization. The synchronized gradients are used for parameter updates, thereby achieving model weight synchronization across all worker nodes.

[0051] The final crucial step in the process is real-time monitoring and adaptive adjustment. The system continuously monitors the communication status and dynamically optimizes the strategy based on the results of performance feedback analysis. This feedback loop can dynamically adjust the communication strategy adopted (such as switching the All-Reduce algorithm or adjusting compression parameters) according to actual operating performance, thereby enabling the entire gradient synchronization process to have the ability to continuously self-optimize, forming a highly efficient closed-loop optimization system.

[0052] In other words, the main control service module and / or the unified distributed computing framework are configured to select the optimal strategy from multiple communication optimization strategies during the gradient synchronization phase, based on real-time collected cluster communication status. These communication optimization strategies include one or more combinations of gradient sharding, gradient compression, and mixed-precision training. Worker Pods are configured to process local gradient data according to the selected optimal strategy and perform gradient aggregation and synchronization operations by calling the underlying aggregated communication library. Furthermore, the main control service module and / or the unified distributed computing framework are configured to continuously monitor gradient synchronization performance metrics and dynamically adjust the adopted communication optimization strategy based on performance feedback, forming a closed-loop optimization. Here, "main control service module and / or unified distributed computing framework" refers to the main control service module, the unified distributed computing framework, or the main control service module and the unified distributed computing framework.

[0053] Therefore, with the help of the above technical solutions, the system can easily handle ultra-large-scale training tasks. Users do not need to worry about the underlying details such as cross-node communication and resource scheduling, but only need to focus on the model architecture and algorithm logic itself. The monitoring interface provided by the system also allows users to view training progress, resource utilization and performance indicators in real time, and its elastic scalability allows users to dynamically adjust the resource scale according to training needs, truly realizing a "one-click" experience and elastic computing for distributed training.

[0054] It should be understood that the above-described distributed training system and method based on Ascend NPU are merely exemplary, and those skilled in the art can make various modifications based on the above method, and the modified solutions also fall within the protection scope of this application.

[0055] Example 2

[0056] Please see Figure 5 , Figure 5 A flowchart illustrating a distributed training method based on Ascend NPU provided in an embodiment of this application is shown. Specifically, the distributed training method includes: Step S510: The main control service receives training task requests submitted by users through the user-facing task submission interface, parses the training task requests and automatically generates the corresponding batch job configuration list, and submits the batch job configuration list by calling the container orchestration platform management interface; wherein, the training task request includes the training code image, the single-machine training start command and the required number of worker nodes. Step S520: Receive the batch job configuration list through the batch job scheduler, wherein the batch job scheduler runs in the cluster managed by the container orchestration platform and is configured with the Ascend device management plugin. In step S530, the batch job scheduler creates and manages a head node Pod and multiple worker node Pods on nodes with idle Ascend NPUs within the cluster managed by the container orchestration platform. The head node Pod and worker node Pods automatically network based on a unified distributed computing framework to form a task-specific distributed computing cluster. In the distributed computing cluster, the head node Pod is configured as the cluster coordinator, and the worker node Pods are automatically registered to the head node Pod through service discovery information injected by the container orchestration platform. Step S540: Monitor the network status of the distributed computing cluster through the master control service, and submit the single-machine training start command to the head node Pod after confirming that the network is completed. Step S550: The received startup command is treated as a distributed task by the head node Pod and scheduled to be executed in parallel on each worker node Pod using a unified distributed computing framework. During the creation of worker node Pods, the container orchestration platform automatically injects the environment variables required for distributed training based on the batch job configuration list. This enables the training code pointed to in the single-machine training startup command to automatically call the Ascend NPU and the collection communication library when running in the worker node Pods, thus completing the training in a distributed data parallel mode.

[0057] In one possible embodiment, the steps for generating the batch job configuration manifest include: determining the total number of Pods to be created and the request for Ascend NPU resources for each Pod based on the required number of worker nodes; defining the Pod startup command in the configuration manifest, which is used to enable the Pod to automatically run the process of a unified distributed computing framework after startup.

[0058] In one possible embodiment, the environment variables automatically injected by the container orchestration platform include at least a global ranking for identifying the global uniqueness of a process, a local ranking for identifying the device index within a node, and a global total number of processes for identifying the total number of processes.

[0059] It should be understood that Figure 5 The method described is part of the method of the first embodiment, so its related process can be found in the relevant description of the first embodiment, and will not be repeated here.

[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0062] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0063] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0064] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0065] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A distributed training system based on a NPU of Ascend, characterized in that, The method comprises the following steps: A master service module is deployed in a cluster managed by a container orchestration platform and is configured to receive a user-submitted training task request through a user-oriented task submission interface, analyze the training task request, and automatically generate a corresponding batch job configuration list, and submit the batch job configuration list by calling a container orchestration platform management interface; the training task request comprises a training code image, a single-machine training start command, and a required number of worker nodes; A batch job scheduler runs in the cluster managed by the container orchestration platform and is configured with an Ascending device management plug-in, configured to receive the batch job configuration list, and batch create and manage a head node Pod and multiple worker node Pods on nodes in the cluster managed by the container orchestration platform that have idle Ascending NPUs, the head node Pod and the worker node Pod are automatically networked based on a unified distributed computing framework to form a task-specific distributed computing cluster; wherein in the distributed computing cluster, the head node Pod is configured as a cluster coordinator, and the worker node Pod is automatically registered to the head node Pod through service discovery information injected by the container orchestration platform; The master service module is further configured to monitor the networking state of the distributed computing cluster, and submit the single-machine training start command to the head node Pod after confirming that the networking is completed; The head node Pod is further configured to receive the start command as a distributed task and schedule it to each worker node Pod for parallel execution through the unified distributed computing framework; During the creation of the worker node Pod, the container orchestration platform automatically injects environment variables required for distributed training according to the batch job configuration list, so that the training code pointed to in the single-machine training start command can automatically call the Ascending NPU and collective communication library when running in the worker node Pod, and complete the training in a distributed data parallel mode.

2. The distributed training system of claim 1, wherein, When generating the batch job configuration list, the master service module is specifically configured to: According to the required number of worker nodes, determine the total number of Pods to be created and the request of each Pod to Ascending NPU resources; Define the start command of the Pod in the configuration list, which is used to make the Pod automatically run the process of the unified distributed computing framework after starting.

3. The distributed training system of claim 1, wherein, The environment variables automatically injected by the container orchestration platform at least include a global rank for identifying the global uniqueness of a process, a local rank for identifying the index of a device within a node, and a global process total for identifying the total number of processes.

4. The distributed training system of claim 1, wherein, The master service module is further configured to perform elastic scaling management, specifically configured to: During the operation of the distributed computing cluster, continuously monitor the performance indicators and resource utilization of the training task; Based on the performance indicators and resource utilization, dynamically determine whether the number of worker node Pods needs to be adjusted, and generate a corresponding expansion or contraction decision; According to the decision, the number of worker nodes in the batch job configuration list is automatically updated, and the batch job scheduler is triggered to perform corresponding Pod instance creation or deletion operations; Wherein, the batch job scheduler is configured to respond to the trigger to create new worker node Pods or delete specified worker node Pods in a group scheduling manner; The newly created worker node Pods automatically join the existing distributed computing cluster based on a unified distributed computing framework, and the computing tasks on the deleted worker node Pods are automatically rescheduled to other worker node Pods in the cluster.

5. The distributed training system of claim 1, wherein, During the process of the worker node Pods executing the training task in parallel, dynamic communication optimization is cooperatively performed by the master service module, the unified distributed computing framework, and the worker node Pods, specifically including: The master service module and / or the unified distributed computing framework are configured to select an optimal strategy from multiple communication optimization strategies based on the real-time collected cluster communication state during the gradient synchronization stage; the communication optimization strategies include one or more combinations of gradient slicing, gradient compression, and mixed precision training; The worker node Pods are configured to process local gradient data according to the selected optimal strategy and perform gradient aggregation and synchronization operations by calling the underlying collective communication library; Wherein, the master service module and / or the unified distributed computing framework are further configured to continuously monitor the performance indicators of gradient synchronization and dynamically adjust the communication optimization strategy according to performance feedback to form a closed-loop optimization.

6. A distributed training method based on the Ascend NPU, characterized in that, Including: Receiving a training task request submitted by a user through a user-oriented task submission interface by the master service, parsing the training task request and automatically generating a corresponding batch job configuration list, and submitting the batch job configuration list by calling a container orchestration platform management interface; wherein the training task request includes a training code image, a single-machine training start command, and the required number of worker nodes; Receiving the batch job configuration list by the batch job scheduler, wherein the batch job scheduler runs in a cluster managed by a container orchestration platform and is configured with an Ascend device management plug-in; Batch creating and managing a head node Pod and multiple worker node Pods on nodes with idle Ascend NPUs in the cluster managed by the container orchestration platform by the batch job scheduler; the head node Pod and the worker node Pods automatically form a task-specific distributed computing cluster based on a unified distributed computing framework; wherein in the distributed computing cluster, the head node Pod is configured as a cluster coordinator, and the worker node Pods automatically register to the head node Pod through service discovery information injected by the container orchestration platform; Monitoring the networking state of the distributed computing cluster by the master service, and submitting the single-machine training start command to the head node Pod after confirming that the networking is complete; The head node Pod receives the received start command as a distributed task, and schedules it to each of the worker node Pods for parallel execution by using the unified distributed computing framework; In the creation process of the worker node Pod, the container orchestration platform automatically injects environment variables required for distributed training according to the batch job configuration manifest, so that when the training code pointed to in the single-machine training start command runs in the worker node Pod, the Ascend NPU and collective communication library can be automatically called to complete training in a distributed data parallel mode.

7. The distributed training method of claim 6, wherein, The generation step of the batch job configuration manifest includes: According to the required number of worker nodes, determine the total number of Pods to be created and the request of each Pod to Ascend NPU resources; Define the start command of the Pod in the configuration manifest, which is used to make the Pod automatically run the process of the unified distributed computing framework after starting.

8. The distributed training method of claim 6, wherein, The environment variables automatically injected by the container orchestration platform at least include a global rank for identifying the global uniqueness of the process, a local rank for identifying the index of the device within the node, and a global process total for identifying the total number of processes.

9. An electronic device, comprising: It includes: A memory for storing a computer program; A processor for calling the computer program, so that the electronic device executes the Ascend NPU-based distributed training method according to any one of claims 6-8.

10. A computer medium, characterized by The computer medium includes computer instructions, when the computer instructions run on the electronic device, the electronic device executes the Ascend NPU-based distributed training method according to any one of claims 6-8.

Images