A multi-machine multi-card distributed training optimization system and method for a signal creation heterogeneous environment

Abstract

This invention discloses a multi-machine, multi-GPU distributed training optimization system and method for heterogeneous environments in the field of domestic IT innovation, relating to the field of distributed training technology. To address the shortcomings of distributed training in heterogeneous environments, the system includes: an application layer that receives training tasks; a core system layer comprising three modules: adaptive task scheduling, hybrid communication topology optimization, and dynamic checkpointing and elastic recovery control. Specifically: the scheduling module generates performance profiles and optimal solutions based on training tasks and adjusts the solution according to node failures; the optimization module executes the solution, first probing the node topology, constructing hierarchical paths, then loading and initializing the optimal communication backend for the target device, and synchronizing the path to the target device; the control module periodically saves the complete training context and enables reliable recovery of the training process; a domestic IT innovation hardware abstraction layer provides a unified interface for accessing underlying resources and acquiring data; and a hardware resource layer, including heterogeneous computing nodes and RoCE network switches.

Description

Technical Field

[0001] This invention relates to the field of distributed training technology, specifically to a multi-machine, multi-card distributed training optimization system and method for heterogeneous environments in the field of domestic IT innovation. Background Technology

[0002] Currently, the industry typically employs solutions like PyTorch's Distributed Data Parallel (DDP) in the heterogeneous computing field. However, these solutions are designed for homogeneous GPU clusters, and their communication backends (such as NCCL) and scheduling strategies fail to consider the complexities of multiple architectures with varying performance, including CPUs, NPUs, and DCUs, in the context of domestic IT innovation environments. This creates a fundamental contradiction between the homogeneous design premise and the inherent heterogeneity of domestic IT innovation environments. Consequently, the communication, scheduling, and fault tolerance mechanisms cannot function efficiently in this environment, let alone achieve synergistic optimization among them. Furthermore, with the rise of large-scale AI models such as Large Language Models (LLMs), distributed training across multiple machines and GPUs has become the norm in model development.

[0003] In the context of emphasizing independent and controllable domestic IT infrastructure, computing infrastructure typically consists of various domestically produced CPUs (such as Hygon and Phytium), computing cards (such as Ascend NPU and Hygon DCU), and domestically produced high-speed network equipment based on RoCE technology. However, in-depth research and practice have revealed that directly migrating existing distributed training technologies to the domestic IT infrastructure environment will face the following unique collaborative technical challenges that existing technologies have not yet adequately addressed:

[0004] (1) Communication and hardware coordination problem: Traditional communication libraries designed for homogeneous GPU clusters cannot efficiently utilize the diverse high-speed buses between NPUs, and their utilization efficiency of RoCE networks is low, resulting in the inability to realize the potential of hardware communication.

[0005] (2) Scheduling and heterogeneous coordination challenges: Simple load balancing algorithms cannot cope with the multi-dimensional heterogeneity of "computing power-memory-communication". Static task allocation will drastically reduce overall efficiency due to the shortcomings of a single node.

[0006] (3) The challenge of coordinating fault tolerance and training: In a heterogeneous environment, simple checkpoint recovery may lead to slight deviations in gradient calculation due to hardware differences. These deviations accumulate during long-term training and will eventually affect the accuracy of the model. Traditional fault tolerance mechanisms lack the ability to verify this.

[0007] Therefore, existing technologies lack a technical solution that can integrate and coordinate communication, scheduling, and fault tolerance at the system level to address the comprehensive challenges of the heterogeneous environment of information technology innovation. Summary of the Invention

[0008] To overcome the problems of low communication efficiency, rigid resource scheduling, poor fault tolerance, and inability to achieve collaborative optimization when existing distributed training technologies are designed for homogeneous environments and directly applied to heterogeneous environments for domestic IT innovation, this invention provides a multi-machine, multi-card distributed training optimization system and method for heterogeneous environments for domestic IT innovation, so as to significantly improve the efficiency and stability of distributed training in heterogeneous environments for domestic IT innovation.

[0009] Firstly, the present invention provides a multi-machine, multi-card distributed training and optimization system for heterogeneous environments in the field of information technology innovation. The technical solution adopted to solve the above-mentioned technical problems is as follows:

[0010] A multi-machine, multi-card distributed training optimization system for heterogeneous environments in the domestic IT innovation field, the system comprising:

[0011] The application layer is used to receive distributed training tasks and related training task information submitted by users and send them down to the core system layer.

[0012] The core system layer comprises three main functional modules: adaptive task scheduling, hybrid communication topology optimization, and dynamic checkpointing and resilient recovery control.

[0013] A) The adaptive task scheduling module is responsible for the full lifecycle management of the distributed training task, specifically including: ① Based on the distributed training task and related information, calling the unified device management interface provided by the domestic IT hardware abstraction layer to obtain a list of available computing devices; ② After the distributed training task is submitted but before it is officially started, testing the computing nodes to which the available computing devices belong, generating a performance profile in the form of a multi-dimensional vector; calling the built-in lightweight cost model to perform simulation analysis, generating the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and issuing execution instructions; ③ After the distributed training is officially started, dynamically adjusting the optimal task allocation and resource scheduling scheme based on the detected computing node failures, and issuing adjustment instructions.

[0014] B) The hybrid communication topology optimization module is responsible for responding to the execution instructions of the optimal task allocation and resource scheduling scheme. First, it calls the unified interface provided by the domestic IT innovation hardware abstraction layer to detect the topology of the computing nodes of each target computing device in the hardware resource layer and construct a hierarchical communication strategy and gradient synchronization path. Then, it calls the unified communication interface of the domestic IT innovation hardware abstraction layer to load and initialize the optimal communication backend for each target computing device in a dynamic selection manner, and synchronizes the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

[0015] C) The dynamic checkpoint and elastic recovery control module is responsible for periodically saving the complete training context and realizing reliable recovery of the training process based on the gradient consistency verification mechanism;

[0016] The IT innovation hardware abstraction layer provides a unified set of device management, memory management, communication interfaces and performance data benchmark interfaces, providing the core system layer with standardized access to underlying resources and performance data acquisition capabilities;

[0017] The hardware resource layer includes multiple heterogeneous computing nodes with different processor architectures and computing card configurations, as well as RoCE network switches for high-speed data interaction between computing nodes. It is the underlying hardware carrier for resource management and interface adaptation of the domestic IT innovation hardware abstraction layer.

[0018] Optionally, the training task-related information includes the model, dataset, hyperparameters, and required computational resources specified for the training task.

[0019] Optionally, the relevant IT innovation hardware abstraction layer has the following functions:

[0020] (1) Unified device management interface: It has a unified device discovery and initialization function, which is used to automatically scan and identify available computing devices in each computing node in the hardware resource layer based on distributed training tasks and related training task information, shielding the API differences in ID recognition and memory query of devices from different manufacturers, and providing a consistent list of available computing devices for performance profiling and task allocation of the core system layer.

[0021] (2) Unified memory management interface: used to provide a unified memory allocation, release and data transfer interface across computing devices, shielding the differences in memory operation between computing devices with different architectures;

[0022] (3) Unified communication interface: It has a unified communication context management function, which is used to encapsulate the initialization and context creation process of dedicated communication libraries for different architecture computing cards; the hybrid communication topology optimization module will be based on this encapsulation and dynamically select and load and initialize the optimal communication backend for each identified target computing device.

[0023] (4) Performance data benchmark interface: Provides a standardized performance data acquisition interface for the adaptive task scheduling module of the core system layer, enabling it to trigger and obtain benchmark test results of available computing devices in different computing nodes in a unified manner.

[0024] Optionally, the adaptive task scheduling module may perform the following operations before the training task officially begins:

[0025] (A1) The adaptive task scheduling module calls the performance data benchmark interface provided by the domestic IT hardware abstraction layer to issue a set of standardized micro-benchmark testing tools to the computing nodes of each identified available computing device. The tools obtain multi-dimensional performance data of each computing node by simulating the computing and communication modes in real model training, and return it to the adaptive task scheduling module through the performance data benchmark interface. The adaptive task scheduling module summarizes and analyzes the returned multi-dimensional performance data of each computing node, and then generates a multi-dimensional vector performance profile of each computing node.

[0026] (A2) Based on training task-related information, performance profiles of each computing node and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis. During the simulation analysis, the computation time, communication time and synchronization overhead that may be caused by non-uniform batches are comprehensively considered to generate the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and then issue execution instructions.

[0027] Optionally, the hybrid communication topology optimization module responds to the execution instructions of the optimal task allocation and resource scheduling scheme, specifically performing the following operations:

[0028] (B1) Call the unified device management interface provided by the domestic IT hardware abstraction layer to first obtain the list of available computing devices, filter the target computing devices from it, and then detect the topology of the computing nodes to which each target computing device belongs, and clarify the network connection relationship between different types of computing cards in the computing nodes and between computing nodes.

[0029] (B2) Based on the topology of the computing nodes to which each target computing device belongs, a hierarchical communication strategy and gradient synchronization path are constructed to limit the core communication traffic to the internal bus of the high-speed node, reduce the access pressure and data volume of the RoCE network between nodes, and reduce communication latency.

[0030] (B3) Call the unified communication interface of the IT innovation hardware abstraction layer, load and initialize the optimal communication backend for each target computing device in a dynamic selection manner, and synchronize the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

[0031] Optionally, the hybrid communication topology optimization module involved constructs a hierarchical communication strategy and gradient synchronization path based on the topology of the computing nodes to which each target computing device belongs. The entire gradient synchronization process follows the path of "intra-node aggregation first, then global synchronization between nodes", as shown below:

[0032] a) Intra-node aggregation: Call the unified communication interface provided by the domestic IT hardware abstraction layer to dynamically select and initialize the optimal dedicated communication backend for different types of computing cards in the computing node, and complete the first gradient aggregation in the computing node.

[0033] b) Aggregation between computing nodes: The adaptive task scheduling module dynamically elects a leader based on the computing power level and network status of each computing node and constructs a logical communication super node; subsequently, the hybrid communication topology optimization module performs global synchronization of the aggregated gradient data within each computing node between the logical communication super nodes based on the All-Reduce gradient synchronization protocol.

[0034] Alternatively, the dynamic checkpoint and resilient recovery control module may perform the following operations:

[0035] (C1) Lightweight full-state asynchronous checkpoint operation: ① Training initialization configuration: When starting a distributed training task, preset the checkpoint saving period, shared storage path and differential compression parameters; ② Periodic asynchronous checkpoint saving: During training, asynchronously collect the current complete training context at preset periods, reduce the data volume through differential compression technology, and write it to shared storage;

[0036] (C2) Node Monitoring and Gradient Consistency Verification Operations: ① Node Fault Monitoring: Start the background monitoring process to monitor the running status of each computing node in the hardware resource layer in real time. Once a computing node fault is detected, immediately alert the adaptive task scheduling module and trigger the training pause mechanism; ② Task Reassignment After Fault: After training is paused, the adaptive task scheduling module uses the unified device management interface provided by the domestic IT hardware abstraction layer to identify healthy nodes in the hardware resource layer. Based on the remaining computing power of the healthy nodes, it dynamically adjusts the optimal task allocation and resource scheduling scheme and issues adjustment instructions to migrate the tasks of the faulty nodes to the healthy nodes; ③ Loading the Latest Checkpoint: On the healthy nodes that have received the adjustment instructions and completed task allocation, the elastic recovery control unit uniformly loads the latest complete checkpoint saved in the shared storage and resumes training. ④ Gradient Consistency Verification: Extract one or more batches of training data corresponding to the checkpoint saving time, re-execute forward and backward propagation calculations on the healthy node to obtain new gradients; ⑤ Gradient Comparison and Judgment: Verify the consistency between the recalculated new gradients and the old gradients saved in the checkpoint by calculating the cosine similarity or norm ratio; ⑧ Resume Training or Retry: If the consistency verification passes, the adaptive task scheduling module synchronously restores the state of the hybrid communication topology optimization module, and the healthy node uses the current learning rate and random number seed to execute the subsequent training process; if the consistency verification fails, it rolls back to an earlier checkpoint or triggers an alarm. After ensuring the mathematical continuity of the training process, the adaptive task scheduling module reissues the training resumption command to start training.

[0037] 8. A multi-machine, multi-card distributed training optimization system for heterogeneous environments in the context of domestic IT innovation, as described in claim 7, is characterized in that, during the gradient comparison judgment process, when performing consistency verification by calculating the cosine similarity or norm ratio, the verification threshold is a dynamic threshold:

[0038] First, based on one or more factors such as the mean norm of the recent gradient tensor, the gradient variance, and the current epoch or step of training, the threshold is dynamically calculated according to a predefined strategy.

[0039] Consistency is then determined using the following rules:

[0040] a) If the calculated cosine similarity is greater than or equal to the dynamic threshold, or the norm ratio is within the range of [1-dynamic threshold, 1+dynamic threshold], then it is determined that the consistency requirement is met.

[0041] b) If the calculated cosine similarity is less than the dynamic threshold and the norm ratio is outside the range of [1-dynamic threshold, 1+dynamic threshold], then it is determined that the consistency requirement is not met.

[0042] Secondly, this invention provides a multi-machine, multi-card distributed training optimization method for heterogeneous environments in the field of domestic IT innovation. The technical solution adopted to solve the above-mentioned technical problems is as follows:

[0043] A distributed training optimization method for multi-machine, multi-card systems in a heterogeneous environment for domestic IT innovation, based on the system described in the first aspect, specifically includes the following operation process:

[0044] S1. The application layer receives the distributed training task and related information submitted by the user and sends it down to the core system layer.

[0045] S2. Based on the distributed training task and related information, the adaptive task scheduling module of the core system layer first calls the unified device management interface provided by the domestic IT hardware abstraction layer to obtain the list of available computing devices, and then tests the computing nodes to which the available computing devices in the list belong, generating a multi-dimensional vector performance profile of each computing node.

[0046] S3. Based on training task-related information, performance profiles of each computing node, and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis, generates the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and issues execution instructions.

[0047] S4. The hybrid communication topology optimization module of the core system layer responds to the execution instructions of the optimal task allocation and resource scheduling scheme, calls the unified device management interface provided by the domestic IT hardware abstraction layer, detects the topology of the computing nodes to which each target computing device belongs in the hardware resource layer, and constructs a hierarchical communication strategy and gradient synchronization path.

[0048] S5. Next, the hybrid communication topology optimization module calls the unified communication interface of the domestic IT hardware abstraction layer to dynamically select and load and initialize the optimal communication backend for each identified target computing device, while synchronizing the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

[0049] S6. Each target computing device begins to execute the training task. In the gradient synchronization stage of each iteration, the hierarchical communication strategy and gradient synchronization path constructed by the hybrid communication topology optimization module are strictly followed to execute the gradient synchronization operation.

[0050] S7. During training, the background monitoring process continuously monitors the running status of each computing node in the hardware resource layer. The dynamic checkpoint and elastic recovery control module periodically saves the complete training context. When a computing node failure is detected, the adaptive task scheduling module dynamically adjusts the task allocation and resource scheduling scheme and issues adjustment instructions. The hybrid communication topology optimization module responds to the instructions and synchronously updates the hierarchical communication strategy and gradient synchronization path. Finally, the reliable recovery of the training process is achieved based on the gradient consistency verification mechanism to ensure training continuity.

[0051] The multi-machine, multi-card distributed training optimization system and method for heterogeneous environments in the field of domestic IT innovation, as proposed in this invention, have the following advantages compared with existing technologies:

[0052] 1. This invention utilizes a hybrid communication topology optimization module to construct a hierarchical communication strategy and gradient synchronization path based on node topology; an adaptive task scheduling module to perform non-uniform task allocation based on benchmark test profiles of domestic chips; and a dynamic checkpoint and elastic recovery control module to achieve rapid fault migration in conjunction with gradient consistency verification. This significantly improves the efficiency and reliability of large-scale model training in environments with domestic CPUs, computing cards, and RoCE networks, effectively overcoming the collaborative challenges faced by traditional methods on heterogeneous domestic IT innovation platforms, and solving problems such as low communication efficiency, rigid resource scheduling, and poor fault tolerance in distributed training in heterogeneous domestic IT innovation environments.

[0053] 2. This invention, through the domestic IT innovation hardware abstraction layer, for the first time realizes the ability to deploy a unified distributed training framework in a heterogeneous environment across multiple domestic computing cards, effectively reducing the system's dependence on the technology stacks of different hardware manufacturers, enabling the entire system framework to evolve smoothly with the evolution of the domestic IT innovation ecosystem, and possessing unprecedented adaptability and lifecycle.

[0054] 3. This invention ensures that the output model of a long-term training task is mathematically reliable even in a heterogeneous environment where faults occur through precise consistency verification. This is crucial for training large models that often consume millions of computing resources. Attached Figure Description

[0055] Appendix Figure 1 This is an implementation architecture diagram of the system described in this invention;

[0056] Appendix Figure 2 This is a schematic diagram illustrating the gradient synchronization process of the system described in this invention, which follows the path of "prioritizing intra-node aggregation, followed by global synchronization between nodes".

[0057] Appendix Figure 3 This is a flowchart illustrating the operation of the adaptive task scheduling module of the system described in this invention after the distributed training task is submitted and before it is officially started.

[0058] Appendix Figure 4 This is a flowchart of the node monitoring and gradient consistency verification operation of the system described in this invention;

[0059] Appendix Figure 5 This is a flowchart of the method of the present invention. Detailed Implementation

[0060] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0061] Example 1:

[0062] Reference Appendix Figure 1 This embodiment proposes a multi-machine, multi-card distributed training optimization system for heterogeneous environments of domestic IT innovation. The system includes: an application layer, a core system layer, a domestic IT innovation hardware abstraction layer, and a hardware resource layer.

[0063] (i) Application layer, used to receive distributed training tasks and related information submitted by users and send them down to the core system layer.

[0064] The training task-related information specifically includes the model, dataset, hyperparameters, and required computing resources specified in the training task.

[0065] (II) The core system layer comprises three major functional modules: adaptive task scheduling, hybrid communication topology optimization, and dynamic checkpointing and resilient recovery control.

[0066] A) The adaptive task scheduling module is responsible for the entire lifecycle management of the distributed training task, specifically including:

[0067] ① Based on the distributed training task and related information, the adaptive task scheduling module calls the unified device management interface provided by the domestic IT hardware abstraction layer to obtain a list of available computing devices;

[0068] ② After submitting the distributed training task but before officially starting it, refer to the appendix. Figure 3Specifically, perform the following operations:

[0069] (A1) The adaptive task scheduling module calls the performance data benchmark interface provided by the domestic IT hardware abstraction layer to issue a set of standardized micro-benchmark testing tools to the computing nodes of each identified available computing device. The tools obtain multi-dimensional performance data of each computing node by simulating the computing and communication modes in real model training (e.g., testing the GEMM operation performance under different matrix sizes, the point-to-point communication bandwidth between nodes, etc.), and return it to the adaptive task scheduling module through the performance data benchmark interface. The adaptive task scheduling module summarizes and analyzes the returned multi-dimensional performance data of each computing node, and then generates a multi-dimensional vector performance profile of each computing node.

[0070] (A2) Based on training task-related information, performance profiles of each computing node and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis. During the simulation analysis, the computation time, communication time and synchronization overhead that may be caused by non-uniform batches are comprehensively considered to generate the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and issue execution instructions.

[0071] ③ After the distributed training officially starts, the optimal task allocation and resource scheduling scheme is dynamically adjusted based on the detected computing node failures, and adjustment instructions are issued.

[0072] B) The hybrid communication topology optimization module is responsible for responding to the execution instructions of the optimal task allocation and resource scheduling scheme, and specifically performs the following operations:

[0073] (B1) Call the unified device management interface provided by the IT innovation hardware abstraction layer to first obtain the list of available computing devices, filter the target computing devices from it, and then detect the topology of the computing nodes to which each target computing device belongs, and clarify the network connection relationship between different types of computing cards (such as Ascend NPU, Hygon DCU) and computing nodes within the computing nodes.

[0074] (B2) Based on the topology of the computing nodes to which each target computing device belongs, a hierarchical communication strategy and gradient synchronization path are constructed to limit the core communication traffic to the internal bus of the high-speed node, reduce the access pressure and data volume of the RoCE network between nodes, and reduce communication latency.

[0075] (B3) Call the unified communication interface of the IT innovation hardware abstraction layer, load and initialize the optimal communication backend for each target computing device in a dynamic selection manner, and synchronize the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

[0076] The entire gradient synchronization process (such as All-Reduce) follows a path of "prioritizing intra-node aggregation, followed by global synchronization between nodes," as specifically demonstrated below:

[0077] a) Intra-node aggregation: Call the unified communication interface provided by the domestic IT hardware abstraction layer to dynamically select and initialize the optimal dedicated communication backend for different types of computing cards in the computing node (for example, call HCCL for Ascend NPU and call its CLR for Hygon DCU) to complete the first gradient aggregation in the computing node.

[0078] b) Aggregation between computing nodes: The adaptive task scheduling module dynamically elects a leader based on the computing power level and network status of each computing node and constructs a logical communication super node; subsequently, the hybrid communication topology optimization module performs global synchronization of the aggregated gradient data within each computing node between the logical communication super nodes based on the All-Reduce gradient synchronization protocol.

[0079] Reference Appendix Figure 1 , 2 Taking a hardware resource layer comprising compute node 1, compute node 2, and a RoCE network switch as an example, the optimal dedicated communication backend is dynamically selected and initialized for different types of compute cards within the compute nodes.

[0080] Node 1 is equipped with four Ascend NPUs (NPU0, NPU1, NPU2, and NPU3), using a group communication topology: NPU0 and NPU1 are in one group, and NPU2 and NPU3 are in another group. The NPUs in the group achieve bidirectional high-speed interconnection through the HCCS (High-Speed ​​CPU-GPU / GPU-GPUInterconnect) interface to meet the low-latency data transmission requirements within the group.

[0081] Node 2 is equipped with four Hygon DCUs (numbered DCU0, DCU1, DCU2 and DCU3), which adopt a group communication topology: DCU0 and DCU1 are in one group, and DCU2 and DCU3 are in another group. The DCUs in the group are interconnected bidirectionally at high speed through the X-Link interface to ensure high bandwidth and low latency for data interaction within the group.

[0082] At the cross-node communication level, Node 1 and Node 2 establish a bidirectional data communication link through the RoCE (RDMA over Converged Ethernet) network protocol, enabling cross-heterogeneous node collaborative computing and gradient synchronization between the Ascend NPU group of Node 1 and the Hygon DCU group of Node 2, thus balancing the efficiency and stability of cross-node communication in the context of domestic IT innovation.

[0083] C) The dynamic checkpoint and elastic recovery control module is responsible for periodically saving the complete training context and achieving reliable recovery of the training process based on the gradient consistency verification mechanism. This process specifically includes the following operations:

[0084] (C1) Lightweight full-state asynchronous checkpoint operation: ① Training initialization configuration: When starting the distributed training task, preset the checkpoint saving period (e.g., every N batches / hour), shared storage path (specify the domestic distributed file system) and differential compression parameters; ② Periodic asynchronous checkpoint saving: During training, asynchronously collect the current complete training context (including model parameters, optimizer state, learning rate, random number seed, etc.) at preset periods, reduce the data volume through differential compression technology, and write it to shared storage;

[0085] (C2) Reference Appendix Figure 4 Node monitoring and gradient consistency verification operations: ① Node fault monitoring: Start a background monitoring process to monitor the running status of each computing node in the hardware resource layer in real time. Once a computing node fault is detected (such as disconnection or abnormal computing power), an alarm is immediately sent to the adaptive task scheduling module, and the training pause mechanism is triggered; ② Task reassignment after fault: After training is paused, the adaptive task scheduling module uses the unified device management interface provided by the domestic IT hardware abstraction layer to identify healthy nodes in the hardware resource layer. Based on the remaining computing power of the healthy nodes, it dynamically adjusts the optimal task allocation and resource scheduling scheme and issues adjustment instructions to migrate the tasks of the faulty nodes to the healthy nodes; ③ Loading the latest checkpoint: On the healthy nodes that have received the adjustment instructions and completed task allocation, the elastic recovery control unit uniformly loads the latest complete checkpoint saved in the shared storage. ④ Gradient Consistency Verification: Extract one or more batches of training data corresponding to the checkpoint saving time, re-execute forward and backward propagation calculations on the healthy node to obtain new gradients; ⑤ Gradient Comparison and Judgment: Verify the consistency between the recalculated new gradients and the old gradients saved in the checkpoint by calculating the cosine similarity or norm ratio; ⑧ Resume Training or Retry: If the consistency verification passes, the adaptive task scheduling module synchronously restores the state of the hybrid communication topology optimization module, and the healthy node uses the current learning rate and random number seed to execute the subsequent training process; if the consistency verification fails, it rolls back to an earlier checkpoint or triggers an alarm. After ensuring the mathematical continuity of the training process, the adaptive task scheduling module reissues the training resumption command to start training.

[0086] It should be added that, in step ⑤, during the gradient comparison judgment process, when performing consistency verification by calculating the cosine similarity or norm ratio, the verification threshold is a dynamic threshold.

[0087] First, based on one or more factors such as the mean norm of the recent gradient tensor, gradient variance, and the current epoch or step of training, the threshold is dynamically calculated according to a predefined strategy (the threshold is appropriately relaxed in the early stage of training and more stringent when the training is close to convergence in the later stage).

[0088] Consistency is then determined using the following rules:

[0089] a) If the calculated cosine similarity is greater than or equal to the dynamic threshold (the closer the similarity is to 1, the more consistent the gradient is), or the norm ratio is within the range of [1-dynamic threshold, 1+dynamic threshold] (the closer the ratio is to 1, the smaller the gradient difference is), then it is determined that the consistency requirement is met.

[0090] b) If the calculated cosine similarity is less than the dynamic threshold (the closer the similarity is to 0, the more inconsistent the gradients are), and the norm ratio is outside the range of [1-dynamic threshold, 1+dynamic threshold] (the farther the ratio deviates from 1, the greater the gradient difference is), then it is determined that the consistency requirement is not met.

[0091] (III) The IT innovation hardware abstraction layer is used to provide a unified set of device management, memory management, communication interface and performance data benchmark interface, and to provide the core system layer with standardized access to underlying resources and performance data acquisition capabilities.

[0092] Specifically, the IT innovation hardware abstraction layer has the following functions:

[0093] (1) Unified device management interface: It has a unified device discovery and initialization function, which is used to automatically scan and identify available computing devices (such as Ascend NPU and Hygon DCU) in each computing node in the hardware resource layer based on distributed training tasks and related training tasks. It shields the API differences of different manufacturers in ID recognition and memory query, and provides a consistent list of available computing devices for the performance profile and task allocation of the core system layer.

[0094] (2) Unified memory management interface: used to provide a unified memory allocation, release and data transfer interface across computing devices, shielding the differences in memory operation between computing devices with different architectures;

[0095] (3) Unified communication interface: It has a unified communication context management function, which is used to encapsulate the initialization and context creation process of dedicated communication libraries for different architecture computing cards (Ascend's HCCL and Hygon's CLR); the hybrid communication topology optimization module will be based on this encapsulation and dynamically select and load and initialize the optimal communication backend for each identified target computing device.

[0096] (4) Performance data benchmark interface: Provides a standardized performance data acquisition interface for the adaptive task scheduling module of the core system layer, enabling it to trigger and obtain benchmark test results of available computing devices in different computing nodes in a unified manner.

[0097] (iv) Hardware resource layer, which includes multiple heterogeneous computing nodes with different processor architectures and computing card configurations, as well as RoCE network switches for high-speed data interaction between computing nodes. It is the underlying hardware carrier for resource management and interface adaptation of the IT innovation hardware abstraction layer.

[0098] Reference Appendix Figure 1 , 2 Taking the hardware resource layer, which includes compute node 1, compute node 2, and RoCE network switch, as an example.

[0099] The hardware configuration of computing node 1 is as follows: it is equipped with a Kunpeng series central processing unit (CPU) and four Ascend series neural network processors (NPUs), forming a heterogeneous computing power architecture of "CPU + multiple NPUs", which can meet the parallel computing needs of distributed training in the information technology innovation environment.

[0100] The hardware configuration of compute node 2 is as follows: it is equipped with Phytium series central processing unit (CPU) and four Hygon series data center processors (DCU), forming a heterogeneous computing architecture of "CPU + multiple DCU", which can meet the parallel computing needs of distributed training in the domestic IT innovation environment.

[0101] The hardware configuration of the RoCE network switch is as follows: it adopts Remote Direct Memory Access over Converged Ethernet (RoCE) technology and is equipped with flow control modules such as PFC / ECN to form a distributed communication support architecture of "low latency data forwarding + high reliability traffic scheduling", which can meet the cross-node high-speed interconnection needs of distributed training in the heterogeneous environment of information technology innovation.

[0102] Example 2:

[0103] Reference Appendix Figure 5 This embodiment proposes a multi-machine, multi-card distributed training optimization method for heterogeneous environments in the field of domestic IT innovation. Based on the system described in the first aspect, it specifically includes the following operation process:

[0104] S1. The application layer receives the distributed training task and related information submitted by the user and sends it down to the core system layer.

[0105] Training task-related information includes the model, dataset, hyperparameters, and required computational resources specified for the training task.

[0106] S2. After receiving the training task and related information, the core system layer first calls the unified device management interface provided by the domestic IT hardware abstraction layer to obtain a list of available computing devices. Then, it tests the computing nodes to which the available computing devices in the list belong, generating a multi-dimensional vector performance profile for each computing node. The specific description is as follows:

[0107] S2.1 The adaptive task scheduling module first calls the unified device management interface provided by the domestic IT hardware abstraction layer to automatically scan and identify available computing devices (such as Ascend NPU and Hygon DCU) in each computing node in the hardware resource layer, shielding the API differences in ID recognition and memory query of devices from different manufacturers, and providing a consistent list of available computing devices for the performance profile and task allocation of the core system layer.

[0108] S2.2 The adaptive task scheduling module then calls the performance data benchmark interface provided by the domestic IT hardware abstraction layer to issue a set of standardized micro-benchmark testing tools to the computing nodes of each identified available computing device. These tools simulate the computing and communication modes in real model training (e.g., testing the GEMM operation performance under different matrix sizes, the bandwidth of point-to-point communication between nodes, etc.) to obtain multi-dimensional performance data of each computing node, and return it to the adaptive task scheduling module through the performance data benchmark interface. The adaptive task scheduling module summarizes and analyzes the returned multi-dimensional performance data of each computing node, and then generates a multi-dimensional vector performance profile of each computing node.

[0109] S3. Based on training task-related information, performance profiles of each computing node, and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis. During the simulation analysis, the computation time, communication time, and synchronization overhead that may be caused by non-uniform batches are comprehensively considered to generate the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and then issue execution instructions.

[0110] S4. The hybrid communication topology optimization module of the core system layer responds to the execution instructions of the optimal task allocation and resource scheduling scheme, calls the unified device management interface provided by the domestic IT hardware abstraction layer, detects the topology of the computing nodes to which each target computing device belongs in the hardware resource layer, and constructs a hierarchical communication strategy and gradient synchronization path; this process is described in detail below:

[0111] S4.1 Call the unified device management interface provided by the domestic IT hardware abstraction layer to first obtain the list of available computing devices, filter the target computing devices from it, and then detect the topology of the computing nodes to which each target computing device belongs, and clarify the network connection relationship between different types of computing cards (such as Ascend NPU, Hygon DCU) and computing nodes within the computing nodes.

[0112] S4.2 Based on the topology of the computing nodes to which each target computing device belongs, a hierarchical communication strategy and gradient synchronization path are constructed to limit the core communication traffic to the internal bus of the high-speed node, reduce the access pressure and data volume of the RoCE network between nodes, and reduce communication latency.

[0113] S5. Next, the hybrid communication topology optimization module calls the unified communication interface of the domestic innovation hardware abstraction layer to dynamically select and load and initialize the optimal communication backend for each identified target computing device (such as initializing HCCL for Ascend), and synchronizes the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

[0114] S6. Each target computing device begins executing the training task. In the gradient synchronization phase of each iteration, the hierarchical communication strategy and gradient synchronization path constructed by the hybrid communication topology optimization module are strictly followed to execute gradient synchronization operations (such as All-Reduce). The entire gradient synchronization (such as All-Reduce) process follows the path of "intra-node aggregation first, then global synchronization between nodes," as described in detail below:

[0115] a) Intra-node aggregation: Call the unified communication interface provided by the domestic IT hardware abstraction layer to dynamically select and initialize the optimal dedicated communication backend for different types of computing cards in the computing node (for example, call HCCL for Ascend NPU and call its CLR for Hygon DCU) to complete the first gradient aggregation in the computing node.

[0116] b) Aggregation between computing nodes: The adaptive task scheduling module dynamically elects a leader based on the computing power level and network status of each computing node and constructs a logical communication super node; subsequently, the hybrid communication topology optimization module performs global synchronization of the aggregated gradient data within each computing node between the logical communication super nodes based on the All-Reduce gradient synchronization protocol.

[0117] S7. During training, the background monitoring process continuously monitors the running status of each computing node in the hardware resource layer. The dynamic checkpoint and elastic recovery control module periodically saves the complete training context, including: ① Training initialization configuration: When starting the distributed training task, the checkpoint saving period (e.g., every N batches / hour), shared storage path (specifying the domestic distributed file system), and differential compression parameters are preset; ② Periodic asynchronous checkpoint saving: During training, the current complete training context (including model parameters, optimizer status, learning rate, random number seed, etc.) is asynchronously collected at a preset period, and after reducing the data volume through differential compression technology, it is written to shared storage;

[0118] When a computing node failure is detected, the adaptive task scheduling module dynamically adjusts the task allocation and resource scheduling scheme and issues adjustment instructions. The hybrid communication topology optimization module responds to the instructions, synchronously updating the hierarchical communication strategy and gradient synchronization path. Finally, based on the gradient consistency verification mechanism, the training process is reliably restored, ensuring training continuity. This process is described as follows: ① Node failure monitoring: A background monitoring process is started to monitor the running status of each computing node in the hardware resource layer in real time. Once a computing node failure is detected (such as disconnection or abnormal computing power), an alarm is immediately sent to the adaptive task scheduling module, triggering the training pause mechanism; ② Task reassignment after failure: After training is paused, the adaptive task scheduling module uses the unified device management interface provided by the domestic IT hardware abstraction layer to identify healthy nodes in the hardware resource layer. Based on the remaining computing power of the healthy nodes, the optimal task allocation and resource scheduling scheme is dynamically adjusted, and adjustment instructions are issued to migrate the tasks of the failed nodes to the healthy nodes; ③ Loading the latest checkpoint: On the healthy nodes that have received the adjustment instructions and completed task allocation, the elastic recovery control unit loads the latest complete checkpoint saved in the shared storage to resume training. ④ Gradient Consistency Verification: Extract one or more batches of training data corresponding to the checkpoint saving time, re-execute forward and backward propagation calculations on the healthy node to obtain new gradients; ⑤ Gradient Comparison and Judgment: Verify the consistency between the recalculated new gradients and the old gradients saved in the checkpoint by calculating the cosine similarity or norm ratio; ⑧ Resume Training or Retry: If the consistency verification passes, the adaptive task scheduling module synchronously restores the state of the hybrid communication topology optimization module, and the healthy node uses the current learning rate and random number seed to execute the subsequent training process; if the consistency verification fails, it rolls back to an earlier checkpoint or triggers an alarm. After ensuring the mathematical continuity of the training process, the adaptive task scheduling module reissues the training resumption command to start training.

[0119] It should be added that when performing operation ⑤, during gradient comparison and consistency verification by calculating cosine similarity or norm ratio, the verification threshold is a dynamic threshold.

[0120] First, based on one or more factors such as the mean norm of the recent gradient tensor, gradient variance, and the current epoch or step of training, the threshold is dynamically calculated according to a predefined strategy (the threshold is appropriately relaxed in the early stage of training and more stringent when the training is close to convergence in the later stage).

[0121] Consistency is then determined using the following rules:

[0122] a) If the calculated cosine similarity is greater than or equal to the dynamic threshold (the closer the similarity is to 1, the more consistent the gradient is), or the norm ratio is within the range of [1-dynamic threshold, 1+dynamic threshold] (the closer the ratio is to 1, the smaller the gradient difference is), then it is determined that the consistency requirement is met.

[0123] b) If the calculated cosine similarity is less than the dynamic threshold (the closer the similarity is to 0, the more inconsistent the gradients are), and the norm ratio is outside the range of [1-dynamic threshold, 1+dynamic threshold] (the farther the ratio deviates from 1, the greater the gradient difference is), then it is determined that the consistency requirement is not met.

[0124] In summary, the multi-machine, multi-card distributed training optimization system and method for heterogeneous environments in the domestic IT innovation field, as proposed in this invention, significantly improves the efficiency and reliability of large-scale model training in environments with domestic CPUs, computing cards, and RoCE networks. This is achieved through a hybrid communication topology optimization module that constructs hierarchical communication strategies and gradient synchronization paths based on node topology; an adaptive task scheduling module that performs non-uniform task allocation based on benchmark profiles of domestic chips; and a dynamic checkpoint and elastic recovery control module that combines gradient consistency verification to achieve rapid fault migration. This effectively overcomes the collaborative challenges faced by traditional methods on heterogeneous domestic IT innovation platforms and solves problems such as low communication efficiency, rigid resource scheduling, and poor fault tolerance in distributed training within these environments.

[0125] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of domestic IT innovation, characterized in that, The system includes: The application layer is used to receive distributed training tasks and related training task information submitted by users and send them down to the core system layer. The core system layer comprises three main functional modules: adaptive task scheduling, hybrid communication topology optimization, and dynamic checkpointing and resilient recovery control. A) The adaptive task scheduling module is responsible for the full lifecycle management of the distributed training task, specifically including: ① Based on the distributed training task and related information, calling the unified device management interface provided by the domestic IT hardware abstraction layer to obtain a list of available computing devices; ② After the distributed training task is submitted but before it is officially started, testing the computing nodes to which the available computing devices belong, generating a performance profile in the form of a multi-dimensional vector; calling the built-in lightweight cost model to perform simulation analysis, generating the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and issuing execution instructions; ③ After the distributed training is officially started, dynamically adjusting the optimal task allocation and resource scheduling scheme based on the detected computing node failures, and issuing adjustment instructions. B) The hybrid communication topology optimization module is responsible for responding to the execution instructions of the optimal task allocation and resource scheduling scheme. First, it calls the unified interface provided by the domestic IT innovation hardware abstraction layer to detect the topology of the computing nodes of each target computing device in the hardware resource layer and construct a hierarchical communication strategy and gradient synchronization path. Then, it calls the unified communication interface of the domestic IT innovation hardware abstraction layer to load and initialize the optimal communication backend for each target computing device in a dynamic selection manner, and synchronizes the constructed hierarchical communication strategy and gradient synchronization path to each target computing device. C) The dynamic checkpoint and elastic recovery control module is responsible for periodically saving the complete training context and realizing reliable recovery of the training process based on the gradient consistency verification mechanism; The IT innovation hardware abstraction layer provides a unified set of device management, memory management, communication interfaces and performance data benchmark interfaces, providing the core system layer with standardized access to underlying resources and performance data acquisition capabilities; The hardware resource layer includes multiple heterogeneous computing nodes with different processor architectures and computing card configurations, as well as RoCE network switches for high-speed data interaction between computing nodes. It is the underlying hardware carrier for resource management and interface adaptation of the domestic IT innovation hardware abstraction layer.

2. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 1, is characterized in that... The training task-related information includes the model, dataset, hyperparameters, and required computing resources specified in the training task.

3. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 1, is characterized in that... The aforementioned IT innovation hardware abstraction layer has the following functions: (1) Unified device management interface: It has a unified device discovery and initialization function, which is used to automatically scan and identify available computing devices in each computing node in the hardware resource layer based on distributed training tasks and related training task information, shielding the API differences in ID recognition and memory query of devices from different manufacturers, and providing a consistent list of available computing devices for performance profiling and task allocation of the core system layer. (2) Unified memory management interface: used to provide a unified memory allocation, release and data transfer interface across computing devices, shielding the differences in memory operation between computing devices with different architectures; (3) Unified communication interface: It has a unified communication context management function, which is used to encapsulate the initialization and context creation process of dedicated communication libraries for different architecture computing cards; the hybrid communication topology optimization module will be based on this encapsulation and dynamically select and load and initialize the optimal communication backend for each identified target computing device. (4) Performance data benchmark interface: Provides a standardized performance data acquisition interface for the adaptive task scheduling module of the core system layer, enabling it to trigger and obtain benchmark test results of available computing devices in different computing nodes in a unified manner.

4. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 3, is characterized in that... Before the training task officially starts, the adaptive task scheduling module performs the following operations: (A1) The adaptive task scheduling module calls the performance data benchmark interface provided by the domestic innovation hardware abstraction layer and sends a set of standardized micro benchmark testing tools to the computing nodes of each identified available computing device. The tool obtains multi-dimensional performance data of each computing node by simulating the computing and communication mode in real model training and returns it to the adaptive task scheduling module through the performance data benchmark interface. The adaptive task scheduling module summarizes and analyzes the multi-dimensional performance data returned by each computing node, and then generates a multi-dimensional vector performance profile for each computing node. (A2) Based on training task-related information, performance profiles of each computing node and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis. During the simulation analysis, the computation time, communication time and synchronization overhead that may be caused by non-uniform batches are comprehensively considered to generate the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and then issue execution instructions.

5. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 4, is characterized in that... The hybrid communication topology optimization module responds to the execution command of the optimal task allocation and resource scheduling scheme, and specifically performs the following operations: (B1) Call the unified device management interface provided by the domestic IT hardware abstraction layer to first obtain the list of available computing devices, filter the target computing devices from it, and then detect the topology of the computing nodes to which each target computing device belongs, and clarify the network connection relationship between different types of computing cards in the computing nodes and between computing nodes. (B2) Based on the topology of the computing nodes to which each target computing device belongs, a hierarchical communication strategy and gradient synchronization path are constructed to limit the core communication traffic to the internal bus of the high-speed node, reduce the access pressure and data volume of the RoCE network between nodes, and reduce communication latency. (B3) Call the unified communication interface of the IT innovation hardware abstraction layer, load and initialize the optimal communication backend for each target computing device in a dynamic selection manner, and synchronize the constructed hierarchical communication strategy and gradient synchronization path to each target computing device.

6. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 5, is characterized in that... The hybrid communication topology optimization module synchronizes the constructed hierarchical communication strategy and gradient synchronization path to each target computing device. The entire gradient synchronization process follows the path of "intra-node aggregation first, then global synchronization between nodes", as shown below: a) Intra-node aggregation: Call the unified communication interface provided by the domestic IT hardware abstraction layer to dynamically select and initialize the optimal dedicated communication backend for different types of computing cards in the computing node, and complete the first gradient aggregation in the computing node. b) Aggregation among computing nodes: The adaptive task scheduling module dynamically elects a leader based on the computing power level and network status of each computing node, and constructs a logical communication super node; Subsequently, the hybrid communication topology optimization module, based on the All-Reduce gradient synchronization protocol, performs global synchronization of the aggregated gradient data within each computing node between logical communication supernodes.

7. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the field of information technology innovation, as described in claim 5, is characterized in that... The dynamic checkpoint and elastic recovery control module specifically performs the following operations: (C1) Lightweight full-state asynchronous checkpoint operation: ① Training initialization configuration: When starting a distributed training task, preset the checkpoint saving period, shared storage path and differential compression parameters; ② Periodic asynchronous saving of checkpoints: During training, the current complete training context is collected asynchronously at preset intervals, and after reducing the amount of data through differential compression technology, it is written to shared storage; (C2) Node Monitoring and Gradient Consistency Verification Operations: ① Node Fault Monitoring: Start the background monitoring process to monitor the running status of each computing node in the hardware resource layer in real time. Once a computing node fault is detected, immediately alert the adaptive task scheduling module and trigger the training pause mechanism; ② Task Reassignment After Fault: After training is paused, the adaptive task scheduling module uses the unified device management interface provided by the domestic IT hardware abstraction layer to identify healthy nodes in the hardware resource layer. Based on the remaining computing power of the healthy nodes, it dynamically adjusts the optimal task allocation and resource scheduling scheme and issues adjustment instructions to migrate the tasks of the faulty nodes to the healthy nodes; ③ Loading the Latest Checkpoint: On the healthy nodes that have received the adjustment instructions and completed task allocation... The elastic recovery control unit loads the latest complete checkpoints saved in the shared storage, restores the basic training context information, and performs gradient consistency verification; ④ Gradient consistency verification: extracts one or more batches of training data corresponding to the checkpoint saving time, re-executes forward and backward propagation calculations on the healthy node, and obtains new gradients; ⑤ Gradient comparison judgment: compares the recalculated new gradients with the old gradients saved in the checkpoints by calculating the cosine similarity or norm ratio for consistency verification; ⑧ Resume training or retry: if the consistency verification passes, the adaptive task scheduling module synchronously restores the state of the hybrid communication topology optimization module, and the healthy node uses the current learning rate and random number seed to execute the subsequent training process; If the consistency verification fails, the process will revert to an earlier checkpoint or trigger an alarm. Once the mathematical continuity of the training process is ensured, the adaptive task scheduling module will reissue the training recovery command and start training again.

8. The multi-machine, multi-card distributed training optimization system for heterogeneous environments in the context of domestic IT innovation, as described in claim 7, is characterized in that... During the gradient comparison process, when verifying consistency by calculating the cosine similarity or norm ratio, the verification threshold is a dynamic threshold. First, based on one or more factors such as the mean norm of the recent gradient tensor, the gradient variance, and the current epoch or step of training, the threshold is dynamically calculated according to a predefined strategy. Consistency is then determined using the following rules: a) If the calculated cosine similarity is greater than or equal to the dynamic threshold, or the norm ratio is within the range of [1-dynamic threshold, 1+dynamic threshold], then it is determined that the consistency requirement is met. b) If the calculated cosine similarity is less than the dynamic threshold and the norm ratio is outside the range of [1-dynamic threshold, 1+dynamic threshold], then it is determined that the consistency requirement is not met.

9. A distributed training optimization method for multi-machine, multi-card systems in a heterogeneous environment for domestic IT innovation, characterized in that... The system based on any one of claims 1-8 specifically includes the following operation process: S1. The application layer receives the distributed training task and related information submitted by the user and sends it down to the core system layer. S2. Based on the distributed training task and related information, the adaptive task scheduling module of the core system layer first calls the unified device management interface provided by the domestic IT hardware abstraction layer to obtain the list of available computing devices, and then tests the computing nodes to which the available computing devices in the list belong, generating a multi-dimensional vector performance profile of each computing node. S3. Based on training task-related information, performance profiles of each computing node, and real-time status, the adaptive task scheduling module calls the built-in lightweight cost model to conduct simulation analysis, generates the optimal task allocation and resource scheduling scheme with the goal of minimizing the time of a single training iteration, and issues execution instructions. S4. The hybrid communication topology optimization module of the core system layer responds to the execution instructions of the optimal task allocation and resource scheduling scheme, calls the unified device management interface provided by the domestic IT hardware abstraction layer, detects the topology of the computing nodes to which each target computing device belongs in the hardware resource layer, and constructs a hierarchical communication strategy and gradient synchronization path. S5. Next, the hybrid communication topology optimization module calls the unified communication interface of the domestic IT hardware abstraction layer to dynamically select and load and initialize the optimal communication backend for each identified target computing device, while synchronizing the constructed hierarchical communication strategy and gradient synchronization path to each target computing device. S6. Each target computing device begins to execute the training task. In the gradient synchronization stage of each iteration, the hierarchical communication strategy and gradient synchronization path constructed by the hybrid communication topology optimization module are strictly followed to execute the gradient synchronization operation. S7. During training, the background monitoring process continuously monitors the running status of each computing node in the hardware resource layer. The dynamic checkpoint and elastic recovery control module periodically saves the complete training context. When a computing node failure is detected, the adaptive task scheduling module dynamically adjusts the task allocation and resource scheduling scheme and issues adjustment instructions. The hybrid communication topology optimization module responds to the instructions and synchronously updates the hierarchical communication strategy and gradient synchronization path. Finally, the reliable recovery of the training process is achieved based on the gradient consistency verification mechanism to ensure training continuity.

Images