Heterogeneous mixed training cross-chip cooperative scheduling method and device

Abstract

This invention provides a cross-chip collaborative scheduling method and apparatus for heterogeneous hybrid training, relating to the field of computer technology. The cross-chip collaborative scheduling method for heterogeneous hybrid training includes: receiving a start request for a heterogeneous hybrid training task; creating replica specifications for each role based on the description information of each role; assembling the replica specifications corresponding to each role into a distributed training task object; instructing a container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calling a batch processing scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and starting the heterogeneous hybrid training task on the target heterogeneous computing power cluster. This invention can efficiently and reliably achieve hybrid parallel training of large-scale models in a heterogeneous computing power environment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a cross-chip collaborative scheduling method and apparatus for heterogeneous hybrid training. Background Technology

[0002] As the scale of AI model parameters reaches hundreds of billions to trillions, the computing power of a single chip is insufficient to meet training needs. Heterogeneous hybrid training has become a key path to aggregate the computing power of multiple chips and reduce reliance on the supply chain. However, achieving heterogeneous hybrid training requires full-stack collaboration of computing power scheduling, distributed training frameworks, and cross-chip communication libraries, and existing technologies have shortcomings at multiple levels.

[0003] Kubeflow Training Operator supports distributed tasks such as PyTorchJob, but assumes all ReplicaSpecs use the same chip type and image. Volcano provides Gang Scheduling to ensure simultaneous scheduling of Pod groups, but treats Pods within a group as homogeneous. Megatron-LM achieves training of hundreds of billions of models through TP / PP, but its parallel topology construction assumes homogeneous nodes. NCCL is incompatible with the protocols of various vendors' communication libraries (CNCL, MCCL, HCCL, etc.), and cross-chip communication relies on CPU-side Gloo relay, requiring D2H / H2D data copying, resulting in high latency and failing to meet the performance requirements of frequent gradient synchronization in large model training.

[0004] Therefore, how to efficiently and reliably achieve hybrid parallel training of large-scale models in a heterogeneous computing environment is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This invention provides a cross-chip collaborative scheduling method and apparatus for heterogeneous hybrid training, which addresses the aforementioned deficiencies in the prior art and enables efficient and reliable hybrid parallel training of large-scale models in a heterogeneous computing environment.

[0006] This invention provides a cross-chip collaborative scheduling method for heterogeneous hybrid training, comprising the following steps.

[0007] The system receives a startup request for a heterogeneous hybrid training task; wherein the startup request carries description information of multiple roles, each role being a resource grouping unit in the training task, and each role corresponding to a type of accelerator card; based on the description information of each role, it creates a replica specification for each role; it assembles the replica specifications corresponding to each role into a distributed training task object; it instructs the container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calls the batch processing scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and it starts the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

[0008] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided. The description information of a role includes the resource specifications and replica count of the accelerator card corresponding to the role, wherein the resource specifications of the accelerator card include the model of the accelerator card. The step of creating replica specifications for each role based on its description information includes: for each role's description information, performing the following operations: obtaining the mirror address corresponding to the accelerator card from the heterogeneous training group configuration information based on the accelerator card model; wherein the heterogeneous training group configuration information includes mapping relationships between various types of accelerator cards and mirror addresses; generating node affinity constraints based on the accelerator card model; wherein the key of the node affinity constraint is the accelerator card type, and the value is the accelerator card model; generating a resource list for the role based on the resource specifications; and obtaining the replica specifications for the role based on the mirror address corresponding to the accelerator card, the node affinity constraints, the resource list, and the replica count.

[0009] According to a cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the present invention, the step of initiating the heterogeneous hybrid training task in the target heterogeneous computing power cluster includes: obtaining the host name, number of slots, and accelerator card model of each container group; generating a host list file based on the host name, number of slots, and accelerator card model of each container group; initiating a training process in each container group according to the host list file and through the training framework of the heterogeneous hybrid training task; and performing the following operations for each container group: obtaining the library path and environment variables of the training process within the container group from the device type specification configuration based on the accelerator card model of the container group; wherein the device type specification configuration is used to store the library path and environment variables corresponding to different types of accelerator cards; injecting the library path and environment variables into the training process within the container group; and initiating the heterogeneous hybrid training task in the target heterogeneous computing power cluster based on the training process within each container group.

[0010] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided, wherein the heterogeneous hybrid training task is launched on the target heterogeneous computing power cluster based on the training processes within each container group, comprising: assigning logical rank to each training process according to the chip type corresponding to each training process and the order defined in the heterogeneous chip type configuration; constructing a process grid corresponding to each type of accelerator card according to the heterogeneous process grid configuration and the number of training processes corresponding to each type of accelerator card, and mapping the logical rank of each training process to the corresponding position in the process grid; wherein the process grid is a multi-dimensional topology structure for organizing parallel training processes; the heterogeneous process grid configuration is used to define the shape and dimension information of each process grid; according to the heterogeneous pipeline layer partitioning configuration and the process grid, the total number of model layers of the heterogeneous hybrid training task is proportionally allocated to each training process along the pipeline parallel dimension; and determining the tensor parallel configuration and / or context parallel configuration of each training process according to the position of each training process in the process grid.

[0011] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided, the method further includes: establishing a cross-mesh communication group between first adjacent training processes corresponding to the pipeline parallel boundary of adjacent process grids, the cross-mesh communication group being used to execute a tensor slicing or splicing scheme when the tensor parallel configurations of the adjacent process grids are different; and establishing a cross-chip data parallel process group between second adjacent training processes corresponding to the data parallel boundary of adjacent process grids, the cross-chip data parallel process group being used to synchronize the gradients of the second adjacent training processes.

[0012] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided, the method further includes: communication between training processes within a homogeneous communication domain via a homogeneous communication handle; wherein the homogeneous communication handle calls the native communication library of the accelerator card through a unified adapter interface, the adapter interface being a unified encapsulation of the native communication libraries of various types of accelerator cards; the homogeneous communication domain is a set of training processes using the same type of accelerator card; between different homogeneous communication domains, representative training processes of each homogeneous communication domain perform cross-domain communication via cross-chip communication handles; wherein the representative training process is the training process closest to the network card in the homogeneous communication domain; the cross-chip communication handle is an operation interface for data interaction between different types of accelerator cards.

[0013] According to a cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the present invention, the method further includes: in response to triggering a full reduction of gradient data across communication domains, performing the following operations: within each homogeneous communication domain, using the same-chip communication handle, performing a fragmentation reduction of the gradient data between each training process within the homogeneous communication domain; based on the fragmented data generated after the fragmentation reduction of the gradient data within each homogeneous communication domain, performing a global reduction of the gradient data between representative training processes in each homogeneous communication domain through the cross-chip communication handle; based on the fragmented data obtained from the global reduction, aggregating the globally reduced fragmented data into complete gradient data within each homogeneous communication domain using the same-chip communication handle.

[0014] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided, wherein each training process stores checkpoints according to the type of its corresponding accelerator card; the method further includes: in response to a failure of a first container group, determining the communication group to which the first container group belongs in the communication topology of the target heterogeneous computing power cluster, and obtaining the peer container groups within the communication group that have communication dependencies with the first container group; restarting the first container group and the peer container groups; for each training process in the restarted first container group and the peer container groups, obtaining a checkpoint shard corresponding to the accelerator card type of the training process; determining a global recovery point based on the minimum number of training steps in each checkpoint shard; and restoring the training task of each training process according to the checkpoint corresponding to the global recovery point.

[0015] According to the present invention, a cross-chip collaborative scheduling method for heterogeneous hybrid training is provided, the method further includes: obtaining the actual single-step time of the target training process at each step from the training log; and determining the heterogeneous training efficiency of the target training process based on the sum of the reciprocal of the actual single-step time and the reciprocal of the single-machine single-step baseline time corresponding to each accelerator card in the target heterogeneous computing power cluster.

[0016] The present invention also provides a cross-chip collaborative scheduling device for heterogeneous hybrid training, comprising: The system comprises the following modules: a receiving module for receiving a startup request for a heterogeneous hybrid training task; wherein the startup request carries descriptions of multiple roles, each role being a resource grouping unit within the training task, and each role corresponding to a type of accelerator card; a creation module for creating replica specifications for each role based on its descriptions; an assembly module for assembling the replica specifications corresponding to each role into a distributed training task object; a scheduling module for instructing a container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calling a batch scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and a startup module for starting the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cross-chip collaborative scheduling method for heterogeneous hybrid training as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-chip collaborative scheduling method for heterogeneous hybrid training as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-chip collaborative scheduling method for heterogeneous hybrid training as described above.

[0020] The cross-chip collaborative scheduling method and apparatus for heterogeneous hybrid training provided by this invention, upon receiving a start request for a heterogeneous hybrid training task, automatically creates replica specifications for each role based on the various role description information carried in the start request, and assembles the replica specifications of each role into a distributed training task object. The container orchestration platform creates container groups corresponding to each role based on this distributed training task object, and calls the batch processing scheduling engine to execute group scheduling strategies on the container groups of all roles, ensuring that the container groups of all roles are scheduled and started simultaneously. Through a unified task submission interface implemented by roles, this invention effectively reduces the complexity of accessing multiple accelerator cards in heterogeneous hybrid training tasks, achieves zero-manual configuration of multiple accelerator card replica specifications, and avoids the error-prone and inefficient problem caused by users having to query and manually configure the replica specifications of each accelerator card one by one. Meanwhile, by aggregating the replica specifications of various accelerator cards using distributed training task objects, the container scheduling platform can execute group scheduling strategies for container groups corresponding to various types of accelerator cards. This ensures that all container groups for all roles are successfully scheduled in the same batch before training starts, avoiding idle waiting of the ready training process. This effectively improves the reliability and determinism of heterogeneous mixed training task startup, and ultimately realizes unified lifecycle management of heterogeneous mixed training tasks, providing users with efficient and stable heterogeneous mixed training services. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the present invention.

[0023] Figure 2 This is a flowchart illustrating the cross-chip communication method for heterogeneous hybrid training provided by the present invention.

[0024] Figure 3 This is a flowchart illustrating the fault handling method for heterogeneous hybrid training provided by the present invention.

[0025] Figure 4 This is a flowchart illustrating the method for determining the training efficiency of heterogeneous hybrid training provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the C2C phased communication algorithm provided by the present invention.

[0027] Figure 6This is a schematic diagram of the architecture of the heterogeneous hybrid training system provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the cross-chip collaborative scheduling device for heterogeneous hybrid training provided by the present invention.

[0029] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] The following is combined Figures 1-4 This invention describes a cross-chip collaborative scheduling method for heterogeneous hybrid training.

[0032] Figure 1 This is a flowchart illustrating the cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the present invention. Figure 1 As shown, the method includes the following: Step 101: Receive the start request for the heterogeneous hybrid training task.

[0033] The launch request carries descriptions of various roles. A role is a resource grouping unit in the training task, and each role corresponds to a type of accelerator card.

[0034] A heterogeneous mixed training task is a task in which two or more AI acceleration chips with different architectures are used simultaneously to complete model training in the same training task.

[0035] In the specific implementation process, the system receives user-submitted heterogeneous hybrid training task start requests; it parses the start requests and extracts description information for multiple roles contained within them. The role description information defines the resource specifications and number of replicas corresponding to that role. Resource specifications may include, but are not limited to, information such as the model of the accelerator card, the number of accelerator cards, the number of CPU cores, and the memory capacity.

[0036] Step 102: Create the copy specifications for each character based on their description information.

[0037] In the actual implementation process, for each role's description information, the following operations are performed to create the replica specifications for each role: Based on the accelerator card model, obtain the corresponding mirror address of the accelerator card from the heterogeneous training group configuration information (HeteroGroup); based on the accelerator card model, generate node affinity constraints; based on the resource specifications, generate the resource list of the role; based on the mirror address of the accelerator card, node affinity constraints (NodeAffinity), resource list, and number of replicas, obtain the replica specifications of the role.

[0038] The heterogeneous training group configuration information includes mapping relationships between various types of accelerator cards and image addresses.

[0039] The key of the node affinity constraint is the accelerator card type, and the value is the accelerator card model.

[0040] As an example, the accelerator card model is extracted from the description information of each role. A full match is then performed in the heterogeneous training group configuration information to find the corresponding image address. The computing power scheduling platform automatically generates NodeAffinity rules, setting the Key to accelerator-type, Operator to In, and Values ​​to the accelerator card model for that role, ensuring that the container group (Pod) for that role can only be scheduled to nodes labeled "Model A". The computing power scheduling platform parses the CPU, memory, and accelerator card quantity from the description information of each role to generate a standard Kubernetes ResourceList. The accelerator card resource name can be obtained by querying the physical machine configuration table; for example, "gpu" is mapped to "nvidia.com / gpu". The obtained image address, the generated NodeAffinity constraints, the ResourceList, and the replica count from the description information of each role are assembled into a standard Kubernetes ReplicaSpec object, which serves as the final deployment configuration for that role.

[0041] Step 103: Assemble the corresponding copy specifications of each character into a distributed training task object.

[0042] In the specific implementation process, the replica specifications of all roles generated in step 102 are aggregated and assembled into a standard distributed training task object, such as a PyTorchJob object on Kubernetes. The distributed training task object is then submitted to the Kubernetes (K8s) API Server for storage.

[0043] Step 104: Instruct the container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and call the batch scheduling engine to execute the group scheduling strategy for the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to the nodes of the target heterogeneous computing power cluster.

[0044] In the specific implementation process, the container orchestration platform, such as Kubernetes, creates corresponding container groups based on the replica specifications of each role in the distributed training task object. Through a batch scheduling engine, such as Kubernetes' Volcano Scheduler, it executes group scheduling policies on all container groups corresponding to each role. For example, Gang Scheduling checks whether there is a combination of nodes in the target heterogeneous computing power cluster that meets the resource requirements of all roles. If resources are sufficient, all container groups corresponding to all roles are simultaneously scheduled to the nodes of the target heterogeneous computing power cluster. If resources are insufficient, all container groups remain in a waiting state until the resources meet the scheduling conditions.

[0045] Step 105: Start the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

[0046] In the specific implementation process, when all the container groups corresponding to all roles are in a running state, the master node (Master Pod) of the target heterogeneous computing power cluster begins to execute the training startup orchestration logic.

[0047] In some embodiments, the hostname, number of slots, and model of the accelerator card for each container group are obtained; a host list file is generated based on the hostname, number of slots, and model of the accelerator card for each container group; and the training process is started in each container group based on the host list file using the training framework of the heterogeneous hybrid training task.

[0048] For each container group, perform the following operations: based on the model of the accelerator card in the container group, obtain the library path and environment variables of the training process within the container group from the device type specification configuration (device_type_specific); based on the training process within each container group, start the heterogeneous hybrid training task in the target heterogeneous computing power cluster.

[0049] Device type specification configuration is used to store the library paths and environment variables corresponding to different types of accelerator cards; the library paths and environment variables are injected into the training process within the container group.

[0050] In the embodiments provided by this invention, after obtaining the host name, number of slots, and accelerator card model of each container group, a host list file is generated, and the training process is started in each container group through the training framework. Simultaneously, for each container group, the corresponding library path and environment variables are obtained from the device type specification configuration based on its accelerator card model, and injected into the training process within that container group. This enables the training process to have chip type awareness at startup, eliminating the need for hard-coding chip configuration in the script. Centralized management and automatic injection of accelerator card library paths and environment variables through device type specification configuration effectively improves the accuracy of the training process's adaptation to different accelerator card models and the startup efficiency, avoiding the problems of poor portability and high maintenance costs caused by hard-coding chip configuration in the script in related technologies. Furthermore, the use of a standardized annotation mechanism to transmit heterogeneous information allows the same startup process to seamlessly adapt to multiple accelerator card combinations, significantly improving the standardization and cross-platform portability of the heterogeneous mixed training startup process, ultimately providing users with an efficient, stable, and easy-to-maintain heterogeneous mixed training startup service.

[0051] As an example only, the above embodiment can be implemented through the following steps.

[0052] Step T1: After all container groups are successfully scheduled, the containers start. The container orchestration platform injects the environment variables PET_NODE_RANK (node ​​number) and AIRS_ACCELERATOR_COUNT (number of accelerator cards) into the container groups.

[0053] Step T2: The master node, i.e. the node with PET_NODE_RANK=0, obtains a list of all container group names using the CLI tool airsctl job podnames.

[0054] Step T3: The master node initiates an SSH connection probe for each container group and waits for the SSH service of all container groups to be ready.

[0055] Step T4: The master node logs into each container group via SSH, executes the accelerator card model detection command, and obtains the accelerator card model identifier.

[0056] Step T5: Dynamically generate a host list file based on the host name, number of slots, and accelerator card model of each container group, in the following format: {host name}slots={number of slots}type={accelerator card model}).

[0057] Step T6: Invoke the training framework, for example, call FlagScale's unified task startup script pythonrun.py, passing in the path to the host list file and the training configuration. SSHTrainRunner parses the host list file and starts the training process for each node via SSH.

[0058] In practice, the host list file can be parsed using the following regular expression operations.

[0059] ^(\S+)\s+slots=(\d+)(?:\s+type=(\S+))? ); Where (\S+) represents capture group 1: hostname; \s+ represents one or more whitespace characters; (\d+) represents capture group 2: slot number; (?:\s+type=(\S+))?) represents the optional non-capture group: including capture group 3, used to capture accelerator card model.

[0060] Step T7: The SSH-based distributed training launcher SSHTrainRunner obtains the environment variables, library path, and build directory of the accelerator card from the device type specification configuration according to the model of the accelerator card in each container group, injects them into the training process environment, and passes the hetero-current-device-type parameter.

[0061] In some embodiments, a logical rank is assigned to each training process based on the chip type corresponding to each training process and the order defined in the heterogeneous chip type configuration (hetero_device_types); a process mesh (ProcessMesh) corresponding to each type of accelerator card is constructed based on the heterogeneous process mesh configuration (hetero_process_meshes) and the number of training processes corresponding to each type of accelerator card, and the logical rank of each training process is mapped to the corresponding position in the process mesh; based on the heterogeneous pipeline layer split configuration (hetero_pipeline_layer_split) and the process mesh, the total number of model layers of the heterogeneous hybrid training task is proportionally allocated to the pipeline stages corresponding to each process mesh along the pipeline parallel dimension; and the tensor parallelism (TP) and / or context parallelism (CP) configuration of each training process is determined based on the position of each training process in the process mesh.

[0062] A process grid is a multi-dimensional topology used to organize parallel training processes. Each dimension in the process grid represents a parallel strategy. For example, [TP=4,CP=2,EP=1,DP=1,PP=2] means: 4-way tensor parallelism; 2-way context parallelism; 1-way expert parallelism; 1-way data parallelism; and 2-way pipeline parallelism.

[0063] The heterogeneous process grid configuration defines the shape and dimensional information of each process grid, i.e., the size of each dimension of the process grid and which dimension is TP (process parametric) and which is PP (process parametric). In practice, the heterogeneous process grid configuration is a multi-dimensional array, where each number represents the number of processes in the corresponding parallel dimension of the process grid. For example, for five parallel strategies: tensor / context / expert / data / pipeline, the heterogeneous process grid configuration is [1,1,1,8,1,1,1,1,8,1], where the two 8s represent the tensor parallelism of 8 for each of the two accelerator cards.

[0064] As an example only, the above embodiment can be implemented through the following steps: Step M1: After each training process starts, initialize the distributed environment and set the communication backend to "cpu:gloo,cuda:flagcx": processes on the CPU communicate using the Gloo library, and processes on CUDA communicate using the FlagCX library.

[0065] Step M2: The RankMapper collects all accelerator card models corresponding to the logical ranks through the set communication operation all_gather_object, sorts them according to the order defined in the heterogeneous chip type configuration, and establishes a mapping table from physical rank to logical rank physical_to_logical and from logical rank to physical rank logical_to_physical.

[0066] Step M3: Based on the heterogeneous process grid configuration, for example [1,1,1,8,1,1,1,1,8,1], construct a process grid for each accelerator card model. For each process grid, calculate the total logical rank TP×CP×EP×DP×PP for that accelerator card model: the product of the five parallel strategy tensors / context / expert / data / pipeline, the result of which equals the total logical rank within that process grid; assign the corresponding number of logical ranks, and construct a mapping from multidimensional coordinates to logical ranks.

[0067] Step M4: Based on the heterogeneous pipeline layer partitioning configuration, proportionally distribute the total number of model layers for the heterogeneous hybrid training task along the pipeline parallel dimension to the pipeline parallel dimension of each process grid. For example, distribute the total 54 layers of the model to two pipeline stages in a 28:26 ratio, with the first 28 layers going to the first stage and the last 26 layers going to the second stage.

[0068] Step M5: The ParallelContext establishes a cross-process grid communication group between training processes corresponding to the pipeline parallel boundaries of adjacent process grids. When the tensor parallel configurations of the source training process and the target training process are different, the tensor slicing / stitching scheme is automatically calculated: if the TP of the source training process is greater than the TP of the target training process, the tensor is sliced ​​according to the TP dimension when sending; if the TP of the source training process is less than the TP of the target training process, the tensor is stitched when receiving.

[0069] Step M6: Establish cross-accelerator card model data parallel process groups between the training processes corresponding to the data parallel boundaries of each process grid for gradient synchronization across accelerator card models.

[0070] In some embodiments, an inter-mesh communication group is established between first adjacent training processes corresponding to the pipeline parallel boundary of adjacent process grids. The inter-mesh communication group is used to execute tensor slicing or splicing schemes when the tensor parallel configurations of adjacent process grids are different. A cross-chip data parallel process group is established between second adjacent training processes corresponding to the data parallel boundary of adjacent process grids. The cross-chip data parallel process group is used to synchronize the gradients of the second adjacent training processes.

[0071] In the specific implementation process, taking a heterogeneous hybrid training task containing both Type A and Type B accelerator cards as an example, the host name, number of slots, and accelerator card model (such as Type A or Type B) of each container group are obtained, and a host list file is generated accordingly. Based on the accelerator card model of each container group, the corresponding library path and environment variables (such as configuring the flagcx communication backend for Type A accelerator cards) are obtained from the device type specification configuration and injected into the training process within the container group, thereby starting the heterogeneous hybrid training task in the target heterogeneous computing power cluster.

[0072] After initializing the distributed environment, all device types of all ranks can be collected via `all_gather_object` and sorted according to the order defined in the heterogeneous chip type configuration (`hetero_device_types`), establishing a mapping table from physical rank to logical rank and vice versa. Next, the system parses the heterogeneous process mesh configuration (`hetero_process_meshes`), for example, [1,1,1,8,1,1,1,1,8,1], constructing process meshes (`ProcessMesh`) for both Type A and Type B accelerator cards. This process mesh is a multi-dimensional topology used to organize parallel training processes. Each `ProcessMesh` calculates the total logical rank based on the product of tensor parallelism (TP), context parallelism (CP), expert parallelism (EP), data parallelism (DP), and pipelined parallelism (PP), and constructs a mapping from multi-dimensional coordinates to logical ranks.

[0073] Subsequently, based on the heterogeneous pipeline layer splitting configuration (hetero_pipeline_layer_split), for example, configured as [28,26], the total number of model layers is proportionally distributed to the PP dimension of each ProcessMesh. On this basis, ParallelContext establishes an inter-mesh communication group between the first adjacent training processes corresponding to the pipeline parallelism (PP) boundaries of adjacent ProcessMesh. When the tensor parallelism configurations of the source mesh and the target mesh are different, this inter-mesh communication group automatically calculates and executes a tensor slicing or splicing scheme: if the TP of the source mesh is greater than the TP of the target mesh, the tensor is sliced ​​according to the TP dimension during transmission; if the TP of the source mesh is less than the TP of the target mesh, the tensor is spliced ​​during reception. Furthermore, an inter-chip data parallel process group is established between the second adjacent training processes corresponding to the data parallelism (DP) boundaries of each ProcessMesh to synchronize the gradients of these second adjacent training processes during the backpropagation phase, ensuring consistent updates of global model parameters.

[0074] The configuration defines the order in which logical ranks are assigned to each training process, and a multi-dimensional process grid is constructed based on the heterogeneous process grid configuration and the number of training processes corresponding to each accelerator card, mapping the logical ranks to the corresponding positions in the process grid. At the same time, the total number of model layers is proportionally allocated to each training process along the pipeline parallel dimension according to the heterogeneous pipeline layer partitioning configuration, and its tensor parallel configuration and / or context parallel configuration are determined according to the position of each training process in the process grid, so that different accelerator card types can have different tensor parallelism and data parallelism.

[0075] Figure 2 This is a flowchart illustrating the cross-chip communication method for heterogeneous hybrid training provided by the present invention. Figure 2 As shown, the method includes the following: Step 201: Training processes within the same homogeneous communication domain communicate via the same chip communication handle.

[0076] The same-chip communication handle calls the accelerator card's native communication library through a unified adapter interface. This adapter interface is a unified encapsulation of the native communication libraries for various types of accelerator cards. The same-chip communication handle can call the corresponding accelerator card's native aggregate communication library, providing a standard aggregate communication call entry point for the training process. For example, the training process on accelerator card A calls allReduce on accelerator card A to complete gradient synchronization through the same-chip communication handle.

[0077] A homogeneous communication domain is a collection of training processes that use the same type of accelerator card. All training processes within a homogeneous communication domain have the same accelerator card architecture and can directly use the native communication library corresponding to that accelerator card for efficient communication. For example, in a heterogeneous mixed training task with 8 GPUs of model A, the training processes on these 8 GPUs form a homogeneous communication domain. Communication between training processes does not require any cross-chip conversion; they directly use the same-chip communication handle.

[0078] Step 202: Between different homogeneous communication domains, the representative training processes of each homogeneous communication domain conduct cross-domain communication through cross-chip communication handles.

[0079] The representative training process is the one closest to the network interface card (NIC) in the homogeneous communication domain. It is responsible for representing the entire cluster in data interaction with other chip clusters. In practice, topology detection can be performed on the communication topology of the target heterogeneous computing cluster, and the training process closest to the NIC in each homogeneous communication domain can be selected as the representative training process for that homogeneous communication domain.

[0080] Cross-chip communication handles are operational interfaces used for data exchange between different types of accelerator cards. In practice, cross-chip communication handles can establish P2P channels based on IB networks or sockets, enabling chip clusters with different architectures to send and receive data. For example, when a chip cluster of model A and a cluster of model B need to perform cross-domain gradient AllReduce, the representative nodes selected by each cluster send gradient fragment data to each other through the IB channel established by the cross-chip communication handle.

[0081] In some embodiments, in response to triggering a full reduction of gradient data across communication domains, the following operations are performed: within each homogeneous communication domain, gradient data fragmentation reduction is performed between training processes within the homogeneous communication domain using a chip-based communication handle; based on the fragmented data generated after gradient data fragmentation reduction within each homogeneous communication domain, global reduction of gradient data is performed between representative training processes in each homogeneous communication domain via a cross-chip communication handle; based on the fragmented data obtained from global reduction, the globally reduced fragmented data is aggregated into complete gradient data within each homogeneous communication domain using a chip-based communication handle.

[0082] The training framework calls `flagcxCommInitRank` to initialize the communication domain of the target heterogeneous computing cluster; FlagCX collects chip manufacturer information of the accelerator cards for all training processes in the target heterogeneous computing cluster through `bootstrap allgather`, and calls `flagcxCollectClusterInfos` to determine the communication domain type. Among these: flagcxCommInitRank is an interface in the FlagCX unified communication library used to initialize communication domains. It determines the communication domain type by collecting chip manufacturer information of the accelerator cards of all training processes, and creates intra-chip communication handle homo_comm and cross-chip communication handle hetero_comm for each training process.

[0083] FlagCX is a cross-chip unified communication library that encapsulates the differences in communication libraries from various chip manufacturers through an Adaptor plugin system to achieve high-performance aggregated communication between heterogeneous clusters.

[0084] Bootstrap Allgather is a basic communication operation performed by FlagCX when initializing a communication domain. It is used to collect the respective vendor identifiers between chips from different manufacturers to determine whether the current communication domain contains multiple types of accelerator cards.

[0085] flagcxCollectClusterInfos is a function in the FlagCX library used to determine the communication domain type based on the chip manufacturer information collected by bootstrapallgather, and initialize the corresponding native intra-cluster communication environment and cross-cluster network communication environment.

[0086] If the target heterogeneous computing cluster is determined to be a heterogeneous communication domain, three communication handles are initialized: a same-chip communication handle, which is created by calling the native communication library's commInitRank, providing a communication interface based on the accelerator card's native communication library for the training process within the homogeneous communication domain; a cross-chip communication handle, which calls flagcxHeteroCommInitRank to establish a P2P channel based on IB / Socket, providing an operation interface for data interaction between different homogeneous communication domains; and host_comm, an optional communication handle used as a Gloo fallback. Among these: commInitRank is a standard interface in the communication library used to initialize the communication environment of a single process in distributed training. It is responsible for establishing the basic communication context for the process to exchange data with other processes.

[0087] flagcxHeteroCommInitRank is an interface in the FlagCX unified communication library specifically used to initialize cross-chip communication environments. It establishes P2P transmission channels based on InfiniBand or Socket between accelerator cards of different architectures to support C2C phased communication algorithms.

[0088] Gloo fallback is a backup communication mechanism in the FlagCX communication library. When the native communication library or RDMA network on the device side fails, it automatically degrades to use the CPU and TCP-based Gloo backend to ensure communication reliability and ensure that training tasks are not interrupted due to transient anomalies.

[0089] Topology detection is enabled. The distance from each training process to the gateway NIC is detected using `flagcxTopoGetServerTopo`. For each chip cluster, the training process closest to the gateway NIC is selected as the representative training process. This representative training process is the one closest to the network card in the homogeneous communication domain and is responsible for representing the entire cluster in data interaction with other chip clusters. The representative training processes of each cluster form a cross-cluster communication representative list. Among them: flagcxTopoGetServerTopo is an interface function provided by the topology detection module in the FlagCX unified communication library. It is used to automatically detect the hardware topology information inside the server, including accelerator cards, network cards, CPUs, PCIe switches, and PCI links between devices, and generate a hierarchical topology. At the same time, by calculating the distance between the accelerator cards and network cards, it intelligently matches the optimal network card for each training process to improve cross-node communication efficiency and bandwidth utilization.

[0090] During training, when calling collection operations such as flagcxAllReduce, FlagCX determines the communication domain type: if it's a homogeneous communication domain, it directly calls the native communication library to execute the collection operation through the same chip's communication handle; if it's a heterogeneous communication domain, it enters the cross-communication domain communication path. Specifically: flagcxAllReduce is the core collective communication interface provided by the FlagCX unified communication library, used to perform full reduction operations in distributed training. It automatically selects the execution path according to the communication domain type: in the homogeneous communication domain, it calls the accelerator card's native communication library to complete the reduction; in the heterogeneous communication domain, it coordinates the training processes of different chip types through a C2C phased algorithm to complete global gradient reduction and broadcasting.

[0091] In cross-communication domain communication paths, C2cPlanner calculates a hash value based on communication mode parameters, including data size, root cluster identifier (rootClusterId), communication operation (commOp), and reduction operation (redOp), and queries the LRU cache. If the cache is hit, the existing communication plan is reused; otherwise, a new plan is generated and written to the cache. Specifically: C2cPlanner is a planner component in the FlagCX unified communication library responsible for orchestrating C2C heterogeneous aggregate communication algorithms. It combines the distance information between the accelerator card and the network card obtained by automatic topology detection, selects the training process closest to the network card to participate in cross-chip communication, and orchestrates the corresponding cross-chip heterogeneous aggregate communication algorithm according to the specific network card configuration and cluster topology.

[0092] C2cPlanner performs four-stage communication to achieve full reduction of gradient data across communication domains: Within each homogeneous communication domain, gradient data fragmentation and reduction are performed between training processes using the same-chip communication handle, distributing the gradient data among the training processes; based on the fragmented data generated after fragmentation and reduction within each homogeneous communication domain, global reduction of gradient data is performed between representative training processes in each homogeneous communication domain via cross-chip communication handles, with representative training processes exchanging fragmented data and completing the reduction through an IB network established by cross-chip communication handles; based on the fragmented data obtained from global reduction, within each homogeneous communication domain, the globally reduced fragmented data is aggregated into complete gradient data using the same-chip communication handles, and the AllGather broadcast result is performed between training processes within the domain via the same-chip communication handles. Wherein: AllGather is a collection communication operation in distributed training, used to collect data fragments held by each training process and broadcast them to all participating processes, so that each process eventually obtains the complete aggregated data.

[0093] In the embodiments provided by this invention, a same-chip communication handle based on the native communication library of the accelerator card is initialized for each homogeneous communication domain. The rank closest to the network card is selected from each homogeneous communication domain as the representative training process, and a cross-chip communication handle is initialized. This allows training processes within the same homogeneous communication domain to achieve efficient communication through the same-chip communication handle, while cross-domain data interaction between different homogeneous communication domains is only performed by the representative training process through the cross-chip communication handle. This layered communication architecture of same-chip and cross-chip communication handles effectively improves the adaptation efficiency of the upper-layer training framework to heterogeneous chip communication, avoiding the high development costs and maintenance difficulties caused by the need for separate communication logic development for each chip in related technologies. Simultaneously, the performance overhead of cross-chip aggregate communication is minimized by utilizing cross-chip communication handles. Intra-cluster communication still uses the native high-performance communication library, with network transmission used only for cross-cluster communication. Furthermore, the cross-cluster communication path is further optimized through topology-aware representative node selection, significantly improving the communication efficiency and scalability of heterogeneous mixed training tasks, ultimately providing users with efficient, stable, and easily scalable heterogeneous mixed training communication services.

[0094] Figure 5 This paper demonstrates a four-stage collaborative process for efficient AllReduce communication in heterogeneous communication domains containing accelerator cards of type A and type B: Within the homogeneous communication domains of accelerator cards A and B, ReduceScatter operations are executed in parallel using the same chip communication handle to complete local gradient reduction and fragmentation; cross-cluster P2P transmission is performed via IB RDMA or Socket network across chip communication handles to exchange the reduction fragments held by the representative training processes in each homogeneous communication domain; each representative training process performs a global AllReduce operation through the cross-chip communication handle to obtain the complete global reduction result; within each homogeneous communication domain, AllGather operation is executed again using the same chip communication handle to broadcast and distribute the global reduction result to all training processes in the domain, ultimately ensuring that all participating Rank processes obtain consistent and complete gradient update data.

[0095] Figure 3 This is a flowchart illustrating the fault handling method for heterogeneous hybrid training provided by the present invention. Figure 3 As shown, the method includes the following: Step 301: In response to the failure of the first container group, determine the communication group to which the first container group belongs in the communication topology of the target heterogeneous computing power cluster, and obtain the peer container groups within the communication group that have communication dependencies with the first container group.

[0096] A communication group is a logical set in a heterogeneous hybrid training communication topology, consisting of gradient synchronization and parameter update dependencies established by training processes using different types of accelerator cards through a communication library.

[0097] The communication topology describes the communication group to which each container group belongs and the gradient synchronization relationships between training processes corresponding to different types of accelerator cards within the communication group. When a training task starts, the training framework reports the communication topology of the target heterogeneous computing power cluster to the platform.

[0098] The first container group is any container group that has failed.

[0099] The peer container group is another container group that has a communication dependency with the first container group and needs to exchange gradient or parameter data with each other during training.

[0100] In practice, each training process stores checkpoint fragments in shared storage according to the type of its corresponding accelerator card. For example, the path is: {root path} / {task ID} / checkpoint / {chip type} / step_{number of steps} / ).

[0101] The computing power scheduling platform listens for container group status change events. When it detects that a container group has entered the Failed / Unknown state, it determines that the first container group has failed. It queries the type of the accelerator card and the communication group to which the first container belongs, traverses the communication topology to find all peer container groups in the same communication group and marks them as pending restart, and marks container groups not in the same communication group as keeping running.

[0102] Step 302: Restart the first container group and the peer container group.

[0103] In the specific implementation process, JobManager sends a rebuild request to the Kubernetes API Server, carrying the original ResourceList and replica configuration of the first container group; the Kubernetes scheduler allocates nodes with the same accelerator card model as before the failure to the first container group according to the physical machine configuration table; JobManager synchronously sends a rebuild request to the peer container group to the Kubernetes API Server to ensure that the peer container group and the first container group are scheduled in the same batch; the Volcano batch scheduling engine executes the Gang Scheduling policy on the first container group and all peer container groups, and enters the running state after all container groups are successfully scheduled at the same time.

[0104] Step 303: For each training process in the first container group and the peer container group after restarting, obtain the checkpoint shards corresponding to the accelerator card type of the training process.

[0105] In the specific implementation, each training process stores checkpoints according to the type of its corresponding accelerator card. For example, the training process on accelerator card A stores checkpoints in a native communication library compatible format corresponding to accelerator card A in local NVMe or shared storage, while the training process on accelerator card B stores checkpoints in a native communication library compatible format corresponding to accelerator card B in local storage. The JobManager traverses all training processes in the first container group and the peer container group after the restart, and reads the end checkpoint fragments from the corresponding storage path according to the accelerator card type of each training process. For the training process with accelerator card type A.com / acc, it reads the checkpoint fragments in the native communication library format of accelerator card A from its mounted PVC; for the training process with accelerator card type B.com / acc, it reads the checkpoint fragments in the native communication library format of accelerator card B from its mounted PVC.

[0106] Step 304: Determine the global recovery point based on the minimum number of training steps in each checkpoint slice.

[0107] The global recovery point is the step number with the smallest number of training steps among all checkpoint slices. It represents the earliest consistent state that all training processes can safely recover to, ensuring that there will be no inconsistency problem after recovery where some processes are ahead of others while others are behind.

[0108] In the specific implementation process, JobManager collects the end checkpoint fragments of each training process in the first container group and all peer container groups; JobManager parses the training step number field recorded in each checkpoint fragment; JobManager compares all training step numbers and selects the minimum value as the global recovery point; if the training step numbers of each checkpoint fragment are inconsistent, the checkpoint fragment corresponding to the minimum training step number is used as the benchmark.

[0109] Step 305: Restore the training tasks of each training process according to the checkpoints corresponding to the global recovery point.

[0110] The training framework re-executes the training iteration from the global recovery point, and continues normal training after completing parameter synchronization.

[0111] In the embodiments provided by this invention, after a failure occurs in the first container group, the communication group to which the container group belongs is determined based on the communication topology, and the peer container groups within the communication group that have communication dependencies with it are obtained. Only the first container group and the peer container groups are restarted in a coordinated manner, rather than restarting all instances. Simultaneously, for each training process after restarting, the corresponding checkpoint shard is obtained based on its accelerator card type, and a global recovery point is determined based on the minimum number of training steps in each checkpoint shard, thereby restoring the training tasks of each training process. This communication topology-driven minimum coordinated restart mechanism avoids the problem in related technologies where restarting all instances due to a failure results in the loss of training progress for instances without communication dependencies. Furthermore, the checkpoint recovery mechanism based on accelerator card type shards avoids full checkpoint rollback, significantly reducing fault recovery time and the amount of training step rollback, improving the fault tolerance efficiency and training continuity of heterogeneous mixed training tasks, and ultimately providing users with a highly available, low-loss heterogeneous mixed training fault recovery service.

[0112] Figure 4 This is a flowchart illustrating the method for determining training efficiency in heterogeneous hybrid training provided by the present invention. Figure 4 As shown, the method includes the following: Step 401: Obtain the actual single-step time of the target training process at each step from the training log.

[0113] The training log is a structured log record output by the training process after each training step. It contains fields such as timestamp, elapsed time, loss value, and iteration number, and is used to reconstruct the running status of the training process. For example, the training log records "step100, elapsed_time=1523.45ms, loss=0.0234, iteration=100", where the elapsed_time field is the actual time elapsed for that step.

[0114] Actual single-step time is the actual time consumed by the target training process to complete one training step, including all overhead such as computation, communication, and I / O, reflecting the true execution speed of the process in the current heterogeneous environment. For example, the target training process on the A-type accelerator card actually took 1523.45ms to complete the 100th training step, and this value is the actual single-step time for that step.

[0115] In the specific implementation process, the metric_reporter background process continuously monitors the writing of training log files; metric_reporter matches fields in the logs that begin with "elapsed_time=" using regular expressions; metric_reporter extracts the value of the elapsed_time field in each step of the log as the actual single-step time of the target training process in that step; metric_reporter caches the extracted actual single-step time along with the corresponding iteration and loss in a memory queue; metric_reporter reports the cached data to the computing power scheduling platform through the airsctl job metric command according to the preset reporting period.

[0116] Step 402: Based on the reciprocal of the actual single-step time and the sum of the reciprocals of the single-machine single-step benchmark time corresponding to each accelerator card in the target heterogeneous computing power cluster, determine the heterogeneous training efficiency of the target training process.

[0117] In practical implementation, the ratio of the reciprocal of the actual single-step time to the sum of the reciprocals of the single-step baseline time of each accelerator card in the target heterogeneous computing power cluster can be used as the heterogeneous training efficiency of the target training process.

[0118] The single-step benchmark time is the pre-determined training time for a single step on a specific accelerator card when running the same model independently. It reflects the pure computing power of the chip without cross-chip communication overhead and serves as a theoretical benchmark for calculating heterogeneous training efficiency. For example, the single-step benchmark time for accelerator card A is 14012.37ms, and the single-step benchmark time for accelerator card B is 10358.385ms. These two values ​​were obtained by pre-determining the same model by running it independently on each accelerator card.

[0119] Heterogeneous training efficiency is a metric that quantifies the efficiency ratio of heterogeneous mixed training relative to the theoretically optimal efficiency of independent training on each chip. Based on harmonic average calculation, 100% represents the theoretically optimal value when all chips are fully parallel with no communication overhead. For example, when the actual single-step time is 1200ms, the heterogeneous training efficiency is (1 / 1200) / (1 / 14012.37+1 / 10358.385)×100%≈85.3%, indicating that the current mixed training efficiency has reached 85.3% of the theoretical optimal level.

[0120] In the specific implementation process, the metric_reporter obtains the actual single-step time of each step of the target training process from the memory queue; the metric_reporter reads the single-machine single-step benchmark time corresponding to each accelerator card model pre-configured in the startup script. For example, the single-machine single-step benchmark time corresponding to accelerator card model A is 14012.37ms, and the single-machine single-step benchmark time corresponding to accelerator card model B is 10358.385ms; the metric_reporter calculates the reciprocal of the actual single-step time 1 / elapsed_time; the metric_reporter calculates the sum of the reciprocals of the single-machine single-step benchmark time of each accelerator card 1 / EFF_A+1 / EFF_B; the metric_reporter divides the two and multiplies by 100% to obtain the heterogeneous training efficiency; the metric_reporter reports this heterogeneous training efficiency, along with loss, iteration, and actual single-step time, to the computing power scheduling platform through the airsctl job metric command; the computing power scheduling platform displays the heterogeneous training efficiency trend curve of the target training process according to chip type on the monitoring interface.

[0121] In the embodiments provided by this invention, the actual single-step time of each step of the target training process is obtained from the training log during the training process. The heterogeneous training efficiency is calculated by summing the reciprocal of this actual single-step time with the reciprocal of the baseline single-step time for each accelerator card in the target heterogeneous computing cluster. This efficiency calculation model, based on the relationship between the actual single-step time and the reciprocal of the baseline time, effectively provides a precise quantitative evaluation capability for heterogeneous mixed training efficiency, avoiding the problem in related technologies where the lack of a unified efficiency metric leads to a lack of data for chip combination scheme selection and pipeline layer allocation optimization. Simultaneously, this efficiency index can intuitively quantify the impact of cross-chip communication overhead and load imbalance on training performance, providing quantifiable data for chip combination scheme selection, pipeline layer allocation optimization, and communication topology adjustment.

[0122] Figure 6 This is a schematic diagram of the architecture of the heterogeneous hybrid training system provided by the present invention.

[0123] like Figure 6 As shown, the heterogeneous hybrid training system provided by the present invention includes a platform layer (AIRS computing power scheduling platform), a training framework layer (FlagScale), and a communication library layer (FlagCX).

[0124] The platform layer (AIRS computing power scheduling platform) includes the following modules: The task submission and identification module (ExperimentHandler) is used to receive heterogeneous mixed training task requests, identify the role configurations of various accelerator card types, and trigger HeteroGroup matching and heterogeneous queue verification.

[0125] The HeteroGroup configuration (ResourceClient) maintains the HeteroGroup configuration table and provides a full set matching query based on the set of accelerator card models.

[0126] The ReplicaSpec generation module (createReplicaSpec) generates an independent KubernetesReplicaSpec for each Role, automatically injecting the image, NodeAffinity, and resource limits.

[0127] The heterogeneous information annotation injection module (setHeterogeneousInfo) extracts the accelerator card model and quantity, serializes them into JSON annotations, and writes them to the Job object. The Volcano scheduling integration layer incorporates all roles into a unified Gang scheduler through Kubeflow SchedulingPolicy.

[0128] The Communication Topology Manager (CommTopologyManager) receives and persists the communication topology reported by the training framework.

[0129] The Fault Correlation Decider module calculates the minimum linkage restart range based on the communication topology.

[0130] The training metric storage and display module (JobMetricStore) stores metric data and displays fault profiles and efficiency trends.

[0131] The training framework layer (FlagScale) includes: The SSHTrainRunner module parses the hostfile, starts the training process on each node via SSH, and injects specific configurations for each accelerator card type.

[0132] The RankMapper module collects all accelerator card types by rank and sorts them in order to create a two-way mapping table between physical and logical ranks.

[0133] The ProcessMesh module constructs an independent multidimensional Cartesian parallel topology (TP×CP×EP×DP×PP) for each accelerator card type.

[0134] The ParallelContext module manages all ProcessMesh, creates inter-mesh process groups, and handles cross-mesh tensor shape adaptation.

[0135] The Layer_Split module allocates model layers to pipeline stages of different accelerator card types according to the computing power ratio of the accelerator cards.

[0136] The communication library layer (FlagCX) includes: The Adaptor plugin system module supports eight types of chip communication libraries, including the native communication library for model A and model B, through the unified interface of flagcxCCLAdaptor and flagcxDeviceAdaptor.

[0137] The dual communication handle module automatically identifies homogeneous or heterogeneous communication domains through bootstrap allgather, and initializes dual communication handles: intra-chip communication handle and cross-chip communication handle.

[0138] The C2cPlanner module is a heterogeneous collection communication phased execution plan generator that caches communication plans using an LRU cache.

[0139] The topology detection and P2P transmission module is used to detect the server topology and select the rank closest to the network card for each chip cluster as the representative training process. The P2P transmission layer implements cross-chip point-to-point transmission based on IB RDMA or Socket, and supports direct RDMA to the device buffer.

[0140] The cross-chip collaborative scheduling device for heterogeneous hybrid training provided by the present invention will be described below. The cross-chip collaborative scheduling device for heterogeneous hybrid training described below can be referred to in correspondence with the cross-chip collaborative scheduling method for heterogeneous hybrid training described above.

[0141] like Figure 7 As shown, the cross-chip collaborative scheduling device 700 for heterogeneous hybrid training includes the following modules.

[0142] The receiving module 710 is used to receive a startup request for a heterogeneous hybrid training task; wherein the startup request carries description information of multiple roles, and each role is a resource grouping unit in the training task, with each role corresponding to a type of accelerator card.

[0143] Create module 720, which is used to create replicas of each character based on their descriptions.

[0144] Assembly module 730 is used to assemble the corresponding copy specifications of each role into a distributed training task object.

[0145] The scheduling module 740 is used to instruct the container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and to call the batch scheduling engine to execute the group scheduling strategy for the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to the nodes of the target heterogeneous computing power cluster.

[0146] The startup module 750 is used to start the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

[0147] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a cross-chip collaborative scheduling method for heterogeneous hybrid training. This method includes: receiving a start request for a heterogeneous hybrid training task; wherein the start request carries description information of multiple roles, each role being a resource grouping unit in the training task, and each role corresponding to a type of accelerator card; creating replica specifications for each role based on the description information of each role; assembling the replica specifications corresponding to each role into a distributed training task object; instructing a container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calling a batch processing scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and starting the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

[0148] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the above methods. The method includes: receiving a startup request for a heterogeneous hybrid training task; wherein the startup request carries description information of multiple roles, each role being a resource grouping unit in the training task, and each role corresponding to a type of accelerator card; creating replica specifications for each role according to the description information of each role; assembling the replica specifications corresponding to each role into a distributed training task object; instructing a container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calling a batch processing scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and starting the heterogeneous hybrid training task in the target heterogeneous computing power cluster.

[0150] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a cross-chip collaborative scheduling method for heterogeneous hybrid training provided by the above methods. The method includes: receiving a startup request for a heterogeneous hybrid training task; wherein the startup request carries description information of multiple roles, each role being a resource grouping unit in the training task, and each role corresponding to a type of accelerator card; creating replica specifications for each role based on the description information of each role; assembling the replica specifications corresponding to each role into a distributed training task object; instructing a container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and calling a batch scheduling engine to execute a group scheduling strategy on the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to nodes of the target heterogeneous computing power cluster; and starting the heterogeneous hybrid training task in the target heterogeneous computing power cluster.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-chip collaborative scheduling method for heterogeneous hybrid training, characterized in that, include: Receive a startup request for a heterogeneous hybrid training task; wherein the startup request carries description information of multiple roles, and each role is a resource grouping unit in the training task, with each role corresponding to a type of accelerator card; Create a copy of each character based on their description information; Assemble the corresponding instance specifications for each role into a distributed training task object; The container orchestration platform is instructed to create container groups corresponding to each role based on the distributed training task object, and to call the batch scheduling engine to execute group scheduling policies on the container groups corresponding to each role, so as to schedule all container groups corresponding to each role to the nodes of the target heterogeneous computing power cluster. The heterogeneous hybrid training task is initiated on the target heterogeneous computing power cluster.

2. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to claim 1, characterized in that, The description information of the character includes the resource specifications and number of copies of the acceleration card corresponding to the character, and the resource specifications of the acceleration card include the model of the acceleration card; The process of creating replicas for each character based on their descriptions includes: For each character's description, perform the following operations: Based on the model of the accelerator card, the mirror address corresponding to the accelerator card is obtained from the heterogeneous training group configuration information; wherein, the heterogeneous training group configuration information contains mapping relationships between various types of accelerator cards and mirror addresses; Based on the accelerator card model, a node affinity constraint is generated; wherein the key of the node affinity constraint is the accelerator card type, and the value is the accelerator card model. Generate a resource list for the character based on the resource specifications. The replica specifications of the character are obtained based on the mirror address corresponding to the accelerator card, the node affinity constraint, the resource list, and the number of replicas.

3. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to any one of claims 1 to 2, characterized in that, The process of launching the heterogeneous hybrid training task on the target heterogeneous computing power cluster includes: Obtain the host name, number of slots, and accelerator card model for each container group; Generate a host list file based on the host name, number of slots, and accelerator card model of each container group; Based on the host list file, the training process is started in each container group through the training framework of the heterogeneous hybrid training task; For each container group, perform the following operations: Based on the model of the accelerator card in the container group, the library path and environment variables of the training process within the container group are obtained from the device type specification configuration; wherein, the device type specification configuration is used to store the library path and environment variables corresponding to different types of accelerator cards; Inject the library path and environment variables into the training process within the container group; Based on the training process within each container group, the heterogeneous hybrid training task is launched on the target heterogeneous computing power cluster.

4. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to claim 3, characterized in that, The process of launching the heterogeneous hybrid training task on the target heterogeneous computing power cluster based on the training process within each container group includes: Logical rank is assigned to each training process based on the chip type corresponding to each training process and the order defined in the heterogeneous chip type configuration. Based on the heterogeneous process grid configuration and the number of training processes corresponding to each type of accelerator card, a process grid corresponding to each type of accelerator card is constructed, and the logical rank of each training process is mapped to the corresponding position in the process grid; wherein, the process grid is a multi-dimensional topology structure used to organize the parallelism of training processes; the heterogeneous process grid configuration is used to define the shape and dimension information of each process grid; Based on the heterogeneous pipeline layer partitioning configuration and the process grid, the total number of model layers for the heterogeneous hybrid training task is proportionally allocated to each training process along the pipeline parallel dimension. Based on the position of each training process in the process grid, determine the tensor parallel configuration and / or context parallel configuration of each training process.

5. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to claim 4, characterized in that, The method further includes: A cross-grid communication group is established between the first adjacent training processes corresponding to the pipeline parallel boundary of adjacent process grids. The cross-grid communication group is used to execute tensor slicing or splicing schemes when the tensor parallel configurations of the adjacent process grids are different. A cross-chip data parallel process group is established between the second adjacent training processes corresponding to the data parallel boundary of the adjacent process grid. The cross-chip data parallel process group is used to synchronize the gradient of the second adjacent training process.

6. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to any one of claims 3 to 5, characterized in that, The method further includes: Training processes within the homogeneous communication domain communicate via a chip-based communication handle. This chip-based communication handle calls the accelerator card's native communication library through a unified adapter interface, which is a unified encapsulation of the native communication libraries of various types of accelerator cards. The homogeneous communication domain is a collection of training processes using the same type of accelerator card. Between different homogeneous communication domains, representative training processes of each homogeneous communication domain conduct cross-domain communication through cross-chip communication handles; wherein, the representative training process is the training process closest to the network card in the homogeneous communication domain; the cross-chip communication handle is an operation interface used for data interaction between different types of accelerator cards.

7. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to claim 6, characterized in that, The method further includes: In response to triggering a full reduction of gradient data across the communication domain, the following operation is performed: Within each homogeneous communication domain, the gradient data fragmentation reduction is performed between each training process within the homogeneous communication domain using the same chip communication handle. Based on the fragmented data generated after the gradient data is fragmented and reduced within each homogeneous communication domain, the gradient data is globally reduced between representative training processes in each homogeneous communication domain through the cross-chip communication handle. Based on the fragmented data obtained from the global reduction, within each homogeneous communication domain, the fragmented data after global reduction is aggregated into complete gradient data using the same chip communication handle.

8. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to any one of claims 3 to 7, characterized in that, Each training process stores checkpoints according to the type of its corresponding accelerator card; The method further includes: In response to a failure in the first container group, the communication group to which the first container group belongs in the communication topology of the target heterogeneous computing power cluster is determined, and the peer container group within the communication group that has a communication dependency with the first container group is obtained. Restart the first container group and the peer container group; For each training process in the first container group and the peer container group after restarting, obtain the checkpoint shard corresponding to the accelerator card type of the training process; The global recovery point is determined based on the minimum number of training steps in each checkpoint slice. The training tasks of each training process are restored based on the checkpoints corresponding to the global recovery point.

9. The cross-chip collaborative scheduling method for heterogeneous hybrid training according to any one of claims 3 to 8, characterized in that, The method further includes: Obtain the actual single-step time of the target training process at each step from the training log; The heterogeneous training efficiency of the target training process is determined by summing the reciprocal of the actual single-step time with the reciprocal of the single-machine single-step baseline time corresponding to each accelerator card in the target heterogeneous computing power cluster.

10. A cross-chip collaborative scheduling device for heterogeneous hybrid training, characterized in that, include: The receiving module is used to receive the start request of the heterogeneous hybrid training task; wherein the start request carries description information of multiple roles, and the role is a resource grouping unit in the training task, and each role corresponds to a type of accelerator card; The creation module is used to create replicas of each character based on their descriptions. The assembly module is used to assemble the corresponding replicas of each role into distributed training task objects; The scheduling module is used to instruct the container orchestration platform to create container groups corresponding to each role based on the distributed training task object, and to call the batch scheduling engine to execute the group scheduling strategy for the container groups corresponding to each role, so as to schedule all container groups corresponding to all roles to the nodes of the target heterogeneous computing power cluster. The startup module is used to start the heterogeneous hybrid training task on the target heterogeneous computing power cluster.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the cross-chip collaborative scheduling method for heterogeneous hybrid training as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-chip collaborative scheduling method for heterogeneous hybrid training as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-chip collaborative scheduling method for heterogeneous hybrid training as described in any one of claims 1 to 9.

Images