Large-scale model hybrid training methods, devices, equipment, storage media, and program products

Abstract

This disclosure relates to the field of artificial intelligence technology, and in particular provides a method, apparatus, device, storage medium, and program product for large-scale model hybrid training. The method includes: in response to initiating a hybrid training task of a target large model on a heterogeneous computing cluster, obtaining the global heterogeneous topology of the heterogeneous computing cluster; the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes the device type in the process information of each processor; determining the heterogeneous parallel mode of the heterogeneous computing cluster based on the hybrid training task; the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode; non-uniformly partitioning the hybrid training task based on the heterogeneous parallel mode and device type to generate sub-training tasks for each processor; and controlling each processor to execute the hybrid training task on the target large model based on the sub-training tasks. This disclosure can enhance compatibility with heterogeneous computing environments, improve training efficiency, and reduce training costs.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for large-scale model hybrid training. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the rise of large-scale language models (LLMs), the number of model parameters and the scale of data have increased dramatically, leading to an exponential increase in the demand for computing resources.

[0003] In related technologies, large-scale model training typically relies on high-performance computing clusters built from a large number of homogeneous Graphics Processing Units (GPUs). However, in practical applications, computing clusters often contain artificial intelligence (AI) chips of different models, brands, and even architectures, forming a heterogeneous computing environment. Therefore, existing model training methods in heterogeneous computing environments suffer from poor compatibility and low training efficiency. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a method, apparatus, device, storage medium, and program product for large-scale model hybrid training.

[0005] According to one aspect of this disclosure, a method for hybrid training of large models is provided, comprising: In response to the launch of a hybrid training task for a target large model on a heterogeneous computing cluster, the global heterogeneous topology of the heterogeneous computing cluster is obtained; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type; Based on the hybrid training task, the heterogeneous parallel mode of the heterogeneous computing cluster is determined; wherein, the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. Based on the heterogeneous parallel mode and the device type of each processor, the hybrid training task is non-uniformly divided to generate sub-training tasks for each processor. During the training iteration, each processor is controlled to perform the hybrid training task on the target large model based on the sub-training task.

[0006] According to another aspect of this disclosure, a large model hybrid training apparatus is provided, comprising: The acquisition module is used to acquire the global heterogeneous topology of the heterogeneous computing cluster in response to the start of a hybrid training task of a target large model in the heterogeneous computing cluster; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type; The processing module is used to determine the heterogeneous parallel mode of the heterogeneous computing cluster based on the hybrid training task; wherein the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. The processing module is further configured to perform non-uniform partitioning of the hybrid training task based on the heterogeneous parallel mode and the device type of each processor, and generate sub-training tasks for each processor. The training module is used to control each of the processors to perform the hybrid training task on the target large model based on the sub-training task during the training iteration.

[0007] In another aspect of exemplary embodiments of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the methods described in exemplary embodiments of this disclosure.

[0008] In another aspect of exemplary embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described in exemplary embodiments of the present disclosure.

[0009] In another aspect of the exemplary embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the exemplary embodiments of this disclosure.

[0010] As will be described in detail below, the large-model hybrid training method according to embodiments of this disclosure obtains the global heterogeneous topology of the heterogeneous computing cluster by responding to the initiation of a hybrid training task for a target large model in response to a heterogeneous computing cluster. The heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information for each processor, with the process information including at least the device type. Based on the hybrid training task, a heterogeneous parallel mode of the heterogeneous computing cluster is determined. This heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. Based on the heterogeneous parallel mode and the device type of each processor, the hybrid training task is non-uniformly partitioned to generate sub-training tasks for each processor. During training iterations, each processor is controlled to execute the hybrid training task for the target large model based on the sub-training tasks. This allows for non-uniform partitioning of the hybrid training task of the target large model according to the heterogeneous parallel mode of the heterogeneous computing cluster and the device type of each processor, enabling each processor to collaboratively complete the entire hybrid training task after completing its own sub-training tasks. This enhances the compatibility of the hybrid training method with heterogeneous computing environments, improves training efficiency, and reduces training costs.

[0011] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A schematic diagram of the architecture of the heterogeneous hybrid training system provided in an exemplary embodiment of this disclosure is shown. Figure 2 A flowchart illustrating the large model hybrid training method provided in an exemplary embodiment of this disclosure is shown. Figure 3 A flowchart illustrating the heterogeneous topology sensing method provided in an exemplary embodiment of this disclosure is shown. Figure 4 This illustration shows a flowchart of a hybrid training process based on a heterogeneous computing cluster, provided by an exemplary embodiment of this disclosure. Figure 5 A schematic diagram of the non-uniform partitioning and construction logic of a large model in a heterogeneous pipeline parallel mode provided by an exemplary embodiment of this disclosure is shown. Figure 6A schematic diagram illustrating the non-uniform partitioning and construction logic of a large model under a heterogeneous data parallel mode provided in an exemplary embodiment of this disclosure is shown. Figure 7 This illustration shows a communication control process for data communication based on a point-to-point communication method using remote direct memory access, as provided in an exemplary embodiment of this disclosure. Figure 8 A schematic diagram illustrating a method for implementing collection communication in a heterogeneous global specification scenario provided by an exemplary embodiment of this disclosure is shown. Figure 9 A schematic diagram of the structure of the large model hybrid training apparatus provided in an exemplary embodiment of this disclosure is shown; Figure 10 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown; Figure 11 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0015] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0016] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] Currently, mainstream large-scale model training solutions are based on homogeneous clusters, while a few research results have proposed their own heterogeneous hybrid training solutions. Looking at current research on heterogeneous hybrid training, early work was mainly limited to mixed training scenarios involving heterogeneous Central Processing Units (CPUs) and GPUs, and heterogeneous GPUs of different generations. The focus was on optimizing the parallel scheduling logic of training tasks in heterogeneous scenarios, and there was a general lack of efficient communication compatibility and optimization solutions between heterogeneous AI chips from different ecosystems. While related technologies have proposed achieving heterogeneous communication compatibility through a unified distributed communication library, the specific implementation logic is not clearly defined, and the experimental data provided can only support mixed training of a maximum of two types of AI chips, failing to support subsequent more diverse heterogeneous computing power mixed training scenarios, and exhibiting problems such as communication isolation, uneven computing power, and poor compatibility.

[0020] Therefore, in order to solve the above problems, this disclosure provides a large model hybrid training method that can actively be compatible with diverse heterogeneous AI chip ecosystems, has high communication efficiency, low intrusion into existing software stacks, and can achieve intelligent parallelism and load balancing with heterogeneous awareness, so as to fully release the potential of heterogeneous computing power clusters, improve resource utilization efficiency, and reduce the total cost of ownership for large model training.

[0021] The large model hybrid training method provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0022] For example, the terminal may include one or more of the following: mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, PDA, and wearable device based on augmented reality (AR) and / or virtual reality (VR) technology. The exemplary embodiments disclosed herein do not impose specific limitations on these.

[0023] To address the issues of interoperability between heterogeneous AI computing power ecosystems and significant differences in chip computing capabilities, this disclosure provides a heterogeneous hybrid training system operating at the AI ​​training software layer. Figure 1 A schematic diagram of the architecture of a heterogeneous hybrid training system provided in an exemplary embodiment of this disclosure is shown. Figure 1As shown, the core components of the heterogeneous hybrid training system include a heterogeneous distributed compatibility library and a heterogeneous communication compatibility library. Through monkey patching, it covers the relevant public methods of the open-source distributed training framework (based on the Megatron-LM heterogeneous ecosystem version in this embodiment) and the deep learning framework (based on the PyTorch heterogeneous ecosystem version in this embodiment), realizing the logic of computing power allocation and communication compatibility in heterogeneous hybrid training scenarios, while also being compatible with the existing homogeneous set communication mode.

[0024] The heterogeneous distributed compatibility library adds related parsing, verification and implementation for heterogeneous mixed-mode training, supports heterogeneous pipeline parallelism (PP) mode and heterogeneous data parallelism (DP) mode, and distinguishes between modes to initialize process groups.

[0025] In heterogeneous pipeline parallel mode, configuration parameters such as heterogeneous pipeline stages (PPStages) are used to control the non-uniform partitioning of pipeline stages, adapting to the computing capabilities of different types of AI chips in a heterogeneous computing cluster. Here, the configuration parameters of the heterogeneous pipeline parallel mode define how many pipeline stages (PP Stages) each type of AI chip is responsible for, and how many layers of neural networks are included in each pipeline stage, allowing for non-uniform partitioning of large models based on these configuration parameters.

[0026] Understandably, in pipelined parallel training, the entire large model is vertically divided into several continuous pipeline segments, each segment being called a "pipeline stage". Each pipeline stage is placed on a different processor (also called a device, such as the AI ​​chip mentioned above), and the data passes through these pipeline stages in sequence to complete forward and backward computations, just like a pipeline.

[0027] In heterogeneous data parallelism, configuration parameters such as mixed-micro-batch sizes control the non-uniform splitting of training data. Here, the configuration parameters define the size of the data subset (micro-batch) processed by each type of AI chip in each forward propagation. In other words, processors with higher computing power are assigned larger micro-batches, and processors with lower computing power are assigned smaller micro-batches. Its core feature is allowing processors with different computing capabilities to process micro-batches of different sizes in the same training step, thus achieving load balancing.

[0028] Compared to the traditional approach of uniformly dividing network layers or training data into equal amounts for each processor, this embodiment allocates different numbers of network layers or micro-batches of training data to processors of different device types (also known as heterogeneous processors) based on their different computing capabilities. This allows the training workload to match the actual computing capabilities of the devices, thereby maximizing the overall training efficiency of the heterogeneous computing cluster. The aim is to enable all processors to complete calculations almost simultaneously, reduce waiting time, and improve overall efficiency.

[0029] The heterogeneous communication compatibility library refactors deep learning frameworks using the `init_process_group` method, enabling processes on heterogeneous processors to be aware of each other and forming a globally unified heterogeneous communication topology. It also refactors asynchronous sending (`isend`) and asynchronous receiving (`irecv`) methods to achieve point-to-point (P2P) tensor transmission and reception capabilities between heterogeneous processors in hybrid training scenarios. Furthermore, it refactors global reduction (`all_reduce`) and other set communication methods, enabling global gradient synchronization in typical heterogeneous dynamic programming (DP) training scenarios through heterogeneous set communication.

[0030] Based on this, the present disclosure provides a method for large-scale model hybrid training. Figure 2 A schematic flowchart of a large model hybrid training method provided in an exemplary embodiment of this disclosure is shown. Figure 2 As shown, the large model hybrid training method includes: S201, in response to the heterogeneous computing cluster launching a hybrid training task of a target large model, obtain the global heterogeneous topology of the heterogeneous computing cluster; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type. S202, Based on the hybrid training task, determine the heterogeneous parallel mode of the heterogeneous computing cluster; wherein, the heterogeneous parallel mode includes heterogeneous pipeline parallel mode or heterogeneous data parallel mode; S203, based on the heterogeneous parallel mode and the device type of each processor, performs non-uniform partitioning of the hybrid training task to generate sub-training tasks for each processor. S204 controls each processor to perform a hybrid training task on the target large model based on sub-training tasks during the training iteration process.

[0031] Specifically, the aforementioned target large model is a neural network model to be trained or built. Users submit a hybrid training task for the target large model on a heterogeneous computing cluster via command line or scheduling system. Here, the aforementioned heterogeneous computing cluster can be multiple processors of different device types (also referred to as nodes or devices in this embodiment). The device type of the processor can be identified according to device architecture, device manufacturer, or other methods, depending on actual needs. This embodiment does not specifically limit this. In the method of this embodiment, the device type of the processor can be identified as NVIDIA graphics cards, Biren chips, etc.

[0032] The aforementioned global heterogeneous topology can be constructed in the form of a dictionary. For example, a dictionary can be constructed using a one-to-one mapping between topology index keys and corresponding values ​​to form a global heterogeneous topology. The topology index key can be the processor's device identifier (such as a process (rank number)), and the corresponding value can be the processor's process information.

[0033] In this embodiment of the disclosure, after obtaining the hybrid training task, the training configuration file of the hybrid training task can be used. This configuration file contains a mixed-mode, which explicitly specifies the heterogeneous parallel mode to be used in this hybrid training task. The heterogeneous parallel mode can include a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. The details regarding the heterogeneous pipeline parallel mode and the heterogeneous data parallel mode, as well as the configuration parameters for the heterogeneous pipeline parallel mode and the heterogeneous data parallel mode, are described above and will not be repeated here.

[0034] After determining the heterogeneous parallel mode of the heterogeneous computing cluster, the hybrid training task can be non-uniformly divided according to the device type of each processor, generating sub-training tasks for each processor to match the division of sub-training tasks with the device type of the processor. In other words, this allows the sub-training tasks of each processor to flexibly adapt to the changing heterogeneous environment. This flexibility enables the hybrid training task of the target large model to be extended from a fixed heterogeneous computing cluster scenario to a variable heterogeneous computing cluster scenario when controlling each processor to perform hybrid training tasks based on sub-training tasks during training iterations, thereby enhancing the adaptability of the hybrid training system to complex production environments.

[0035] According to the technical solution of the example embodiment of this disclosure, in response to the initiation of a hybrid training task for a target large model by a heterogeneous computing cluster, the global heterogeneous topology of the heterogeneous computing cluster is obtained. The heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information for each processor, with the process information including at least the device type. Based on the hybrid training task, a heterogeneous parallel mode of the heterogeneous computing cluster is determined. This heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. Based on the heterogeneous parallel mode and the device type of each processor, the hybrid training task is non-uniformly partitioned to generate sub-training tasks for each processor. During training iteration, each processor is controlled to execute the hybrid training task for the target large model based on the sub-training tasks. This allows for non-uniform partitioning of the hybrid training task for the target large model according to the heterogeneous parallel mode of the heterogeneous computing cluster and the device type of each processor, enabling each processor to collaboratively complete the entire hybrid training task after completing its own sub-training tasks. This enhances the compatibility of the hybrid training method with heterogeneous computing environments, improves training efficiency, and reduces training costs.

[0036] In some embodiments, process information may further include network address and service port. The network address, service port, and device type are stored together in the value corresponding to the topology index key in the global heterogeneous topology. For example, the value can be {ip, port, device_type}, where ip represents the network address, port represents the service port, and device_type represents the device type.

[0037] The method may also include: Each processor is controlled to determine its own process information; The target processor among multiple processors generates a global heterogeneous topology based on the process information of each processor, and broadcasts the global heterogeneous topology to each processor.

[0038] Specifically, topology awareness and construction are prerequisites for achieving distributed training. It requires that each processor in the distributed system, such as the GPU process (rank), be able to perceive basic information such as the IP address, service port, and device type of other processes.

[0039] For heterogeneous hybrid training scenarios, this disclosure embodiment specifically implements the Transmission Control Protocol (TCP) all-gather communication logic for heterogeneous topology awareness and replaces the initialization process group in the deep learning framework. Figure 3 A flowchart illustrating the heterogeneous topology sensing method provided in an exemplary embodiment of this disclosure is shown, as follows: Figure 3 As shown, tcp_all_gather consists of three key stages: heterogeneous information collection, topology information aggregation, and topology information broadcasting.

[0040] The first step is heterogeneous information collection. Since all heterogeneous processors configure the network address and service port of their root process (rank0) when starting a task, each processor's rank0, after completing its own `init_process_group` initialization, will send its own process information, such as network address, service port, and device type, to the rank0 processes of other processors via TCP Socket.

[0041] The second step is topology information aggregation. After receiving process information from the rank0 of all other processors, the rank0 of the main processor will construct a global heterogeneous topology from this process information in the form of a dictionary.

[0042] The third step is topology information broadcasting. Since the main processor's rank0 has recorded the process information of all other r processors' rank0, the aggregated process information can be rebroadcast to the rank0 of all processors in the heterogeneous computing cluster via TCP Socket. This ensures that the global heterogeneous topology is synchronized across the entire cluster, solving the problem of mutual awareness between processors in the heterogeneous computing cluster in a hybrid training scenario, and laying a solid foundation for efficient heterogeneous communication in the future.

[0043] In some embodiments, based on the heterogeneous parallel mode and the device type of each processor, the hybrid training task is non-uniformly divided to generate sub-training tasks for each processor, which may include: Based on the heterogeneous parallel mode, the configuration parameters corresponding to the heterogeneous parallel mode are parsed and verified; based on the configuration parameters, the partitioning objects of the hybrid training task are determined. Based on the computing power of each processor, the partitioning object is non-uniformly divided to obtain sub-partitioning objects for each processor; among them, processors with the same device type have the same computing power and the same sub-partitioning objects. The sub-partition objects are assigned to each processor, and sub-training tasks for each processor are generated based on the sub-partition objects.

[0044] Specifically, Figure 4 The illustration shows a flowchart of a hybrid training process based on a heterogeneous computing cluster, as provided in an exemplary embodiment of this disclosure. Figure 4 As shown, the heterogeneous pipeline parallel mode and heterogeneous data parallel mode provided in this disclosure are based on the pipeline parallel capabilities of open source Megatron-LM and PyTorch, and are customized for heterogeneous computing scenarios for model parameter initialization, process group initialization, model building, forward training communication process, etc.

[0045] The configuration parameters corresponding to the heterogeneous parallel mode can include the correspondence between the heterogeneous parallel mode, the partitioning object, and the device type. For example, the parameter parser in the heterogeneous distributed compatibility library starts working, parses and verifies the configuration parameters corresponding to the heterogeneous parallel mode, confirms that the currently running heterogeneous parallel mode is consistent with the heterogeneous parallel mode in the configuration parameters, and determines the partitioning object that needs to be non-uniformly partitioned, as well as the non-uniform partitioning rules corresponding to processors of different device types.

[0046] Here, the non-uniform partitioning rules corresponding to processors of different device types are adapted to the computing power of that type of processor. Based on this, embodiments of this disclosure can perform non-uniform partitioning of the partitioning object according to the non-uniform partitioning rules and based on the computing power of each processor to obtain sub-partitioning objects for each processor. It is understood that, since the non-uniform partitioning rules are adapted to the device type of the processor (that is, the non-uniform partitioning rules are adapted to the computing power of the processor), processors with the same device type have the same computing power and the same sub-partitioning objects.

[0047] After determining the sub-partition objects for each processor, the sub-partition objects can be assigned to each processor, and sub-training tasks for each processor can be generated based on the sub-partition objects. This ensures that the division of sub-training tasks matches the device type of the processor, allowing for flexible adaptation to diverse heterogeneous environments. It can extend the hybrid training tasks of the target large model from fixed heterogeneous computing cluster scenarios to diverse heterogeneous computing cluster scenarios, enhancing the adaptability of the hybrid training system to complex production environments.

[0048] For example, if the heterogeneous parallel mode is a heterogeneous pipeline parallel mode, then the partitioning object is multiple network layers of the target large model, and the sub-partitioning object is at least one pipeline stage among multiple pipeline stages obtained by non-uniformly partitioning multiple network layers. At least one pipeline stage is matched with the computing power of each processor, and each pipeline stage includes at least one network layer.

[0049] Specifically, for the heterogeneous pipeline parallel mode, a pipeline stage is defined for each type of processor. Each processor may be responsible for one or more pipeline stages, and this embodiment does not specifically limit this. Each pipeline stage may include at least one network layer (also called a model layer).

[0050] The definition format can be: The order of processor device types should match the order of device types in the mixed-device-types list. Indicates the first The sub-partitions of a processor are pipeline stages, and each pipeline stage corresponds to a network layer. , Indicates the first i Class processor in j The network layer corresponding to each pipeline stage This represents the total number of pipeline stages obtained by non-uniformly partitioning multiple network layers. This represents the total number of network layers. In non-uniform partitioning rules...

[0051] In heterogeneous pipelined parallel mode, due to the differences in computing power among different processors, uniformly splitting the model cannot guarantee a balanced load on computing and communication tasks across heterogeneous processors, leading to the "weakest link" effect in computing power. Therefore, heterogeneous distributed compatible libraries need to allocate different numbers of network layers based on the computing power of heterogeneous AI chips. Simultaneously, considering the memory requirements generated by activation values ​​during backpropagation, heterogeneous devices with larger amounts of memory are more suitable for undertaking tasks in the earlier stages of the pipeline (PP). This ensures overall balanced computing power across the entire hybrid training system in heterogeneous pipelined parallel scenarios.

[0052] In the native Megatron-LM, the large model is evenly divided according to the total number of network layers and the size of the pipeline parallel group, and a corresponding model copy is built for each pipeline parallel stage (PP Stage). The embodiments disclosed herein are based on this process and further optimized for heterogeneous scenarios.

[0053] Figure 5 This illustration shows a schematic diagram of the non-uniform partitioning and construction logic of a large model in a heterogeneous pipeline parallel mode provided by an exemplary embodiment of this disclosure. For example... Figure 5 As shown, during the initialization phase, the relevant configuration parameters of the heterogeneous pipeline parallelism are parsed and verified to obtain the number of model layers corresponding to each pipeline parallelism stage (PP Stage) for model construction.

[0054] During the model replica construction phase, each node (i.e., processor) determines the portion of the model it needs to maintain according to its corresponding pipeline parallel stage (PPStage), and defines and initializes the model structure and related computational logic accordingly. Specifically, when initializing the Transformer Language Model, each node is assigned a specific pipeline parallel stage (PPStage). Based on the given non-uniform partitioning strategy and its pipeline parallel stage (PPStage), the node determines and initializes the distributed Transformer (Parallel Transformer) model. The number of model layers in the Parallel Transformer models handled by different nodes may vary, and the first and last layers may contain special architectures.

[0055] Finally, the target large model built on the CPU is transferred to the corresponding GPU to complete the building process.

[0056] For example, if the heterogeneous parallel mode is a heterogeneous data parallel mode, then the partitioning object is the training data of the target large model, and the sub-partitioning object is one of the multiple micro-batches of different sizes obtained by non-uniformly partitioning the training data, and each micro-batch is matched with the computing power of each processor.

[0057] Specifically, mixed-micro-batch sizes are configured for heterogeneous data parallelism, defining different micro-batch sizes for each type of processor. This definition format can be... The order of processor device types should match the order of device types in the mixed-device-types list. Indicates the first i The sub-partition objects of the class processor are micro-batches. This represents the total number of micro-batches of different sizes obtained by non-uniformly dividing the training data. Indicates the first i Micro-batch size of the class processor.

[0058] The core problem to be solved in heterogeneous data parallel scenarios is the non-uniform splitting of training data into micro-batches from the input of large models. During the initialization phase, the heterogeneous distributed parallelism library parses and verifies the configuration parameters of the heterogeneous data parallelism mode, including the processor's device type (device_type) and mixed-micro-batch sizes (mixed-micro-batch-sizes), to determine the size of the micro-batch data subset for each processor. The core components involved in data splitting in the heterogeneous distributed parallelism library are the micro-batch size calculator and the data loader. The overall implementation logic of the micro-batch size calculator and the data loader is as follows: Figure 6 As shown, Figure 6 This illustration shows a schematic diagram of the non-uniform partitioning and construction logic of a large model under a heterogeneous data parallel mode provided by an exemplary embodiment of this disclosure.

[0059] like Figure 6 As shown, the first step, the micro-batch quantity calculator, calculates the total size of a single mixed-micro-batch by parsing the mixed-micro-batch-sizes parameter. And verify the quantity of mixed micro-batches. It can be divided by the total micro-batch size. After successful validation, the micro-batch number calculator will actually divide the input global batch data into multiple identical mixed micro-batches.

[0060] The second step involves the data loader re-parseing the `mixed-micro-batch-sizes` parameter to determine the number of model replicas for each device type and the amount of data to be allocated. It then further divides the data in the mixed micro-batch into multiple micro-batches. During this process, the micro-batch size is the same for homogeneous processors, but may differ for heterogeneous processors.

[0061] The third step involves the data loader distributing this data to various model replicas, thereby completing the non-uniform data partitioning and distribution that matches the computing capabilities of the devices, effectively balancing the computing load of heterogeneous devices and reducing device waiting time.

[0062] In some embodiments, before non-uniformly partitioning the hybrid training task based on the heterogeneous parallel mode and the device type of each processor to generate sub-training tasks for each processor, the method may further include: If the heterogeneous parallel mode is a heterogeneous pipelined parallel mode, then the process groups are initialized in the order of tensor parallel group, data parallel group, and pipelined parallel group, and it is ensured that the device types of the processors in the same tensor parallel group and the same data parallel group are the same. If the heterogeneous parallel mode is a heterogeneous data parallel mode, then the process groups are initialized in the order of tensor parallel group, pipeline parallel group, and data parallel group, and it is ensured that the device types of the processors in the same tensor parallel group and the same pipeline parallel group are the same.

[0063] Specifically, in the heterogeneous pipelined parallel mode, process groups are initialized in the order of tensor parallelism group, data parallelism group, and pipelined parallelism group, ensuring that processors within the same tensor parallelism group and data parallelism group have the same device type. All processors within the tensor parallelism (TP) group (also called TP group) and data parallelism group (also called DP group) must belong to the same device type. Verification logic ensures that the node ranking (node_rank) of processors of the same type remains adjacent, guaranteeing that homogeneous processors can be automatically assigned to the same tensor parallelism group and data parallelism group when dividing the tensor parallelism group and data parallelism group. Within the pipelined parallelism group, heterogeneous AI chip devices constitute different pipeline stages (PP stages). To ensure communication compatibility, all computation and communication operations within the same pipeline stage (PP stage) should be executed on processors of the same type. Based on the final process group division goal of the heterogeneous pipelined parallel mode, the initialization process follows the TP-DP-PP sequence during the initialization phase.

[0064] Assume there are a total of world_size processors (representing the total number of processors in the heterogeneous computing cluster), and the number of processors in the TP, DP, and PP groups are tp_size, dp_size, and pp_size, respectively. Each of these processors is responsible for a continuous pipeline stage of the target large model, collectively forming the complete pipeline of the target large model. The relationship between world_size and p_size, dp_size, and pp_size should satisfy: tp_size × dp_size × pp_size = world_size.

[0065] First, based on tp_size, adjacent processors are assigned to the same TP group, for a total of G_tp TP groups, where G_tp = world_size / tp_size. For the i-th (i∈[0,G_tp-1]) parallel TP group, the rank number of the processors it contains (denoted by r in the following formula) is: TP_GROUP_RANKS[i] = {r | r = i × tp_size + tp_rank, tp_rank ∈ [0, tp_size)}. For example, if tp_size = 4, then the rank number of the processors in the 0th TP group is {0,1,2,3}; the rank number of the processors in the 1st TP group is {4,5,6,7}, and so on.

[0066] Secondly, DP groups are set up according to an interval of tp_size, that is, a certain number of processors are skipped between each TP group to form a DP group, for a total of G_dp DP parallel groups, where G_dp = world_size / dp_size. For the i-th (i∈[0,G_dp-1]) DP parallel group, the starting index formula for the rank number of the first processor is:

[0067] The rank of the processors included in the i-th DP group is: DP_GROUP_RANKS[i]={r|r=start_index(i)+dp_rank×tp_size,dp_rank∈[0,dp_size)}.

[0068] Each DP group consists of dp_size processors, whose global rank numbers increment by tp_size. Although the dp_size processors in each DP group are in different TP groups, they have the same local index within their respective TP groups.

[0069] Finally, PP groups are set up according to the interval tp_size × dp_size, for a total of G_pp = world_size / pp_size PP parallel groups. For the i-th (i∈[0,G_pp-1]) PP parallel group, the rank number of the processors contained therein is: PP_GROUP_RANKS[i]={r|r=i+pp_rank×(tp_size×dp_size),pp_rank∈[0,pp_size)}.

[0070] The core objective of heterogeneous data parallelism is to allow for heterogeneous processors of different device types among DP groups, while ensuring that the device types within TP and PP groups are consistent. Therefore, in hybrid training using heterogeneous data parallelism, initialization needs to be completed in the order of TP-PP-DP. Other processes are similar to those in heterogeneous pipelined parallelism and will not be elaborated here.

[0071] In some embodiments, the method may further include: During the training iteration, when data communication is performed between two processors to complete forward propagation, backward propagation and gradient synchronization, the device types of the two processors are determined based on the global heterogeneous topology. When two processors have the same device type, the homogeneous set communication library corresponding to the device type is invoked for data communication. If the two processors have different device types, and the data communication is point-to-point, then the communication protocol selected by the amount of data to be communicated will be used for data communication. If the data communication is a set communication, the set communication is broken down into three steps: homogeneous set communication within the homogeneous sub-communication group where the two processors are located, point-to-point communication between homogeneous sub-communication groups, and broadcast communication within the homogeneous sub-communication group. These steps are executed sequentially to complete global data synchronization. The homogeneous sub-communication group is obtained by dividing multiple processors according to their device type. The number of homogeneous sub-communication groups is the same as the number of device types of the multiple processors.

[0072] Specifically, as described above, through heterogeneous topology awareness technology, each processor in the heterogeneous computing cluster has acquired information such as the global heterogeneous topology and device type, and can identify whether processors communicating with each other belong to the same device type when performing communication operations. Based on this characteristic, the embodiments of this disclosure further design a basic communication process in a heterogeneous hybrid training scenario.

[0073] During training iterations, if data communication between two processors is required for forward propagation, backward propagation, and gradient synchronization, the device types of the two processors to be communicating are first checked based on the global heterogeneous topology before any communication occurs. If the device types of the two processors are the same, the homogeneous set communication library corresponding to the device type is invoked for data communication; if the device types of the two processors are different, it is determined whether the data communication between the two processors is point-to-point communication or set communication.

[0074] If the data communication is point-to-point, then the communication protocol selected by the amount of data to be communicated is used.

[0075] Specifically, heterogeneous AI chips are typically absent within the same machine; therefore, heterogeneous communication is limited to machine-to-machine interaction. The most typical machine-to-machine communication methods include TCP and Remote Direct Memory Access (RDMA) transmission. When using traditional RDMA for data transfer between processors, control information is exchanged via TCP. This process usually involves several Transmission Control Protocol Socket (TCP Socket) control flow transmissions, resulting in lower communication efficiency for RDMA compared to TCP Socket in small-batch data transfer scenarios. Furthermore, the traditional RDMA transmission process has room for optimization in terms of control and transmission resource allocation in heterogeneous GPU scenarios. Based on this premise, the embodiments of this disclosure optimize the traditional RDMA P2P transmission mechanism and specifically implement RDMA control and communication transmission resource reuse for heterogeneous scenarios, further optimizing transmission efficiency while achieving heterogeneous RDMA P2P communication.

[0076] More specifically, for heterogeneous pipeline parallel mode and heterogeneous data parallel mode, the communication protocol is selected based on the amount of data communicated. If the amount of data communicated is less than a preset threshold, point-to-point communication based on Transmission Control Protocol sockets (i.e., TCP Socket) is used for data communication; if the amount of data communicated is greater than or equal to the preset threshold, point-to-point communication based on Remote Direct Memory Access (i.e., RDMA) is used for data communication. Here, the preset threshold can be set according to actual needs, and this embodiment does not specifically limit it.

[0077] The RDMA implementation in this disclosure can be achieved using Pyverbs. Pyverbs is an RDMA programming interface based on the Cython language, which can be directly called by Python programs. Pyverbs is implemented based on the C language Verbs API, meaning it can provide almost all the functionality implemented by C language Verbs. This disclosure optimizes the RDMA P2P communication control process to a three-way handshake. Figure 7 This illustration shows a communication control process for data communication based on a point-to-point communication method using remote direct memory access, as provided in an exemplary embodiment of this disclosure.

[0078] like Figure 7As shown, in the first handshake, the sender notifies the receiver of the communication data packet size, and the receiver allocates memory space according to the data packet size. Regarding this, the optimization scheme of this embodiment is that the TCPSocket resources used for transmission control and the RDMA resources that actually generate communication are initialized only once for the same pair of senders and receivers, and the channel resources are reused after the transmission channel is established. Furthermore, different ranks on a single node are grouped by affinity and evenly bound to multiple network interface devices used for RDMA communication, ensuring load balancing of RDMA communication bandwidth as much as possible. Through the above optimization scheme, communication resources can be saved and communication efficiency improved.

[0079] In the second handshake, the receiver notifies the sender of the address for receiving data, and then the sender begins sending data to the receiver. This embodiment of the disclosure defines the sender directly using the address of the data to be sent to allocate a memory region (MR). Thus, when the sender supports GDR (GPU Direct RDMA), data can be directly moved from the GPU to the RDMA, thereby improving communication efficiency.

[0080] In the third handshake, the sender notifies the receiver that data transmission has been completed, and the receiver begins reading the received data. Here, this embodiment defines the receiver as reusing the MR (Mean Memory Adapter) to the maximum extent possible based on the tensor dimension, avoiding the increased time consumption caused by creating new MRs and further improving communication efficiency.

[0081] If the data communication is a set communication, the set communication is broken down into three steps: homogeneous set communication within the homogeneous sub-communication group where the two processors are located, point-to-point communication between homogeneous sub-communication groups, and broadcast communication within the homogeneous sub-communication group. These steps are executed sequentially to complete global data synchronization. The homogeneous sub-communication group is obtained by dividing multiple processors according to their device type. The number of homogeneous sub-communication groups is the same as the number of device types of the multiple processors.

[0082] Specifically, using heterogeneous point-to-point communication mechanisms as atomic capabilities, this disclosure redesigns a custom heterogeneous aggregate communication method, including send, isend, recv, irecv, broadcast, reduce, gather, all_reduce, and all_gather. This enables efficient transmission of training data between heterogeneous nodes in hybrid training scenarios through aggregate communication. For point-to-point communication methods such as send, isend, recv, and irecv, this disclosure directly reuses the heterogeneous point-to-point communication process. For higher-order aggregate communication methods such as global reduce and global gather (all_reduce, all_gather), they are implemented by splitting them into homogeneous aggregate communication and heterogeneous point-to-point communication. Taking the heterogeneous global reduce all_reduce as an example, data reduction is achieved by splitting the specific communication logic into three parts: homogeneous reduce, heterogeneous point-to-point communication, and homogeneous broadcast.

[0083] Figure 8 The illustration shows a schematic diagram of a set communication implementation method in a heterogeneous global specification scenario provided by an exemplary embodiment of this disclosure, such as... Figure 8 As shown, in the heterogeneous global reduction (all_reduce) scenario, all processors (rank) to communicate constitute a global heterogeneous communication group, including multiple homogeneous sub-communication groups. Each homogeneous sub-communication group consists of homogeneous processors (processors of the same device type). The ranks within the homogeneous sub-communication groups are numbered sequentially. For example, the rank numbers for the A-class device subgroup (A homogeneous sub-communication group) are rank A0, rank A1... rank AN, and the rank numbers for the B-class device subgroup (B homogeneous sub-communication group) are rank B0, rank B1... rank BN. When performing heterogeneous global reduction (all_reduce) communication: (1) First, the homogeneous sub-communication group calls the proprietary collection communication library of various ecosystems to perform the reduction operation. The data after reduction will be stored on the first node of each homogeneous sub-communication group, such as rank A0 and rank B0.

[0084] (2) Then the first node of each type of device will send the reduced data of its own storage to rankA0. The designated aggregation node rankA0 will perform the reduce operation of its own node to obtain the final reduced data (the node will perform the reduction operation again to obtain the final unified reduction result), and send the data to the first node of each homogeneous sub-communication group through P2P communication.

[0085] (3) Each homogeneous head node directly calls the native collection communication library to perform a broadcast operation within the sub-homogeneous communication group. At this point, the heterogeneous global reduction all reduce communication operation is completed, and all global heterogeneous nodes have collected the data after the final reduction.

[0086] Here, we can see that the core idea of ​​heterogeneous global reduction is: first, group aggregation; then, cross-group summarization; and finally, intra-group broadcasting. First, intra-group reduction is performed: leveraging the advantages of each device's native high-performance communication library, preliminary data aggregation is completed within each homogeneous sub-communication group, transforming high-cost cross-device interactions into a small number of interactions between the first nodes. Next, cross-group summarization is performed: through custom heterogeneous point-to-point communication, data is exchanged and further aggregated between the first nodes of different types of devices to obtain a globally unique result, solving the core cross-ecosystem communication problem. Finally, intra-group broadcasting: again utilizing native broadcasting capabilities, the global result is efficiently distributed to every process. This design minimizes the amount and frequency of cross-device communication while maximizing the reuse of highly optimized homogeneous communication libraries from various vendors, balancing efficiency and compatibility.

[0087] Based on this, during the forward and backward propagation processes in a heterogeneous pipeline parallel mode, tensors may be transferred between different devices, such as... Figure 5 As shown, point-to-point communication methods including isend and irecv are used between PP Stage 0 and PP Stage 1. The isend and irecv methods are used here for heterogeneous point-to-point communication to ensure message passing compatibility between heterogeneous devices.

[0088] In the training process of a heterogeneous pipelined parallel mode, the tensor data generated during both forward and backward propagation needs to be transferred between different devices. For example, in Figure 5 In the illustrated process, the calculation results of PP Stage 0 need to be sent to PP Stage 1. This transmission process relies on point-to-point communication methods such as isend and irecv. In this embodiment of the disclosure, isend and irecv are implemented using the heterogeneous point-to-point communication method designed above, thereby ensuring compatibility and correctness when transmitting messages between different types of devices.

[0089] During training in a heterogeneous data parallel mode, since model replicas on different devices process different data, gradient synchronization is required among all model instances. Therefore, collective communication methods such as all_reduce (global reduction) are inevitably used. This disclosure overrides and refactors these native collective communication methods, enabling them to call custom heterogeneous communication logic. This ensures that message passing functions correctly when performing gradient synchronization and other operations between different types of processors.

[0090] Based on this, the large model hybrid training method provided in this disclosure can support heterogeneous AI chips to perceive and be compatible with each other in the same hybrid training system. Based on the monkey-patch coverage of Pytorch init_process_group, it can actively detect and adapt to heterogeneous AI chips, ensuring that they can recognize and cooperate with each other based on a unified deep learning framework and a distributed framework.

[0091] The large-model hybrid training method provided in this disclosure enables communication compatibility in heterogeneous hybrid training scenarios, reduces training costs, implements a custom point-to-point heterogeneous communication method between heterogeneous chips based on TCP Socket and Pyverbs, optimizes the communication process, supports GDR, improves hybrid training efficiency, and implements a heterogeneous aggregate communication method based on this. Through monkey-patch overlay of relevant PyTorch communication methods, communication between heterogeneous AI chips is achieved.

[0092] The large-model hybrid training method provided in this disclosure supports a heterogeneous distributed parallel mechanism, that is, it supports the selection of heterogeneous data parallel mode and heterogeneous pipelined parallel mode. Process groups are initialized according to the heterogeneous parallel mode to ensure a reasonable homogeneous and heterogeneous allocation of process groups. Then, the non-uniform partitioning of network layers and training data is completed according to the configuration. Finally, a custom heterogeneous communication method is used to complete forward propagation, backward propagation, and gradient synchronization.

[0093] The foregoing mainly describes the solutions provided by the embodiments of this disclosure. It is understood that, in order to achieve the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0094] This disclosure embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0095] By dividing each functional module according to its corresponding function, an exemplary embodiment of this disclosure provides a large model hybrid training device, which can be a terminal or a chip applied to a terminal. Figure 9 A schematic diagram of the structure of a large model hybrid training apparatus provided in an exemplary embodiment of this disclosure is shown. Figure 9 As shown, the device 900 includes: The acquisition module 901 is used to acquire the global heterogeneous topology of the heterogeneous computing cluster in response to the start of a hybrid training task of a target large model in the heterogeneous computing cluster; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type. Processing module 902 is used to determine the heterogeneous parallel mode of the heterogeneous computing cluster based on the hybrid training task; wherein the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. The processing module 902 is further configured to perform non-uniform partitioning of the hybrid training task based on the heterogeneous parallel mode and the device type of each processor, and generate sub-training tasks for each processor. The training module 903 is used to control each of the processors to perform the hybrid training task on the target large model based on the sub-training task during the training iteration.

[0096] In some embodiments, the process information further includes a network address and a service port; The processing module 902 is further configured to control each of the processors to determine the process information of each processor; control the target processor among the plurality of processors to generate the global heterogeneous topology based on the process information of each processor, and broadcast the global heterogeneous topology to each of the processors.

[0097] In some embodiments, the processing module 902 is further configured to parse and verify the configuration parameters corresponding to the heterogeneous parallel mode based on the heterogeneous parallel mode; and determine the partitioning object of the hybrid training task based on the configuration parameters; Based on the computing power of each processor, the partitioning object is non-uniformly divided to obtain sub-partitioning objects for each processor; wherein, processors of the same device type have the same computing power and the same sub-partitioning objects; The sub-partition objects are assigned to each of the processors, and sub-training tasks for each processor are generated based on the sub-partition objects.

[0098] In some embodiments, if the heterogeneous parallel mode is the heterogeneous pipeline parallel mode, then the partitioning object is multiple network layers of the target large model, and the sub-partitioning object is at least one of the multiple pipeline stages obtained by non-uniformly partitioning the multiple network layers. At least one pipeline stage is matched with the computing power of each processor, and each pipeline stage includes at least one network layer. If the heterogeneous parallel mode is the heterogeneous data parallel mode, then the partitioning object is the training data of the target large model, and the sub-partitioning object is one of the multiple micro-batches of different sizes obtained by non-uniformly partitioning the training data, and each micro-batch is matched with the computing power of each processor.

[0099] In some embodiments, the processing module 902 is further configured to initialize process groups in the order of tensor parallel group, data parallel group, and pipeline parallel group if the heterogeneous parallel mode is the heterogeneous pipeline parallel mode, and ensure that the device types of the processors in the same tensor parallel group and the same data parallel group are the same. If the heterogeneous parallel mode is a heterogeneous data parallel mode, then the process groups are initialized in the order of tensor parallel group, pipeline parallel group, and data parallel group, and it is ensured that the device types of the processors in the same tensor parallel group and the same pipeline parallel group are the same.

[0100] In some embodiments, the training module 903 is further configured to, during the training iteration process, when data communication is performed between the two processors to complete forward propagation, backward propagation and gradient synchronization, determine whether the device types of the two processors are the same based on the global heterogeneous topology; When the two processors have the same device type, the homogeneous set communication library corresponding to the device type is invoked to perform the data communication. If the two processors have different device types, and the data communication is point-to-point, then the data communication is performed using a communication protocol selected by the amount of data in the data communication. If the data communication is a set communication, then the set communication is divided into three steps: homogeneous set communication within the homogeneous sub-communication groups of the two processors, point-to-point communication between homogeneous sub-communication groups, and broadcast communication within the homogeneous sub-communication groups. These steps are executed sequentially to complete global data synchronization. The homogeneous sub-communication groups are obtained by dividing the multiple processors according to their device types, and the number of homogeneous sub-communication groups is the same as the number of device types of the multiple processors.

[0101] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the methods disclosed in this disclosure.

[0102] Figure 10 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this disclosure is shown. For example... Figure 10 As shown, the electronic device 1000 includes at least one processor 1001 and a memory 1002 coupled to the processor 1001. The processor 1001 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0103] The processor 1001 described above can also be referred to as a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1001 or by software instructions. The processor 1001 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1002, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1001 reads information from the memory 1002 and, in conjunction with its hardware, completes the steps of the method described above.

[0104] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 11 The computer system 1100 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 11 A schematic diagram of the structure of a computer system provided in an exemplary embodiment of this disclosure is shown.

[0105] Computer system 1100 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0106] like Figure 11 As shown, the computer system 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from a storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of the computer system 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0107] Multiple components in computer system 1100 are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information into computer system 1100. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, hard disks and optical disks. Communication unit 1109 allows computer system 1100 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0108] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1102 and / or communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0109] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0110] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0112] This disclosure also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the methods disclosed in the embodiments of this disclosure.

[0113] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0116] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0117] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0118] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for hybrid training of large models, characterized in that, include: In response to the launch of a hybrid training task for a target large model on a heterogeneous computing cluster, the global heterogeneous topology of the heterogeneous computing cluster is obtained; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type; Based on the hybrid training task, the heterogeneous parallel mode of the heterogeneous computing cluster is determined; wherein, the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. Based on the heterogeneous parallel mode and the device type of each processor, the hybrid training task is non-uniformly divided to generate sub-training tasks for each processor. During the training iteration, each processor is controlled to perform the hybrid training task on the target large model based on the sub-training task.

2. The method as described in claim 1, characterized in that, The process information also includes a network address and service port; the method further includes: Each of the processors is controlled to determine its own process information; The target processor among the plurality of processors is controlled to generate the global heterogeneous topology based on the process information of each processor, and the global heterogeneous topology is broadcast to each processor.

3. The method as described in claim 1, characterized in that, The step of non-uniformly partitioning the hybrid training task based on the heterogeneous parallel mode and the device type of each processor to generate sub-training tasks for each processor includes: Based on the heterogeneous parallel mode, the configuration parameters corresponding to the heterogeneous parallel mode are parsed and verified; based on the configuration parameters, the partitioning objects of the hybrid training task are determined. Based on the computing power of each processor, the partitioning object is non-uniformly divided to obtain sub-partitioning objects for each processor; wherein, processors of the same device type have the same computing power and the same sub-partitioning objects; The sub-partition objects are assigned to each of the processors, and sub-training tasks for each processor are generated based on the sub-partition objects.

4. The method as described in claim 3, characterized in that, If the heterogeneous parallel mode is the heterogeneous pipeline parallel mode, then the partitioning object is multiple network layers of the target large model, and the sub-partitioning object is at least one of the multiple pipeline stages obtained by non-uniformly partitioning the multiple network layers. At least one pipeline stage is matched with the computing power of each processor, and each pipeline stage includes at least one network layer. If the heterogeneous parallel mode is the heterogeneous data parallel mode, then the partitioning object is the training data of the target large model, and the sub-partitioning object is one of the multiple micro-batches of different sizes obtained by non-uniformly partitioning the training data, and each micro-batch is matched with the computing power of each processor.

5. The method as described in claim 3, characterized in that, Before generating sub-training tasks for each processor by non-uniformly partitioning the hybrid training task based on the heterogeneous parallel mode and the device type of each processor, the method further includes: If the heterogeneous parallel mode is the heterogeneous pipelined parallel mode, then the process groups are initialized in the order of tensor parallel group, data parallel group, and pipelined parallel group, and it is ensured that the device types of the processors in the same tensor parallel group and the same data parallel group are the same. If the heterogeneous parallel mode is a heterogeneous data parallel mode, then the process groups are initialized in the order of tensor parallel group, pipeline parallel group, and data parallel group, and it is ensured that the device types of the processors in the same tensor parallel group and the same pipeline parallel group are the same.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: During the training iteration, when data communication is performed between the two processors to complete forward propagation, backward propagation and gradient synchronization, it is determined whether the device types of the two processors are the same based on the global heterogeneous topology. When the two processors have the same device type, the homogeneous set communication library corresponding to the device type is invoked to perform the data communication. If the two processors have different device types, and the data communication is point-to-point, then the data communication is performed using a communication protocol selected by the amount of data in the data communication. If the data communication is a set communication, then the set communication is divided into three steps: homogeneous set communication within the homogeneous sub-communication groups of the two processors, point-to-point communication between homogeneous sub-communication groups, and broadcast communication within the homogeneous sub-communication groups. These steps are executed sequentially to complete global data synchronization. The homogeneous sub-communication groups are obtained by dividing the multiple processors according to their device types, and the number of homogeneous sub-communication groups is the same as the number of device types of the multiple processors.

7. A large-scale model hybrid training device, characterized in that, include: The acquisition module is used to acquire the global heterogeneous topology of the heterogeneous computing cluster in response to the start of a hybrid training task of a target large model in the heterogeneous computing cluster; wherein, the heterogeneous computing cluster includes multiple processors of different device types, and the global heterogeneous topology includes process information of each processor, and the process information includes at least the device type; The processing module is used to determine the heterogeneous parallel mode of the heterogeneous computing cluster based on the hybrid training task; wherein the heterogeneous parallel mode includes a heterogeneous pipeline parallel mode or a heterogeneous data parallel mode. The processing module is further configured to perform non-uniform partitioning of the hybrid training task based on the heterogeneous parallel mode and the device type of each processor, and generate sub-training tasks for each processor. The training module is used to control each of the processors to perform the hybrid training task on the target large model based on the sub-training task during the training iteration.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 6.

Images