Heterogeneous hardware cluster distributed training method and device, electronic equipment and medium

Abstract

The application provides a heterogeneous hardware cluster distributed training method and device, electronic equipment and medium, and belongs to the technical field of computers. The method comprises the following steps: determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware; constructing a process grid for executing a model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being called; constructing a heterogeneous communication group between the process grids based on the heterogeneous parallel training strategy and the mapping relationship; and calling the process grid to execute the model training task of the to-be-trained model. The technical scheme of the application can realize the mixing of any multiple hardware clusters, realize the efficient expansion of a heterogeneous cluster, can determine an optimal heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware, and provide a more flexible parallel strategy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to a heterogeneous hardware cluster distributed training method and device, electronic equipment and medium. BACKGROUND

[0002] Parallel training of large-scale models involves underlying hardware topology relationships that are usually one-to-one corresponding to parallel training strategies. Due to the complex topology relationship under multiple hardware, it is difficult to express the top-level parallel training strategy, and it is not easy to expand the cluster.

[0003] In the prior art, large-scale heterogeneous parallel training strategies usually include data parallel heterogeneous mode and pipeline parallel heterogeneous mode, but only one of the two parallel training strategies can be used according to the underlying hardware topology relationship, making the parallel training strategy single and unable to flexibly set the parallel training strategy based on heterogeneous hardware resources. Moreover, large-scale multi-dimensional mixed heterogeneous parallel training strategies also bring difficulties to communication libraries and communication algorithms, and the existing communication methods are not suitable for multi-dimensional mixed heterogeneous parallel training strategies. SUMMARY

[0004] The present application aims to provide a heterogeneous hardware cluster distributed training method and device, electronic equipment and medium, which aims to solve the problem of single parallel training strategy in the prior art, which cannot flexibly set the parallel training strategy based on the resources of heterogeneous hardware.

[0005] According to a first aspect of the present application, a heterogeneous hardware cluster distributed training method is provided, comprising:

[0006] Based on the to-be-trained model and the heterogeneous hardware, a heterogeneous parallel training strategy is determined;

[0007] Based on the heterogeneous parallel training strategy and the heterogeneous hardware, a process grid for executing a model training task of the to-be-trained model is constructed; wherein different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being called;

[0008] Based on the heterogeneous parallel training strategy and the mapping relationship, a heterogeneous communication group between the process grids is constructed;

[0009] The process grid is called to execute the model training task of the to-be-trained model.

[0010] In an optional embodiment, based on the to-be-trained model and the heterogeneous hardware, the heterogeneous parallel training strategy is determined, comprising:

[0011] Based on the composition of the to-be-trained model and the resource information of the heterogeneous hardware, the heterogeneous parallel training strategy is determined.

[0012] In an optional implementation, based on the heterogeneous parallel training strategy and the heterogeneous hardware, a process grid for executing a model training task of the model to be trained is constructed, comprising:

[0013] A process grid corresponding to each chip type included in the heterogeneous hardware is established;

[0014] In each group of process grids, chip types and serial numbers corresponding to other process grids are aggregated;

[0015] In each group of process grids, a mapping relationship between a chip physical identifier belonging to the chip type and a process logical identifier corresponding to the process grid is established according to the chip type and the serial number; wherein the chip physical identifiers of all chips of the same chip type correspond to the process logical identifier of the same group of process grids.

[0016] In an optional implementation, when the heterogeneous parallel training strategy includes a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy, based on the heterogeneous parallel training strategy and the mapping relationship, a heterogeneous communication group between process grids is constructed, comprising:

[0017] Based on the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy, a data correspondence relationship and a data rearrangement result between process grids corresponding to different pipeline stages are determined;

[0018] Based on the data correspondence relationship and the data rearrangement result, a heterogeneous communication group in which one process sends data to multiple processes and / or multiple processes send data to one process is established.

[0019] In an optional implementation, based on the heterogeneous parallel training strategy and the mapping relationship, the heterogeneous communication group between process grids is constructed, further comprising:

[0020] According to the data parallel dimension and the tensor parallel dimension on the heterogeneous hardware of different hardware types corresponding to different process grids, the associativity of data between different process grids is determined;

[0021] Based on the associativity of the data, a heterogeneous communication group required for global gradient normalization of the model to be trained in the training process is established.

[0022] In an optional implementation, the heterogeneous parallel training strategy includes a combination of one or more of the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy.

[0023] According to a second aspect of the present application, a heterogeneous hardware cluster distributed training device is provided, comprising:

[0024] determining, based on the to-be-trained model and the heterogeneous hardware, a heterogeneous parallel training strategy;

[0025] a first constructing module configured to construct, based on the heterogeneous parallel training strategy and the heterogeneous hardware, a process grid for executing a model training task of the to-be-trained model; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type having the mapping relationship after being invoked;

[0026] a second constructing module configured to construct, based on the heterogeneous parallel training strategy and the mapping relationship, a heterogeneous communication group between the process grids;

[0027] an invoking module configured to invoke the process grid to execute the model training task of the to-be-trained model.

[0028] In an optional implementation, the determining module comprises:

[0029] a first determining submodule configured to determine the heterogeneous parallel training strategy based on a constitution of the to-be-trained model and resource information of the heterogeneous hardware.

[0030] In an optional implementation, the first constructing module comprises:

[0031] a first establishing submodule configured to establish a process grid corresponding to a chip type included in the heterogeneous hardware in a one-to-one manner;

[0032] a converging submodule configured to converge, in each group of process grids, a chip type and a serial number corresponding to other process grids;

[0033] a second establishing submodule configured to establish, in each group of process grids, a mapping relationship between a chip physical identifier belonging to the chip type and a process logical identifier corresponding to the process grid according to the chip type and the serial number; chip physical identifiers of all chips of the same chip type correspond to a process logical identifier of the same group of process grids.

[0034] In an optional implementation, when the heterogeneous parallel training strategy comprises a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy, the second constructing module comprises:

[0035] a second determining submodule configured to determine, based on the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy, a data correspondence relationship and a data rearrangement result between process grids corresponding to different pipeline stages;

[0036] The third establishing sub-module is configured to establish a heterogeneous communication group in which one process sends data to multiple processes and / or multiple processes send data to one process based on the data correspondence and the data rearrangement result.

[0037] In an optional implementation, the second establishing module further includes:

[0038] The third determining sub-module is configured to determine the relevance of data between different process meshes according to the data parallel dimension and the tensor parallel dimension on the heterogeneous hardware of different hardware types corresponding to different process meshes.

[0039] The first establishing sub-module is configured to establish a heterogeneous communication group required for global gradient normalization of the to-be-trained model in the training process based on the relevance of the data.

[0040] In an optional implementation, the heterogeneous parallel training strategy includes a combination of one or more of a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy.

[0041] The third aspect of the present application provides an electronic device including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method of the first aspect.

[0042] The fourth aspect of the present application provides a computer-readable storage medium storing a plurality of instructions, wherein the plurality of instructions can be read by a processor and execute the method of the first aspect.

[0043] Compared with the related art, the technical solution of the present application has at least the following advantages:

[0044] Based on the above scheme, first, the present application proposes the concept of process mesh (ProcessMesh) to shield the difference between the underlying hardware, which can realize the mixing of any multiple hardware clusters and the efficient expansion of heterogeneous clusters. Second, the present application provides multiple optional heterogeneous parallel training strategies, and determines the preferred heterogeneous parallel training strategy based on the to-be-trained model and the heterogeneous hardware, which provides a more flexible parallel strategy and further improves the resource utilization rate of the heterogeneous cluster. Third, the present application constructs the heterogeneous communication group between processes in the process mesh according to the heterogeneous parallel dimension strategy, which solves the communication algorithm problem.

[0045] Other features and advantages of the present application will be illustrated in the following description, and some will become apparent from the description, or will be understood by those skilled in the art from the description and the accompanying drawings. The purposes and other advantages of the present application can be achieved and obtained by the structures and processes indicated in the description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings described below are only some embodiments of the present application, and the drawings can also be obtained by those of ordinary skill in the art without any creative work.

[0047] Figure 1 is a flow diagram of a heterogeneous hardware cluster distributed training method according to an exemplary embodiment of the present application.

[0048] Figure 2 is an effect diagram of a process grid according to an exemplary embodiment of the present application.

[0049] Figure 3 is a mapping relationship diagram of a process grid and a heterogeneous hardware according to an exemplary embodiment of the present application.

[0050] Figure 4 is a construction principle diagram of a heterogeneous communication group between different pipeline stages according to an exemplary embodiment of the present application.

[0051] Figure 5 is a construction principle diagram of an aggregated heterogeneous communication group in a global gradient normalization process according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work are within the scope of protection of the present application.

[0053] The method provided by the present application can be implemented in a terminal environment, which can include one or more of the following components: a processor, a memory and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0054] The processor can include one or more processing cores. The processor connects various parts in the entire terminal through various interfaces and lines, executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0055] The memory can include random access memory (RAM) and can also include read-only memory (ROM). The memory can be used to store instructions, programs, codes, code sets, or instructions.

[0056] The display screen is used to display the user interface of each application program.

[0057] In addition, those skilled in the art can understand that the structure of the terminal described above does not constitute a limitation on the terminal, and the terminal can include more or fewer components, or combine certain components, or different component arrangements. For example, the terminal also includes radio frequency circuitry, input units, sensors, audio circuitry, power supplies, and other components, which are not described here.

[0058] The application provides a heterogeneous hardware cluster distributed training method and device, wherein the method comprises the following steps: determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware; constructing a process mesh for executing a model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware; wherein different process meshes have a mapping relationship with different hardware types in the heterogeneous hardware; the process mesh is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being invoked; constructing a heterogeneous communication group between the process meshes based on the heterogeneous parallel training strategy and the mapping relationship; and invoking the process mesh to execute the model training task of the to-be-trained model. Based on the above scheme, first, the application proposes the concept of process mesh, which can shield the differences between the underlying hardware, realize the mixing of any multiple hardware clusters, and realize the efficient expansion of the heterogeneous cluster. Secondly, the application provides multiple optional heterogeneous parallel training strategies, determines an optimal heterogeneous parallel training strategy based on the to-be-trained model and the heterogeneous hardware, provides a more flexible parallel strategy, and further improves the resource utilization rate of the heterogeneous cluster. Thirdly, the application constructs a heterogeneous communication group between processes in the process mesh according to the heterogeneous parallel dimension strategy, and solves the communication algorithm problem.

[0059] The technical scheme of the application will be described below through specific embodiments.

[0060] Referring to the flowchart of Figure 1 The heterogeneous hardware cluster distributed training method provided by the application comprises the following steps:

[0061] Step 101: determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware;

[0062] Step 102: constructing a process grid for executing the model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being invoked;

[0063] Step 103: constructing a heterogeneous communication group between the process grids based on the heterogeneous parallel training strategy and the mapping relationship;

[0064] Step 104: invoking the process grid to execute the model training task of the to-be-trained model.

[0065] In this embodiment, the training method can be applied to a heterogeneous hardware cluster, so that a preferred heterogeneous parallel training strategy can be selected according to the to-be-trained model and the heterogeneous hardware executing the model training task of the to-be-trained model. In some embodiments, the optional heterogeneous parallel training strategy can include, but is not limited to, a combination of one or more of the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy.

[0066] In some embodiments, one or more combinations of the heterogeneous parallel training strategy can be selected to complete the training of the to-be-trained model.

[0067] After determining the heterogeneous parallel training strategy, a plurality of process grids for executing the model training task of the to-be-trained model can also be constructed according to the heterogeneous parallel training strategy and the heterogeneous hardware. A process grid can include a plurality of processes, and each process grid corresponds to one hardware type in the heterogeneous hardware, that is, one hardware type corresponds to one process grid, and different hardware types correspond to different process grids. In the model training process, the training task to be executed on the hardware of a certain hardware type is executed by invoking the process corresponding to the hardware type. It should be noted that the hardware type can be a chip type in the heterogeneous hardware cluster.

[0068] Referring to Figure 2 If the heterogeneous hardware cluster executing the model training task of the to-be-trained model has three different types of chips, three process grids A, B and C can be constructed, each process grid corresponding to a set of heterogeneous parallel training strategies and a chip sub-cluster of a chip type.

[0069] In addition, when training a model on mixed heterogeneous hardware, a heterogeneous communication library supporting multiple hardware needs to be supported, a corresponding communication group needs to be constructed for the heterogeneous parallel training strategy, and the data needs to be re-partitioned to ensure the correctness of the communication results. The communication across different process grids may involve heterogeneous communication. Therefore, the application creates a heterogeneous communication group between the process grids, which ensures normal communication between the training tasks executed on different process grids.

[0070] After creating the heterogeneous communication group between the process grids, the process grids can be called to execute the model training task of the to-be-trained model, so that the training of the to-be-trained model is completed by executing the corresponding process grid on the heterogeneous hardware at the corresponding stage.

[0071] In some optional implementations of the embodiment, step S101, i.e., determining the heterogeneous parallel training strategy based on the to-be-trained model and the heterogeneous hardware, can be implemented as follows:

[0072] Based on the composition of the to-be-trained model and the resource information of the heterogeneous hardware, the heterogeneous parallel training strategy is determined.

[0073] In this optional embodiment, the composition of the to-be-trained model can include the model structure and size of the to-be-trained model, and the resource information of the heterogeneous hardware can include key information of the heterogeneous hardware cluster resources, such as computing power, communication bandwidth, and topology relationship. Based on the composition of the heterogeneous hardware and the resource information of the heterogeneous hardware, the preferred heterogeneous parallel training strategy for training the to-be-trained model can be determined, such as using the pipeline parallel training strategy, the data parallel training strategy, and the tensor parallel training strategy at the same time when the model structure of the to-be-trained model is complex and the computing power of the heterogeneous hardware is strong.

[0074] In some optional implementations of the embodiment, step S102, i.e., constructing a process grid for executing the model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware, can be implemented as follows:

[0075] A process grid corresponding to each chip type included in the heterogeneous hardware is established;

[0076] In each group of process grids, the chip types and serial numbers corresponding to other process grids are aggregated;

[0077] In each group of process grids, a mapping relationship between the chip physical identifier belonging to the chip type and the process logical identifier corresponding to the process grid is established according to the chip type and the serial number. The chip physical identifiers of all chips of the same chip type correspond to the process logical identifier of the same group of process grids.

[0078] In the optional implementation, the process grid can be understood as a virtual representation of the physical chip, and the process grid needs to be mapped to the corresponding physical chip before the heterogeneous communication group is constructed and the training task is started.

[0079] As shown in Figure 3 , the heterogeneous hardware includes three different chip types, and thus three groups of process grids can be constructed, process grid A corresponding to the chips of the first chip type, process grid B corresponding to the chips of the second chip type, and process grid C corresponding to the chips of the third chip type. In some embodiments, each process in a group of process grids can correspond to one chip of the corresponding chip type (as shown by the small squares distributed in the array in Figure 3 , that is, the process grid corresponds to one chip type, and the processes in each group of process grids correspond to one chip.

[0080] It should be noted that, since the original training platform does not guarantee that the chip physical identifier of the chip and the sequence number (i.e., process logical identifier) of the process grid are one-to-one corresponding when it is initialized, the chip physical sequence number of the chip can be reordered and bound to the process logical identifier of the process grid in this embodiment. In this process, the chip physical identifiers of all chips can be aggregated first, and then reordered according to the chip type, and then bound according to the manner that a group of process grids correspond to the chips of the same chip type.

[0081] In this process, the chip physical identifier of each process can be aggregated to the current process using the All_Gather operation in each process, so that each process has the complete correspondence between the chip and the process.

[0082] In addition, when the process grid is reordered according to the chip type, all chips can be reordered according to the chip type as the primary order and the chip physical identifier as the secondary order, and a mapping relationship table between the process logical identifier and the chip physical identifier is established. In this way, the logical sequence number (corresponding to the process logical identifier) and the physical sequence number (i.e., the chip physical identifier) of the chip are bound, the communication group is constructed according to the logical sequence number, and the corresponding chip can be found through the mapping relationship table.

[0083] In some optional implementations of the present embodiment, when the heterogeneous parallel training strategy includes the pipeline parallel strategy, the data parallel strategy, and the tensor parallel strategy, the step S103, i.e., the step of constructing the heterogeneous communication group between the process grids based on the mapping relationship and the heterogeneous parallel training strategy, can be implemented in the following manner:

[0084] Based on the pipeline parallelism strategy, data parallelism strategy and tensor parallelism strategy, the data correspondence and data rearrangement results between process grids corresponding to different pipeline stages are determined.

[0085] Based on the data correspondence and the data rearrangement result, a heterogeneous communication group is established whereby a process sends data to multiple processes and / or multiple processes send data to one process.

[0086] In this optional implementation, heterogeneous communication exists between pipeline stages that span different process grids. Different pipelines rely on P2P (send / recv) communication, so the heterogeneous communication library needs to support P2P communication. However, the support of the heterogeneous communication library alone cannot guarantee data correctness.

[0087] like Figure 4 As shown, Model 1, 2, 3 and Model 4, 5 belong to two adjacent pipeline stages. These two pipeline stages adopt data parallelism and tensor parallelism strategies. Moreover, the data parallelism and tensor parallelism strategies are different between different pipeline stages. For example, Model 1, 2, 3 adopts a data parallelism strategy with two degrees of parallelism and a tensor parallelism strategy with three degrees of parallelism, while Model 4, 5 adopts a data parallelism strategy with four degrees of parallelism and a tensor parallelism strategy with two degrees of parallelism.

[0088] To ensure the correctness of communication, P2P heterogeneous communication groups can be created based on the correspondence and rearrangement results of the data, such as "one chip sending to multiple downstream chips" and "multiple chips sending to one downstream chip", to complete the forward and backward propagation process of training.

[0089] In some optional implementations of this embodiment, step S103, namely the step of constructing a heterogeneous communication group between process grids based on the heterogeneous parallel training strategy and the mapping relationship, can also be implemented in the following manner:

[0090] The correlation between data in different process grids is determined based on the data parallelism and tensor parallelism dimensions on heterogeneous hardware of different hardware types corresponding to different process grids.

[0091] Based on the correlation of the data, a heterogeneous communication group is established for global gradient normalization of the model to be trained during the training process.

[0092] In this alternative implementation, Allreduce communication needs to span all pipeline stages when calculating global gradient normalization.

[0093] Data parallelism on different hardware types of the heterogeneous hardware can be inconsistent, and the heterogeneous communication groups required for global gradient normalization can be established according to the correlation between data in different pipeline stages. Heterogeneous hardware with low data parallelism can correspond to multiple heterogeneous communication groups, such as Figure 5 As shown, two adjacent pipeline stages are the first pipeline stage corresponding to model 1, 2 and 3 layers and the second pipeline stage corresponding to model 4 and 5 layers, the data parallelism on the first process grid corresponding to the first pipeline stage is 2, and the data parallelism on the second process grid corresponding to the second pipeline stage is 4, then each communication group of the first process grid corresponds to two communication groups of the second process grid. In some embodiments, in order to ensure the correctness of the result of multiple aggregations (Allreduce), a reentrant communication mode can be constructed to ensure that the effect of multiple aggregations is equivalent to that of one aggregation, and the correctness of the global gradient normalization calculation result is ensured.

[0094] Correspondingly, the second aspect of the present application provides a heterogeneous hardware cluster distributed training device, comprising:

[0095] A determination module is configured to determine a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware;

[0096] A first construction module is configured to construct a process grid for executing a model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being invoked;

[0097] A second construction module is configured to construct a heterogeneous communication group between process grids based on the heterogeneous parallel training strategy and the mapping relationship;

[0098] A calling module is configured to call the process grid to execute the model training task of the to-be-trained model.

[0099] In an optional implementation, the determination module comprises:

[0100] A first determination submodule is configured to determine the heterogeneous parallel training strategy based on the composition of the to-be-trained model and resource information of the heterogeneous hardware.

[0101] In an optional implementation, the first construction module comprises:

[0102] A first establishment submodule is configured to establish a process grid corresponding to each chip type included in the heterogeneous hardware;

[0103] The convergence sub-module is configured to converge the chip types and the serial numbers of the other process grids in each group of process grids;

[0104] The second establishing sub-module is configured to establish, in each group of process grids, a mapping relationship between the chip physical identifiers of the chips of the chip types and the process logical identifiers of the process grids according to the chip types and the serial numbers; the chip physical identifiers of all chips of the same chip type correspond to the process logical identifier of the same group of process grids.

[0105] In optional embodiments, when the heterogeneous parallel training strategy includes a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy, the second constructing module includes:

[0106] The second determining sub-module is configured to determine, based on the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy, a data correspondence relationship and a data rearrangement result between the process grids corresponding to different pipeline stages;

[0107] The third establishing sub-module is configured to establish, based on the data correspondence relationship and the data rearrangement result, a heterogeneous communication group in which one process sends data to multiple processes and / or multiple processes send data to one process.

[0108] In optional embodiments, the second constructing module further includes:

[0109] The third determining sub-module is configured to determine, according to the data parallel dimension and the tensor parallel dimension on the heterogeneous hardware of different hardware types corresponding to different process grids, the associativity of data between the different process grids;

[0110] The fourth establishing sub-module is configured to establish, based on the associativity of data, a heterogeneous communication group required for global gradient normalization of the to-be-trained model in a training process.

[0111] In optional embodiments, the heterogeneous parallel training strategy includes a combination of one or more of the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy.

[0112] The above-described device corresponds to the heterogeneous hardware cluster distributed training method provided by the above-described embodiments, and specific details can be referred to the description of the heterogeneous hardware cluster distributed training method in the above-described embodiments, which will not be described herein again.

[0113] It can be understood that the circuit structure, name and parameter described in the above-described embodiments are only examples. Those skilled in the art can also easily combine and adjust the structural features of the above multiple embodiments according to the use needs, and the concept of the present application should not be limited to the specific details of the above examples.

[0114] The application also provides an electronic device, comprising a processor and a memory, the memory storing a plurality of instructions, and the processor being configured to read the instructions and perform any of the methods of the first aspect. The processor and the memory can be connected through a bus or other means, taking the bus connection as an example. The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination thereof.

[0115] The memory, as a non-transitory computer-readable storage medium, can be configured to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions / modules corresponding to the methods in the embodiments of the application. The processor can execute various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implement the methods in the method embodiments.

[0116] The memory can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0117] As another aspect, the application also provides a computer-readable storage medium, which can be the computer-readable storage medium included in the apparatus described in the above embodiments; or can exist independently and not be assembled into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the application.

[0118] Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood that modifications can be made to the foregoing embodiments, or additional implementations can be implemented, without departing from the spirit and scope of the embodiments.

Claims

1. A heterogeneous hardware cluster distributed training method, characterized in that, The method comprises the steps of: determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware; the heterogeneous parallel training strategy comprises a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy; based on the heterogeneous parallel training strategy and the heterogeneous hardware, a process grid for executing a model training task of the to-be-trained model is constructed; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; the process grid is executed on the heterogeneous hardware corresponding to the hardware type with the mapping relationship after being called; based on the heterogeneous parallel training strategy and the mapping relationship, a heterogeneous communication group between the process grids is constructed; the process grid is called to execute the model training task of the to-be-trained model; in the model training process, the training task to be executed on a hardware of a certain hardware type is executed by calling a process having a mapping relationship with the hardware type; wherein, based on the heterogeneous parallel training strategy and the mapping relationship, the heterogeneous communication group between the process grids is constructed, comprising: based on the pipeline parallel strategy, the data parallel strategy and the tensor parallel strategy, a data correspondence relationship and a data rearrangement result between process grids corresponding to different pipeline stages are determined; based on the data correspondence relationship and the data rearrangement result, a heterogeneous communication group in which one process sends data to multiple processes and / or multiple processes send data to one process is established.

2. The heterogeneous hardware cluster distributed training method of claim 1, wherein, determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware, comprising: based on the composition of the to-be-trained model and the resource information of the heterogeneous hardware, the heterogeneous parallel training strategy is determined. 3.The heterogeneous hardware cluster distributed training method of claim 1, wherein, based on the heterogeneous parallel training strategy and the heterogeneous hardware, a process grid for executing a model training task of the to-be-trained model is constructed, comprising: establishing a process grid corresponding to each chip type included in the heterogeneous hardware; gathering the chip types and serial numbers corresponding to other process grids in each group of process grids; establishing a mapping relationship between the chip physical identifier belonging to the chip type and the process logical identifier corresponding to the process grid according to the chip type and serial number in each group of process grids; wherein the chip physical identifiers of all chips of the same chip type correspond to the process logical identifier of the same group of process grids.

4. The heterogeneous hardware cluster distributed training method of claim 1, wherein, based on the heterogeneous parallel training strategy and the mapping relationship, a heterogeneous communication group between the process grids is constructed, further comprising: determining the associativity of data between different process grids according to the data parallel dimension and the tensor parallel dimension on the heterogeneous hardware of different hardware types corresponding to different process grids; based on the associativity of the data, a heterogeneous communication group required for global gradient normalization of the to-be-trained model in the training process is established.

5. A heterogeneous hardware cluster distributed training device, characterized in that, The method comprises the steps of: a determination module for determining a heterogeneous parallel training strategy based on a to-be-trained model and heterogeneous hardware; the heterogeneous parallel training strategy comprises a pipeline parallel strategy, a data parallel strategy and a tensor parallel strategy; The first construction module is configured to construct a process grid for executing a model training task of the to-be-trained model based on the heterogeneous parallel training strategy and the heterogeneous hardware; different process grids have a mapping relationship with different hardware types in the heterogeneous hardware; and the process grid is executed on the heterogeneous hardware corresponding to the hardware type having the mapping relationship after being invoked. The second construction module is configured to construct a heterogeneous communication group between the process grids based on the heterogeneous parallel training strategy and the mapping relationship. The calling module is configured to invoke the process grid to execute the model training task of the to-be-trained model; and in a model training process, a training task to be executed on hardware of a hardware type is executed by invoking a process having a mapping relationship with the hardware type. The second construction module includes: The second determination submodule is configured to determine a data correspondence relationship and a data rearrangement result between the process grids corresponding to different pipeline stages based on the pipeline parallel strategy, the data parallel strategy, and the tensor parallel strategy. The third establishment submodule is configured to establish a heterogeneous communication group in which one process sends data to multiple processes and / or multiple processes send data to one process based on the data correspondence relationship and the data rearrangement result.

6. The heterogeneous hardware cluster distributed training apparatus of claim 5, wherein, The determination module includes: The first determination submodule is configured to determine the heterogeneous parallel training strategy based on a composition of the to-be-trained model and resource information of the heterogeneous hardware.

7. The heterogeneous hardware cluster distributed training apparatus of claim 5, wherein, The first construction module includes: The first establishment submodule is configured to establish a process grid corresponding to each chip type included in the heterogeneous hardware. The aggregation submodule is configured to aggregate chip types and serial numbers of other process grids in each group of process grids. The second establishment submodule is configured to establish a mapping relationship between a chip physical identifier belonging to the chip type and a process logical identifier corresponding to the process grid according to the chip type and the serial number in each group of process grids; and chip physical identifiers of all chips of the same chip type correspond to a process logical identifier of the same group of process grids.

8. The heterogeneous hardware cluster distributed training apparatus of claim 5, wherein, The second construction module further includes: The third determination submodule is configured to determine data association between different process grids according to data parallel dimensions and tensor parallel dimensions on the heterogeneous hardware of different hardware types corresponding to different process grids. The third establishment submodule is configured to establish a heterogeneous communication group required for global gradient normalization of the to-be-trained model in a training process based on the data association.

9. An electronic device, comprising: The computer readable storage medium stores a plurality of instructions, and the plurality of instructions can be read and executed by the processor to perform the heterogeneous hardware cluster distributed training method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, and the plurality of instructions can be read and executed by the processor to perform the heterogeneous hardware cluster distributed training method according to any one of claims 1-4.

Images