A Distributed Parallel Training Method and System for Heterogeneous Computing Clusters

Abstract

The distributed parallel training method for heterogeneous computing clusters provided by the present invention is to obtain the communication overhead of N computing devices on each pipeline stage, combine the computing parameters of each pipeline stage, determine the network topology of N computing devices; obtain the model partitioning method through the pre-trained neural network model, network topology and preset algorithm; according to the model partitioning method, use the pipeline parallel method between N different types of computing devices, and use the data parallel method between M virtual machines to iteratively train the neural network model. Compared with the prior art, the present invention combines the network topology and model partitioning method of N different types of computing devices to balance the computing time of each pipeline stage, and can improve the efficiency of the hybrid parallel method in a heterogeneous computing cluster environment.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network model training, and in particular to a distributed parallel training method and system for heterogeneous computing clusters. Background Art

[0002] The training of a neural network model (hereinafter referred to as the "model") requires inputting data into the model in batches for calculation, and then updating the model parameters based on the calculation results. This process will be repeated until the model achieves the expected effect on the actual task. Large models can achieve better results on actual tasks by increasing the number of model parameters and the amount of data used in the training process.

[0003] The large number of parameters used by large models and the training mode of large amounts of data make it impossible for single-machine hardware devices to meet the needs of model training in terms of computing power and storage capacity. Therefore, it has become a consensus to use distributed training methods to train large models. Distributed training methods can use computing devices on multiple machines to train models. While meeting the needs of model training, it can also shorten the model training cycle. On the other hand, computing devices are rapidly iterating with the advancement of science and technology, and computing clusters composed of various types of computing devices (commonly referred to as "heterogeneous computing clusters") have begun to appear in large numbers. The acceleration effect of existing distributed training technologies on heterogeneous computing clusters is average and their applicability is low.

[0004] Data parallel method and pipeline parallel method are currently common hybrid parallel methods. They have low requirements for the network bandwidth of the server cluster and a wide range of applications. However, the current hybrid parallel training methods pay little attention to heterogeneous computing clusters, and the hybrid parallel training methods implemented based on heterogeneous computing clusters are not efficient.

[0005] Therefore, it is an urgent problem to be solved by those skilled in the art to provide a distributed parallel training method and system for heterogeneous computing clusters that solves the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a distributed parallel training method for heterogeneous computing clusters. The method has clear logic, is safe, effective, reliable and easy to operate, and can improve the efficiency of hybrid parallel methods in a heterogeneous computing cluster environment.

[0007] Based on the above objectives, the technical solution provided by the present invention is as follows:

[0008] A distributed parallel training method for heterogeneous computing clusters includes the following steps:

[0009] According to the computing capabilities of the N different types of computing devices in each virtual machine, a first parameter of each pipeline stage in the M virtual machines is set to obtain the communication overhead of the N different types of computing devices on each pipeline stage;

[0010] Determine the network topology of the N different types of computing devices in each virtual machine according to the first parameters of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage;

[0011] Obtaining a model partitioning method according to the pre-trained neural network model, the network topology of N different types of computing devices in each virtual machine, and a preset algorithm;

[0012] According to the model partitioning method, the neural network model is iteratively trained using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines;

[0013] Each virtual machine includes one computing device of different types, and a plurality of the virtual machines constitute a heterogeneous computing cluster.

[0014] Preferably, according to the computing capabilities of the N different types of computing devices in each virtual machine, setting the first parameter of each pipeline stage in the M virtual machines and obtaining the communication overhead of the N different types of computing devices on each pipeline stage comprises the following steps:

[0015] Obtain the computing capabilities of N different types of computing devices in each virtual machine;

[0016] The computing time of each pipeline stage in the M virtual machines is set to be the same, so that the size of the neural network model of the N different types of computing devices on each pipeline stage is proportional to the computing power of the N different types of computing devices;

[0017] Setting communication synchronization between the same pipeline stages on different virtual machines so that the communication overhead of the N different types of computing devices on each pipeline stage is proportional to the size of the neural network model of the N different types of computing devices;

[0018] According to the computing capabilities of the N different types of computing devices and the above proportional relationship, the communication overheads of the N different types of computing devices on each pipeline stage are obtained.

[0019] Preferably, determining the network topology of the N different types of computing devices in each virtual machine according to the first parameter of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage comprises the following steps:

[0020] Calculate and obtain a communication ratio between a first parameter of each pipeline stage in the M virtual machines and a communication overhead of N different types of computing devices on each pipeline stage;

[0021] According to the communication ratio from large to small, the first pipeline stage to the Nth pipeline stage are set for the N different types of computing devices on each pipeline stage respectively, as the network topology of the N different types of computing devices in each virtual machine.

[0022] Preferably, obtaining a pre-trained neural network model comprises the following steps:

[0023] Provide L training samples and configure the second parameter;

[0024] A samples are selected from the L training samples, and the neural network model is pre-trained according to the second parameter. After the pre-trained neural network model is obtained, the third parameters of each neural network layer are obtained.

[0025] Preferably, the method of obtaining the model partitioning method according to the pre-trained neural network model, the network topology of the N different types of computing devices in each virtual machine and the preset algorithm comprises the following steps:

[0026] Preset simulator for model partitioning;

[0027] Inputting the third parameters of each neural network layer and the network topology of N different types of computing devices in each virtual machine into the simulator;

[0028] The simulator obtains the model partitioning method of dividing the neural network model into N parts according to an integer programming algorithm.

[0029] Preferably, according to the model division method, the neural network model is iteratively trained using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines, specifically:

[0030] Select b samples from the L training samples;

[0031] In each iterative training process, b training samples are used as a batch to perform forward calculation, backward calculation and communication of the neural network model until the neural network model reaches a preset accuracy;

[0032] Among them, L=a+b.

[0033] Preferably, in each iterative training process, using b training samples as a batch to perform forward calculation, backward calculation and communication of the neural network model comprises the following steps:

[0034] Divide the b training samples into J small batches of training samples, input the current small batch of training samples into each virtual machine, and construct each pipeline level index;

[0035] Divide the small batch training samples into K micro-batch training samples, input them into the computing device where the first pipeline stage is located in sequence for forward calculation, backward calculation and communication, and construct a micro-batch index for each computing device;

[0036] During iterative training using data parallelism between M virtual machines, the forward calculation, backward calculation and communication of each pipeline level are scheduled according to the number of micro-batches K, the number of pipeline stages N, the current pipeline level index and the current micro-batch index, and the idle time of the pipeline level is used to hide the recalculation overhead.

[0037] A distributed parallel training system for heterogeneous computing clusters, comprising:

[0038] A communication overhead module, used to set the first parameter of each pipeline stage in the M virtual machines according to the computing capabilities of the N different types of computing devices in each virtual machine, and obtain the communication overhead of the N different types of computing devices on each pipeline stage;

[0039] A network topology module, configured to determine a network topology of the N different types of computing devices in each of the M virtual machines based on the first parameters of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage;

[0040] A model partitioning module, used to obtain a model partitioning method according to the pre-trained neural network model, the network topology of N different types of computing devices in each virtual machine, and a preset algorithm;

[0041] An iterative training module, used for iteratively training the neural network model using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines according to the model partitioning method;

[0042] Each virtual machine includes one computing device of different types, and a plurality of the virtual machines constitute a heterogeneous computing cluster.

[0043] The distributed parallel training method for heterogeneous computing clusters provided by the present invention is to determine the network topology of N computing devices by obtaining the communication overhead of N computing devices on each pipeline stage and combining the computing parameters of each pipeline stage; obtain the model partitioning method through the pre-trained neural network model, network topology and preset algorithm; according to the model partitioning method, use the pipeline parallel method between N different types of computing devices, and use the data parallel method between M virtual machines to iteratively train the neural network model.

[0044] Compared with the prior art, the present invention combines the network topology and model partitioning method of N different types of computing devices to balance the computing time of each pipeline stage, and can improve the efficiency of the hybrid parallel method in a heterogeneous computing cluster environment.

[0045] The present invention also provides a distributed parallel training system for heterogeneous computing clusters. Since it solves the same technical problem as the method and belongs to the same technical concept, it should have the same beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a distributed parallel training method for heterogeneous computing clusters provided by an embodiment of the present invention;

[0048] Figure 2 A specific implementation process diagram of a distributed parallel training method for heterogeneous computing clusters provided by an embodiment of the present invention;

[0049] Figure 3 A flowchart of step S1 provided in an embodiment of the present invention;

[0050] Figure 4 A flowchart of step S2 provided in an embodiment of the present invention;

[0051] Figure 5 A flowchart of step S3 provided in an embodiment of the present invention;

[0052] Figure 6 A flowchart of each iterative training provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the structure of a distributed parallel training system for heterogeneous computing clusters provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The embodiments of the present invention are written in a progressive manner.

[0056] The embodiment of the present invention provides a distributed parallel training method and system for heterogeneous computing clusters, which mainly solves the technical problem that the hybrid parallel training method in the prior art pays little attention to the heterogeneous computing clusters and the hybrid parallel training method based on the heterogeneous computing clusters is not efficient.

[0057] like Figure 1 As shown, a distributed parallel training method for heterogeneous computing clusters includes the following steps:

[0058] S1. According to the computing capabilities of N different types of computing devices in each virtual machine, set the first parameter of each pipeline stage in M ​​virtual machines, and obtain the communication overhead of N different types of computing devices on each pipeline stage;

[0059] S2. Determine the network topology of N different types of computing devices within each virtual machine according to the communication overhead of the first parameter of each pipeline stage in the M virtual machines and the N different types of computing devices on each pipeline stage;

[0060] S3. Obtain a model partitioning method according to the pre-trained neural network model, the network topology of N different types of computing devices in each virtual machine, and a preset algorithm;

[0061] S4. Iteratively train the neural network model using a pipeline parallel approach between N different types of computing devices and a data parallel approach between M virtual machines according to the model partitioning method;

[0062] Each virtual machine contains one computing device of different types, and multiple virtual machines constitute a heterogeneous computing cluster.

[0063] In step S1, the communication overhead of the N computing devices on each pipeline stage is calculated and obtained by combining the computing capabilities of the N computing devices in each virtual machine and the first parameters of each pipeline stage;

[0064] In step S2, the communication ratio is calculated according to the first parameter of each pipeline stage in combination with the communication overhead of the N computing devices on each pipeline stage, so as to determine the network topology of the N different computing devices in each virtual machine;

[0065] In step S3, the neural network is pre-trained in advance to obtain relevant parameters, and the model partitioning method is obtained by combining the network topology of N different computing devices and the preset algorithm;

[0066] In step S4, according to the model partitioning method, pipeline parallelism is used between N computing devices, and data parallelism is used between M virtual machines to iteratively train the neural network model until the target accuracy is met.

[0067] The specific implementation process of the present invention is as follows Figure 2 shown.

[0068] like Figure 3 As shown, preferably, step S1 includes the following steps:

[0069] A1. Obtain the computing capabilities of N different types of computing devices in each virtual machine;

[0070] A2. Setting the computing time of each pipeline stage in the M virtual machines to be the same, so that the size of the neural network model of the N different types of computing devices on each pipeline stage is proportional to the computing power of the N different types of computing devices;

[0071] A3. Setting the communication synchronization between the same pipeline stages on different virtual machines so that the communication overhead of N different types of computing devices on each pipeline stage is proportional to the size of the neural network model of the N different types of computing devices;

[0072] A4. According to the computing capabilities of N different types of computing devices and the above proportional relationship, the communication overhead of N different types of computing devices on each pipeline stage is obtained.

[0073] In step A1, the computing capabilities of N different types of computing devices in each virtual machine are obtained by setting corresponding parameters of each virtual machine in the virtual environment;

[0074] It should be noted that when selecting or evaluating a device, attention should be paid to the two indicators of floating-point operation speed (FLOPS) and trillion operations (TOPS). Among them, floating-point operation speed (FLOPS) refers to how many floating-point operations can be performed per second. Floating-point operations are a method of operation used to calculate complex mathematical models (such as neural networks). Therefore, FLOPS is an important indicator for evaluating devices for deep learning. And trillion operations (TOPS) is another unit of measurement for computing power, which refers to how many trillion operations can be performed per second. Although TOPS also involves mathematical operations, it mainly focuses on the overall computing power of the device, which is not limited to floating-point operations, but also includes other types of operations;

[0075] In this embodiment, the computing capabilities of N different types of computing devices may be determined by evaluating one of the above two indicators;

[0076] In step A2 and step A3, in order to ensure the parallel operation efficiency of the pipeline, the first parameters of each pipeline stage are set, that is, the calculation time of each pipeline stage is the same and the same pipeline stages communicate with each other to ensure data synchronization; thereby, the size of the neural network model stored by the computing device on each pipeline stage is proportional to the computing power of the computing device, and the communication overhead of the computing device is proportional to the size of the neural network model stored by the computing device on each pipeline stage;

[0077] In step A4, according to the above proportional relationship, the computing capabilities of the N different types of computing devices in each virtual machine are known, and the communication overheads of the N different types of computing devices on each pipeline stage can be obtained.

[0078] like Figure 4 As shown, preferably, step S2 includes the following steps:

[0079] B1. Calculate and obtain the communication ratio between the first parameter of each pipeline stage in the M virtual machines and the communication overhead of N different types of computing devices on each pipeline stage;

[0080] B2. According to the communication ratio from large to small, set the first pipeline stage to the Nth pipeline stage for the N different types of computing devices on each pipeline stage respectively, as the network topology of the N different types of computing devices in each virtual machine.

[0081] In step B1 to step B2, the communication ratio between the single reverse calculation time in the calculation time of each pipeline level and the communication overhead of N different types of computing devices on each pipeline level is calculated, and the pipeline levels of different types of computing devices in the virtual machine are set in descending order according to the ratio;

[0082] In this embodiment, the computing device with the largest communication ratio is set as the first pipeline stage, the computing device with the second largest communication ratio is set as the second pipeline stage, and so on, the computing device with the smallest communication ratio is set as the Nth pipeline stage.

[0083] Preferably, obtaining a pre-trained neural network model comprises the following steps:

[0084] Provide L training samples and configure the second parameter;

[0085] A samples are selected from L training samples, and the neural network model is pre-trained according to the second parameter. After the pre-trained neural network model is obtained, the third parameter of each neural network layer is obtained.

[0086] In actual application, L training samples are provided, a small portion a of the training samples are selected from them, and the second parameters of the computing nodes (specifically the number of computing devices, network bandwidth and other parameters) are configured to pre-train the neural network model, and the computing time of each neural network layer is obtained from the pre-trained neural network model.

[0087] In this embodiment, specifically: the training sample is regarded as a batch of data, which is divided into M micro-batches, N computing devices of different types are organized into a pipeline, and the model is divided by layer and placed on each pipeline level. There is no need to consider the uniformity of the division, and it is only necessary to ensure that the pipeline can run normally. Then the divided micro-batches are input into the pipeline, and then the forward and reverse calculation time of each layer of the model on each type of device can be obtained.

[0088] like Figure 5 As shown, preferably, step S3 includes the following steps:

[0089] C1. Preset simulator for model partitioning;

[0090] C2. Input the third parameters of each neural network layer and the network topology of N different types of computing devices in each virtual machine into the simulator;

[0091] C3. The simulator obtains a model partitioning method for dividing the neural network model into N parts according to an integer programming algorithm.

[0092] In step C1 to step C3, a simulator for model partitioning is designed in advance, and the computing time of each neural network layer and the network topology of N different types of computing devices in each virtual machine are input into the simulator. The simulator solves the optimal way to divide the neural network model into N parts according to the set integer programming algorithm, thereby maximizing the efficiency of hybrid parallel training.

[0093] In this embodiment, the simulator preset for model partitioning is specifically: after determining the network topology of N different types of devices, the forward and reverse calculation time of each pipeline stage in the pipeline can be determined in combination with the calculation time of each neural network layer on each type of device and the current layer partitioning strategy, because the pipeline stage is actually a certain type of device, and there is a current model partitioning strategy (that is, knowing how many network layers there are on the device) and the calculation time of each layer of the neural network on the device.

[0094] Define the pillar pipeline stage in the pipeline as the pipeline stage with the largest forward and backward calculation time among all pipeline stages.

[0095] Assume that the current model partitioning strategy is , s is the number of pipeline stages, mi represents the number of network layers in the current pipeline stage. Note that the network topology has been determined at this point, so the type of equipment used by mi is known.

[0096] Use array L to represent array The prefix sum of , , , , …, . Then the total iteration time can be expressed as:

[0097] ;

[0098] in, Represents a micro-batch of data in The sum of the forward and reverse calculation times of the pipeline stages, Represents the data parallel AllReduce overhead of the first pipeline stage when the load is m1.

[0099] Based on the formal representation of the total iteration time above, by adding constraints, the optimal model partitioning strategy solution is modeled as an integer programming problem, as shown below:

[0100]

[0101] (1);

[0102] (2);

[0103] (3);

[0104] in,

[0105] It represents the memory usage of the i-th pipeline model state.

[0106] It represents the memory usage of a single activation value of the i-th pipeline stage.

[0107] The constraint expressed by Formula 1 is that the sum of the number of model layers of all pipeline stages is equal to the total number of layers in the model;

[0108] The constraint expressed by formula 2 is that the video memory overhead of a pipeline stage cannot exceed the video memory of the device used by the pipeline stage;

[0109] The constraint expressed by Formula 3 is that the forward-backward calculation time of a micro-batch of each pipeline stage is less than the forward-backward calculation time of the pillar pipeline stage.

[0110] Preferably, step S4 is specifically:

[0111] Select b training samples from L;

[0112] In each iterative training process, b training samples are used as a batch to perform forward calculation, backward calculation and communication of the neural network model until the neural network model reaches the preset accuracy;

[0113] Among them, L=a+b.

[0114] In actual application, after completing the division of the neural network model and the division of computing devices in the first step, the training calculation step is entered. The training calculation step is the same as the traditional training process, using an iterative training method until the neural network model reaches the target accuracy. In each iterative training process, the remaining b training samples are used as a batch to perform forward calculation, backward calculation and communication of the neural network model.

[0115] like Figure 6 As shown, preferably, in each iterative training process, b training samples are used as a batch to perform forward calculation, backward calculation and communication of the neural network model, including the following steps:

[0116] D1. Divide the b training samples into J small batches of training samples, input the current small batch of training samples into each virtual machine, and construct each pipeline level index;

[0117] D2. Divide the small batch training samples into K micro-batch training samples, input them into the computing device where the first pipeline stage is located in sequence for forward calculation, backward calculation and communication, and construct the micro-batch index of each computing device;

[0118] D3. During iterative training using data parallelism between M virtual machines, the forward computation, backward computation, and communication of each pipeline stage are scheduled according to the number of micro-batches K, the number of pipeline stages N, the current pipeline stage index, and the current micro-batch index, and the idle time of the pipeline stage is used to hide the recomputation overhead.

[0119] In step D1 to step D3, a batch of data is divided into J small batches of data, and each virtual machine uses one small batch of data; in each virtual machine, the currently used small batch of data is divided into K micro-batch data, which are sequentially input into the computing device where the first pipeline stage is located for forward calculation, backward calculation and communication, and at the same time, the index of each pipeline stage and the micro-batch index of each computing device are constructed; in the process of running the pipeline parallel method in each virtual machine, the total number of micro-batches K, the total number of pipeline stages N, the index of the current pipeline stage, and the micro-batch index of the current forward calculation and the micro-batch index of the reverse calculation are used to schedule the calculation process of each pipeline stage, and the inherent idle time of the pipeline is used to hide the heavy calculation overhead.

[0120] In this embodiment, the following steps are used to implement the scheduling: let the current micro-batch index of the forward calculation be forward_index, the micro-batch index of the reverse calculation be backward_index, the current pipeline level index be i, recompute_set represents the set of micro-batch indexes that need to be recomputed, the total number of micro-batches K, and the total number of pipeline levels N. The scheduling process is as follows:

[0121] 1. If forward_index is less than or equal to 3 times (Ni), go to step 2, otherwise go to step 6.

[0122] 2. If K is less than or equal to 3 times (Ni) or forward_index is less than or equal to 2 times (Ni), go to step 3, otherwise go to step 4.

[0123] 3. Perform forward computation on the micro-batch indexed as forward_index, then release its activation value, add forward_index to recompute_set, and proceed to step 5.

[0124] 4. After performing the forward calculation of the micro-batch indexed as forward_index, do not release its activation value and go to step 5.

[0125] 5. Increase the index forward_index by 1 and return to step 1.

[0126] 6. If backward_index is less than or equal to K, go to step 7, otherwise go to step 14.

[0127] 7. If backward_index is in the set recompute_set, go to step 8, otherwise go to step 9.

[0128] 8. Recalculate the micro-batch with index backward_index and go to step 13.

[0129] 9. If (K-forward_index) is less than (Ni), go to step 10, otherwise go to step 11.

[0130] 10. Perform forward computation on the micro-batch indexed as forward_index and release its activation value, then add forward_index to recompute_set and proceed to step 12.

[0131] 11. Perform forward calculation on the micro-batch with index forward_index and retain its activation value, and go to step 12.

[0132] 12. Increase the index forward_index by 1 and go to step 13.

[0133] 13. Perform reverse calculation on the micro-batch with index backward_index, then increase backward_index by 1 and return to step 6.

[0134] 14. End the scheduling.

[0135] like Figure 7 As shown, a distributed parallel training system for heterogeneous computing clusters includes:

[0136] A communication overhead module, used to set the first parameter of each pipeline stage in the M virtual machines according to the computing capabilities of the N different types of computing devices in each virtual machine, and obtain the communication overhead of the N different types of computing devices on each pipeline stage;

[0137] A network topology module, for determining a network topology of N different types of computing devices in each virtual machine according to a first parameter of each pipeline stage in the M virtual machines and a communication overhead of the N different types of computing devices on each pipeline stage;

[0138] A model partitioning module, used to obtain a model partitioning method according to a pre-trained neural network model, a network topology of N different types of computing devices in each virtual machine, and a preset algorithm;

[0139] An iterative training module, used for iteratively training the neural network model using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines according to the model partitioning method;

[0140] Each virtual machine contains one computing device of different types, and multiple virtual machines constitute a heterogeneous computing cluster.

[0141] In actual application, the present invention also discloses a distributed parallel training system for heterogeneous computing clusters, which is provided with a communication overhead module, a network topology module, a model partitioning module and an iterative training module. The above modules are used to implement the various steps of the distributed parallel training method for heterogeneous computing clusters, which will not be repeated here.

[0142] In the embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0143] In addition, all functional modules in the embodiments of the present invention may be integrated into one processor, or each module may be a separate device, or two or more modules may be integrated into one device; each functional module in the embodiments of the present invention may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0144] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, the steps of the above method embodiment are executed; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes.

[0145] It should be understood that the use of "system", "device", "unit" and / or "module" in this application is only a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0146] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0147] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0148] If a flow chart is used in the present application, the flow chart is used to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or a certain step or several steps of operations can be removed from these processes.

[0149] The above is a detailed introduction to a distributed parallel training method and system for heterogeneous computing clusters provided by the present invention. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined in this article can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this article, but will conform to the widest scope consistent with the principles and novel features disclosed in this article.

Claims

1. A distributed parallel training method for heterogeneous computing clusters, characterized in that: The steps include: According to the computing capabilities of the N different types of computing devices in each virtual machine, a first parameter of each pipeline stage in the M virtual machines is set to obtain the communication overhead of the N different types of computing devices on each pipeline stage; Determine the network topology of the N different types of computing devices in each virtual machine according to the first parameters of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage; Obtaining a model partitioning method according to the pre-trained neural network model, the network topology of N different types of computing devices in each virtual machine, and a preset algorithm; According to the model partitioning method, the neural network model is iteratively trained using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines; Each virtual machine includes one computing device of different types, and a plurality of the virtual machines constitute a heterogeneous computing cluster; The method of setting the first parameter of each pipeline stage in the M virtual machines according to the computing capabilities of the N different types of computing devices in each virtual machine and obtaining the communication overhead of the N different types of computing devices on each pipeline stage includes the following steps: Obtain the computing capabilities of N different types of computing devices in each virtual machine; The computing time of each pipeline stage in the M virtual machines is set to be the same, so that the size of the neural network model of the N different types of computing devices on each pipeline stage is proportional to the computing power of the N different types of computing devices; Setting communication synchronization between the same pipeline stages on different virtual machines so that the communication overhead of the N different types of computing devices on each pipeline stage is proportional to the size of the neural network model of the N different types of computing devices; According to the computing capabilities of the N different types of computing devices and the above proportional relationship, obtaining the communication overheads of the N different types of computing devices on each pipeline stage; Determining the network topology of the N different types of computing devices in each virtual machine according to the first parameters of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage comprises the following steps: Calculate and obtain a communication ratio between a first parameter of each pipeline stage in the M virtual machines and a communication overhead of N different types of computing devices on each pipeline stage; According to the communication ratio from large to small, the first pipeline stage to the Nth pipeline stage are set for the N different types of computing devices on each pipeline stage respectively, as the network topology of the N different types of computing devices in each virtual machine.

2. The distributed parallel training method for heterogeneous computing clusters according to claim 1, characterized in that: Obtaining a pre-trained neural network model includes the following steps: Provide L training samples and configure the second parameter; A samples are selected from the L training samples, and the neural network model is pre-trained according to the second parameter. After the pre-trained neural network model is obtained, the third parameters of each neural network layer are obtained.

3. The distributed parallel training method for heterogeneous computing clusters according to claim 2, characterized in that: The method of obtaining a model partitioning method according to the pre-trained neural network model, the network topology of the N different types of computing devices in each virtual machine, and a preset algorithm comprises the following steps: Preset simulator for model partitioning; Inputting the third parameters of each neural network layer and the network topology of N different types of computing devices in each virtual machine into the simulator; The simulator obtains the model partitioning method of dividing the neural network model into N parts according to an integer programming algorithm.

4. The distributed parallel training method for heterogeneous computing clusters according to claim 3, characterized in that: According to the model division method, the neural network model is iteratively trained using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines, specifically: Select b samples from the L training samples; In each iterative training process, b training samples are used as a batch to perform forward calculation, backward calculation and communication of the neural network model until the neural network model reaches a preset accuracy; Among them, L=a+b.

5. The distributed parallel training method for heterogeneous computing clusters according to claim 4, characterized in that: The method of using b training samples as a batch in each iterative training process to perform forward calculation, backward calculation and communication of the neural network model includes the following steps: Divide the b training samples into J small batches of training samples, input the current small batch of training samples into each virtual machine, and construct each pipeline level index; Divide the small batch training samples into K micro-batch training samples, input them into the computing device where the first pipeline stage is located in sequence for forward calculation, backward calculation and communication, and construct a micro-batch index for each computing device; During iterative training using data parallelism between M virtual machines, the forward calculation, backward calculation and communication of each pipeline level are scheduled according to the number of micro-batches K, the number of pipeline stages N, the current pipeline level index and the current micro-batch index, and the idle time of the pipeline level is used to hide the recalculation overhead.

6. A distributed parallel training system for heterogeneous computing clusters, characterized in that: include: A communication overhead module, used to set the first parameter of each pipeline stage in the M virtual machines according to the computing capabilities of the N different types of computing devices in each virtual machine, and obtain the communication overhead of the N different types of computing devices on each pipeline stage; A network topology module, configured to determine a network topology of the N different types of computing devices in each of the M virtual machines based on the first parameters of each pipeline stage in the M virtual machines and the communication overhead of the N different types of computing devices on each pipeline stage; A model partitioning module, used to obtain a model partitioning method according to the pre-trained neural network model, the network topology of N different types of computing devices in each virtual machine, and a preset algorithm; An iterative training module, used for iteratively training the neural network model using a pipeline parallel method between N different types of computing devices and a data parallel method between M virtual machines according to the model partitioning method; The communication overhead module is specifically used to obtain the computing capabilities of N different types of computing devices in each virtual machine; The computing time of each pipeline stage in the M virtual machines is set to be the same, so that the size of the neural network model of the N different types of computing devices on each pipeline stage is proportional to the computing power of the N different types of computing devices; Setting communication synchronization between the same pipeline stages on different virtual machines so that the communication overhead of the N different types of computing devices on each pipeline stage is proportional to the size of the neural network model of the N different types of computing devices; According to the computing capabilities of the N different types of computing devices and the above proportional relationship, obtaining the communication overheads of the N different types of computing devices on each pipeline stage; The network topology module is specifically used to calculate and obtain the communication ratio between the first parameter of each pipeline stage in the M virtual machines and the communication overhead of N different types of computing devices on each pipeline stage; According to the communication ratio from large to small, the first pipeline stage to the Nth pipeline stage are respectively set for the N different types of computing devices on each pipeline stage in order, as the network topology of the N different types of computing devices in each virtual machine; Each virtual machine includes one computing device of different types, and a plurality of the virtual machines constitute a heterogeneous computing cluster.

Images