Model parallel training method and device of heterogeneous system and computer equipment

Abstract

The invention relates to a model parallel training method and device for a heterogeneous system and computer equipment, and the method comprises the steps: constructing a plurality of mapping relationships between a to-be-trained model and each piece of computing equipment under each preset condition, and generating a strategy set; the preset conditions comprise the batch size of the training data, the pipeline parallel granularity corresponding to the to-be-trained model and the preset video memory budget of each computing device; traversing each preset condition, and processing different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy; and based on the model parallel training strategy corresponding to the minimum execution cost, executing model training through each computing device. By means of the method and device, the problem that efficient parallel training cannot be achieved through multiple computing devices in a heterogeneous system when a fixed processor is adopted for training is solved, efficient model parallel training is achieved, and the resource utilization rate of the computing devices is increased.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and computer equipment for parallel model training of heterogeneous systems. Background Art

[0002] Artificial intelligence models are widely used in computer vision, natural language processing and other fields, but the model training process requires processing a large amount of model parameters and related data. However, in existing model training methods, fixed processors are usually used for training, and it is impossible to use multiple computing devices in heterogeneous systems to achieve efficient parallel training.

[0003] There is currently no effective solution to the problem that related technologies use fixed processors for training and cannot use multiple computing devices in heterogeneous systems to achieve efficient parallel training. Summary of the invention

[0004] In this embodiment, a method, apparatus and computer device for parallel model training of a heterogeneous system are provided to solve the problem in the related art that a fixed processor is used for training and efficient parallel training cannot be achieved by using multiple computing devices in the heterogeneous system.

[0005] In a first aspect, a model parallel training method for a heterogeneous system is provided in this embodiment, wherein the heterogeneous system includes a plurality of nodes, each of which is distributed with a plurality of computing devices; the method includes:

[0006] Under each preset condition, multiple mapping relationships between the model to be trained and each of the computing devices are constructed, and a set of policies corresponding to the preset conditions are generated; the preset conditions include a batch size of training data, a pipeline parallel granularity corresponding to the model to be trained, and a preset video memory budget of each of the computing devices;

[0007] Traversing each of the preset conditions, and based on the preset conditions, processing different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0008] Determine the minimum execution cost among the execution costs, and output the model parallel training strategy corresponding to the minimum execution cost;

[0009] Based on the model parallel training strategy corresponding to the minimum execution cost, model training is performed by each of the computing devices.

[0010] In some of the embodiments, under each preset condition, multiple mapping relationships between the model to be trained and each of the computing devices are constructed, including:

[0011] Under each of the preset conditions, based on the number of computing devices of each of the nodes, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each of the computing devices, the model to be trained is divided into multiple model stages;

[0012] Mapping processing is performed on each of the model stages and each of the computing devices to obtain a plurality of mapping relationships between the model to be trained and each of the computing devices.

[0013] In some of the embodiments, under each preset condition, a strategy set corresponding to the preset condition is generated, including:

[0014] Under each of the preset conditions, based on the number of computing devices of each of the nodes and the pipeline parallel granularity corresponding to the model to be trained, a plurality of preset parallel strategies are combined and enumerated to obtain a plurality of candidate strategies;

[0015] Based on each of the candidate strategies, a corresponding strategy set is generated.

[0016] In some of the embodiments, under each of the preset conditions, different mapping relationships and candidate strategies in the strategy set are processed by a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy, including:

[0017] Under each of the preset conditions, the different mapping relationships and each candidate strategy in the strategy set are processed by the dynamic programming algorithm to obtain the currently optimal model parallel training strategy;

[0018] Under the currently optimal model parallel training strategy, determining a first execution cost of each model stage in the model to be trained executed by the heterogeneous system;

[0019] Based on the first execution cost of each model stage in the model to be trained, determine the execution cost of the heterogeneous system under the currently optimal model parallel training strategy.

[0020] In some embodiments, determining a first execution cost of executing each model stage in the to-be-trained model by the heterogeneous system includes:

[0021] Determine the multi-layer network included in each model stage in the model to be trained;

[0022] Based on the layer execution cost of each network layer in the model stage executed by the heterogeneous system, the memory consumption cost of each network layer and the conversion cost between different layers, the first execution cost of the heterogeneous system executing the model stage is determined.

[0023] In some embodiments, the layer execution cost includes a forward propagation cost and a backward propagation cost of the model to be trained; wherein determining the backward propagation cost of the model to be trained includes:

[0024] During the back propagation of the model to be trained, determining the computing time and communication time of the heterogeneous system for executing the current layer of the model to be trained;

[0025] The computing time and the communication time are calculated according to a preset overlap coefficient to obtain a back propagation cost of the model to be trained; the overlap coefficient is used to indicate the degree of influence of the overlap between the computing operation and the communication operation on the computing time and the communication time.

[0026] In some embodiments, the model parallel training strategy corresponding to the minimum execution cost is based on the model parallel training strategy corresponding to the minimum execution cost, and the model training is performed by each computing device, including:

[0027] Allocating each of the computing devices in the heterogeneous system to a server and a client;

[0028] Based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is performed through the server and the client;

[0029] The data transmission between the server and the client is encrypted or obfuscated.

[0030] In a second aspect, a model parallel training device for a heterogeneous system is provided in this embodiment, wherein the heterogeneous system includes a plurality of nodes, each of which is provided with a plurality of computing devices; the device includes: a generation module, a planning module, an output module, and a training module;

[0031] The generation module is used to construct multiple mapping relationships between the model to be trained and each of the computing devices under different preset conditions, and generate a set of policies corresponding to the preset conditions; the preset conditions include a batch size of training data, a pipeline parallel granularity corresponding to the model to be trained, and a preset video memory budget of each of the computing devices;

[0032] The planning module is used to traverse each of the preset conditions, and under each of the preset conditions, process different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0033] The output module is used to determine the minimum execution cost among the execution costs and output the model parallel training strategy corresponding to the minimum execution cost;

[0034] The training module is used to perform model training through each of the computing devices based on the model parallel training strategy corresponding to the minimum execution cost.

[0035] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model parallel training method for heterogeneous systems described in the first aspect above is implemented.

[0036] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the model parallel training method for heterogeneous systems described in the first aspect above is implemented.

[0037] Compared with the related art, the model parallel training method, apparatus and computer equipment for a heterogeneous system provided in this embodiment, by constructing multiple mapping relationships between the model to be trained and each computing device under each preset condition, and generating a strategy set corresponding to the preset condition; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained and the preset video memory budget of each computing device; traversing each preset condition, based on the preset conditions, processing each candidate strategy in different mapping relationships and strategy sets through a dynamic programming algorithm, and obtaining the execution cost of the heterogeneous system under the current optimal model parallel training strategy; further, determining the minimum execution cost among each execution cost, outputting the model parallel training strategy corresponding to the minimum execution cost, and executing model training through each computing device based on the model parallel training strategy corresponding to the minimum execution cost, thereby solving the problem that a fixed processor is used for training and multiple computing devices in a heterogeneous system cannot be used to achieve efficient parallel training, and realizing efficient parallel training using multiple computing devices in a heterogeneous system, while improving the resource utilization and system throughput of the computing devices.

[0038] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 It is a hardware structure block diagram of a terminal device of a model parallel training method for a heterogeneous system provided in an embodiment of the present application;

[0041] Figure 2This is a flow chart of a model parallel training method for a heterogeneous system provided in one embodiment of the present application;

[0042] Figure 3 is a schematic diagram of a layer execution cost calculation method provided by an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of a model parallel training method based on a local cluster and a remote cluster provided in an embodiment of the present application;

[0044] Figure 5 This is a flow chart of a model parallel training method for a heterogeneous system provided in a preferred embodiment of the present application;

[0045] Figure 6 It is a structural block diagram of a model parallel training device for a heterogeneous system provided in one embodiment of the present application.

[0046] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, generation module; 20, planning module; 30, output module; 40, training module. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 : is a hardware structure block diagram of a terminal of the model parallel training method of a heterogeneous system of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the model parallel training method of the heterogeneous system in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0052] In this embodiment, a model parallel training method for a heterogeneous system is provided. Figure 2 is a flow chart of the model parallel training method for the heterogeneous system of this embodiment. Figure 2 As shown, the process includes the following steps:

[0053] Step S210, under each preset condition, construct multiple mapping relationships between the model to be trained and each computing device, and generate a policy set corresponding to the preset condition; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device;

[0054] Step S220, traversing each preset condition, and based on the preset condition, processing each candidate strategy in different mapping relationships and strategy sets through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0055] Step S230, determining the minimum execution cost among the execution costs, and outputting the model parallel training strategy corresponding to the minimum execution cost;

[0056] Step S240, based on the model parallel training strategy corresponding to the minimum execution cost, execute model training through each computing device.

[0057] In this embodiment, the heterogeneous system includes multiple nodes, each of which can be a server. Each node contains multiple computing devices, and each computing device can be set with a different video memory budget. For example, each node contains a certain number of graphics processing units (GPUs) or other processors.

[0058] Specifically, different preset conditions are traversed, and under each preset condition, a model parallel training strategy is dynamically planned for the model to be trained, and the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device. The video memory budget of each computing device and multiple batch sizes of training data are preset, and different pipeline parallel granularities are enumerated under each batch size to obtain different preset conditions.

[0059] It should be noted that the model to be trained can be a large language model in different application fields, such as a large machine translation model, a complex image recognition model, etc., and the pipeline parallel granularity refers to the maximum number of stages into which the model to be trained can be divided. In order to better adapt to the current heterogeneous hardware environment, this embodiment sets the parallel granularity P of each pipeline to 1 and a multiple of 2 to the power of n of the number of nodes N, up to DN, that is, P∈{1, N, 2N, 4N, …, DN}, to achieve pruning of the huge search space, reduce the search space, and thus reduce the time overhead of searching for the optimal parallel training strategy.

[0060] Under each preset condition, based on the number of computing devices at each node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, the model to be trained is segmented to obtain multiple model stages, each of which contains multiple layers of the model to be trained. At the same time, based on the number of computing devices at each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies, and a corresponding strategy set is generated based on each candidate strategy.

[0061] Furthermore, under each preset condition, different mapping relationships and candidate strategies in the strategy set are processed through a dynamic programming algorithm to obtain the current optimal model parallel training strategy, and under the current optimal model parallel training strategy, the first execution cost of the heterogeneous system executing each model stage in the model to be trained is determined, and based on the first execution cost of each model stage in the model to be trained, the execution cost of the heterogeneous system under the current optimal model parallel training strategy is determined.

[0062] After that, the execution cost of the heterogeneous system obtained by dynamic programming under each preset condition is obtained, and the minimum execution cost among the execution costs is determined. When the minimum execution cost is not an infinite value, the maximum throughput of the system is updated according to the minimum execution cost, and the model parallel training strategy corresponding to the minimum execution cost is output. The parallel training strategy includes the currently selected preset conditions, mapping relationships, and candidate strategies applied to each layer of the network in the current model stage. In this way, the dynamic programming algorithm is used to automatically generate the optimal hybrid parallel strategy, that is, the parallel strategy is automatically adjusted according to the hardware configuration and model structure to ensure that the best training performance can be achieved under the memory budget of various devices, and then the model parallel training strategy corresponding to the minimum execution cost is used to perform model training through each computing device. Among them, if the maximum throughput of the system is not updated after the optimization algorithm of the parallel training strategy is completed, it indicates that the video memory of the current hardware conditions is insufficient to complete the training.

[0063] Artificial intelligence models are widely used in computer vision, natural language processing and other fields, but the model training process requires processing a large amount of model parameters and related data. However, in existing model training methods, fixed processors are usually used for training, and it is impossible to use multiple computing devices in heterogeneous systems to achieve efficient parallel training.

[0064] Compared with the prior art, the present application constructs multiple mapping relationships between the model to be trained and each computing device under each preset condition, and generates a strategy set corresponding to the preset condition; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device; traverses each preset condition, and based on the preset conditions, processes different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy; further, determines the minimum execution cost among each execution cost, outputs the model parallel training strategy corresponding to the minimum execution cost, and executes model training through each computing device based on the model parallel training strategy corresponding to the minimum execution cost. Based on this, for heterogeneous systems containing multiple computing devices, dynamic planning of model parallel training strategies is performed under different preset conditions to obtain the optimal model parallel training strategy under each preset condition, so that the optimal model parallel training strategy can be selected for model training according to the system execution time corresponding to each strategy. This solves the problem of using a fixed processor for training and being unable to achieve efficient parallel training using multiple computing devices in a heterogeneous system. It realizes efficient parallel training using multiple computing devices in a heterogeneous system, while improving the resource utilization and system throughput of the computing devices.

[0065] In some embodiments, in step S210, under each preset condition, constructing multiple mapping relationships between the model to be trained and each computing device includes the following steps:

[0066] Step S211, under each preset condition, based on the number of computing devices of each node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, the model to be trained is divided into multiple model stages;

[0067] Step S212, mapping processing is performed on each model stage and each computing device to obtain multiple mapping relationships between the models to be trained and each computing device.

[0068] Specifically, the number of computing devices D of each node and the preset video memory budget E of each computing device are obtained. N×D}, the number of computing devices on each node is usually the same, e i represents the preset video memory budget of each computing device, and N represents the number of nodes. Under each preset condition, the pipeline parallel granularity corresponding to the current model to be trained is determined. The pipeline parallel granularity represents the maximum number of stages that the model to be trained can be divided into. Based on the number of computing devices per node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, the model to be trained M is divided into multiple model stages M. i , each model stage contains a multi-layer network of the model to be trained.

[0069] Furthermore, each model stage M obtained by dividing the model i Map to the corresponding computing devices to obtain the mapping relationship between the model to be trained and each computing device, so as to obtain a variety of different mapping relationships through different mapping association methods, that is, in different mapping relationships, each model stage M i Can be mapped to different computing devices.

[0070] Through this embodiment, under each preset condition, based on the number of computing devices of each node, the pipeline parallel granularity corresponding to the model to be trained and the preset video memory budget of each computing device, the model to be trained is split to obtain multiple model stages, and each model stage is mapped with each computing device to obtain multiple mapping relationships between the model to be trained and each computing device, thereby realizing model segmentation based on the video memory budget of different computing devices, thereby improving the video memory utilization and system throughput of the computing devices.

[0071] In some embodiments, generating a policy set corresponding to each preset condition under each preset condition in step S210 includes the following steps:

[0072] Under each preset condition, based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies;

[0073] Based on each candidate strategy, a corresponding strategy set is generated.

[0074] Specifically, the number of computing devices of each node is obtained, and under each preset condition, based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies. The preset parallel strategies include data parallelism, tensor parallelism, and pipeline parallelism.

[0075] Furthermore, based on the combined enumeration of candidate strategies S j , generate the corresponding strategy set S, which contains all possible parallel methods under the current preset conditions.

[0076] Through this embodiment, under each preset condition, based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies, so as to generate a strategy set including multiple candidate strategies, which is convenient for subsequent dynamic planning with different mapping relationships and realizes multi-dimensional optimization of model parallel training strategy.

[0077] In some embodiments, in step S220, under each preset condition, different mapping relationships and candidate strategies in the strategy set are processed by a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy, including the following steps:

[0078] Step S221, under each preset condition, processing different mapping relationships and candidate strategies in the strategy set by a dynamic programming algorithm to obtain the current optimal model parallel training strategy;

[0079] Step S222, under the current optimal model parallel training strategy, determining the first execution cost of each model stage in the model to be trained executed by the heterogeneous system;

[0080] Step S223, based on the first execution cost of each model stage in the model to be trained, determine the execution cost of the heterogeneous system under the current optimal model parallel training strategy.

[0081] Specifically, under each preset condition, a dynamic programming algorithm is used to process different mapping relationships and candidate strategies in the strategy set, and each model stage obtained by model segmentation is traversed to obtain the optimal parallel training strategy for each model stage in the model to be trained. When the optimal parallel training strategy is used, the first execution cost of the model stage is minimized. Among them, the mapping relationship refers to the mapping relationship between each model stage obtained by model segmentation and each computing device, and the parallel training strategy includes the currently selected preset conditions, mapping relationships, and candidate strategies for each layer of network application in the current model stage.

[0082] Furthermore, based on the optimal parallel training strategy corresponding to each model stage, the current optimal model parallel training strategy is determined, that is, the optimal parallel training strategy for the entire model to be trained under the current preset conditions. Under the current optimal model parallel training strategy, the first execution cost of each model stage in the model to be trained executed by the heterogeneous system is calculated, and the first execution cost of each model stage in the model to be trained is accumulated to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy.

[0083] Through this embodiment, under each preset condition, different mapping relationships and candidate strategies in the strategy set are processed through a dynamic programming algorithm to obtain the current optimal model parallel training strategy, and under the current optimal model parallel training strategy, the first execution cost of the heterogeneous system executing each model stage in the model to be trained is determined. Based on the first execution cost of each model stage in the model to be trained, the execution cost of the heterogeneous system under the current optimal model parallel training strategy is determined, so as to achieve accurate calculation of the execution cost, and automatically search for the optimal model parallel training strategy based on the execution cost.

[0084] In some embodiments, determining the first execution cost of each model stage in the model to be trained executed by the heterogeneous system under the currently optimal model parallel training strategy in step S222 includes:

[0085] Determine the multi-layer network contained in each model stage of the model to be trained;

[0086] Under the current optimal model parallel training strategy, the first execution cost of the heterogeneous system execution model stage is determined based on the layer execution cost of each layer of the network, the memory consumption cost of each layer of the network, and the conversion cost between different layers in the heterogeneous system execution model stage.

[0087] Specifically, for each model stage in the model to be trained, determine the multi-layer network included in the current model stage, and calculate the first execution cost of the heterogeneous system executing the current model stage based on the layer execution cost of each layer of the network in the execution model stage, the memory consumption cost of each layer of the network, and the conversion cost between different layers. In the dynamic programming search process, the state transition formula is derived based on C(0,·)=0 and C(·,0)=∞ as follows:

[0088]

[0089] In formula (1), C(L,E) represents the memory budget E = {e1, e2, ..., e N×D}, the first execution cost of the current model stage, e i represents the preset video memory budget of each computing device, N represents the number of nodes; c(L,S j ) represents the candidate strategy S for the Lth layer network application in the model stage j Memory consumption cost of R(L,S i ,S j ) indicates that the candidate strategy S is applied at layer L j Apply candidate strategy S to the previous layer i The conversion cost between them; C(0,·)=0 means that the execution cost is 0 when the number of model layers L is 0, and C(·,0)=∞ means that the execution cost is infinite when the memory budget is 0. If two adjacent layers adopt different candidate strategies, the output of the previous layer of the model needs to be converted to the data layout required by the candidate strategy of the next layer. For example, the previous layer of the model adopts a combination of 2-way data parallelism and 2-way tensor parallelism, and the current layer adopts 4-way data parallelism, then a conversion step is required to prepare a complete model copy and 1 / 4 forward activation of the current layer on each computing device, where X-way data parallelism means processing data in parallel on X computing devices, and Y-way tensor parallelism means processing the model's tensors in parallel on Y computing devices. In addition, during the state transfer process, if the actual memory usage exceeds the device memory budget E, the execution cost returns to infinity.

[0090] Among them, models with different batch sizes and layer configurations are run on the computing device, and different parallelism configurations are traversed, such as different pipeline parallel granularities, tensor parallel granularities, etc., to obtain the parameter memory and activation memory usage of each layer. The memory storage situation is used as training data, and the training obtains inputs such as batch size, number of model layers and parallel strategy. The output is a prediction model of memory consumption cost. This model is used to estimate the memory size occupied by each layer of the model when running the model on the computing device.

[0091] It should be noted that for the model stage containing L layers, in order to obtain the minimum first execution time C(L,E), its solution must include the solution to the subproblem C(L′,E′). C(L′,E′) represents the minimum execution time of the first L′ layers within the smaller computing device memory budget E′. If the optimal solution C(L,E) does not contain a specific C(L′,E′), the total execution time can always be reduced by replacing the subproblem solution with C(L′,E′). Due to the linear sequence model structure, the memory budget EE′ remains unchanged, and the parallelization plan of the first L′ layer will not affect the remaining LL′ layers. That is, the solution to the subproblem C(L',E') and the solutions to the remaining layers are independent of each other and can be optimized separately. Therefore, dynamic programming is used to obtain the minimum first execution time of the current model stage, following the optimal substructure property.

[0092] Through this embodiment, the multiple layers contained in each model stage in the model to be trained are determined, and the first execution cost of the heterogeneous system execution model stage is determined based on the layer execution cost of each layer in the heterogeneous system execution model stage, the memory consumption cost of each layer, and the conversion cost between different layers, so as to accurately calculate the execution cost corresponding to each model stage under the current optimal model parallel training strategy.

[0093] In some embodiments, the layer execution cost includes a forward propagation cost and a backward propagation cost of the model to be trained; wherein determining the backward propagation cost of the model to be trained comprises the following steps:

[0094] During the back propagation of the model to be trained, determine the computation time and communication time of the heterogeneous system to execute the current layer of the model to be trained;

[0095] According to the preset overlap coefficient, the computing time and the communication time are calculated to obtain the back propagation cost of the model to be trained; the overlap coefficient is used to indicate the degree of influence of the overlap between the computing operation and the communication operation on the computing time and the communication time.

[0096] Specifically, Figure 3 As shown in the figure, the layer execution cost refers to the execution cost of each layer of the model under the current model parallel training strategy. The layer execution cost includes the forward propagation cost and the backward propagation cost of the model to be trained. When the model is forward propagated, the sum of the computing time and the communication time of the computing device executing the current layer of the model to be trained is determined, and the sum of the computing time and the communication time is taken as the forward propagation cost of the model to be trained.

[0097] It should be noted that, during the back propagation of the model to be trained, since each computing device can perform computing operations and communication operations at the same time, thread bundle resource contention of multiple processors is likely to occur. This embodiment measures the different overlaps between computing operations and communication operations, and the impact on computing time and communication time, and sets an overlap coefficient based on the measurement results. The overlap coefficient is used to indicate the degree of impact of the overlap between computing operations and communication operations on computing time and communication time. Therefore, when the model to be trained is back propagated, the computing time and communication time of the heterogeneous system executing the current layer of the model to be trained are determined, and according to the preset overlap coefficient, the computing time and communication time are multiplied respectively, and the two multiplication results are added together to obtain the back propagation cost of the model to be trained.

[0098] Among them, models with different numbers of layers and different batch sizes are executed on the computing device, and the calculation time of each layer in the model is recorded. The calculation time is set as the product of the batch size and the calculation time of each sample. By sampling the average calculation time required for each layer under different batch sizes and different number of layers, the corresponding linear regression model is constructed, and the linear regression model is used to estimate the calculation time of each layer of the model under the current model parallel training strategy. As for the communication time of each layer of the model under the current model parallel training strategy, the collective communication library (NVIDIA Collective Communications Library, NCCL) is first installed, and the NCCL test tool is used to verify the correct installation and functionality of the NCCL library in the current hardware environment. After the verification is completed, the communication bandwidth between different computing devices is measured to obtain the communication performance data between each pair of computing devices, including communication bandwidth, delay and other data information, and then the communication time is estimated based on the communication bandwidth value obtained by the bandwidth measurement and the size of the tensor to be transmitted. Specifically, the size of the tensor to be communicated can be divided by the bandwidth of the connection between the computing devices to obtain the communication time.

[0099] Through this embodiment, when the model to be trained is back-propagated, the computing time and communication time of the heterogeneous system executing the current layer of the model to be trained are determined, and the computing time and communication time are calculated according to the preset overlap coefficient to obtain the back-propagation cost of the model to be trained. In this way, the computing time and communication time in the back-propagation process are accurately calculated based on the influence of the overlap between the computing operation and the communication operation on the computing time and communication time, so as to improve the accuracy of the execution cost of the computing layer, facilitate the subsequent optimization of the parallel training strategy based on the layer execution cost of each layer of the model network, and comprehensively consider the information such as the video memory budget of the computing device and the communication bandwidth, so as to find the parallel training strategy that best suits the current model and hardware configuration, that is, automatically search for the optimal model parallel training strategy.

[0100] In some embodiments, the model parallel training strategy based on the minimum execution cost corresponding to step S240, performing model training by each computing device, includes the following steps:

[0101] Step S241, allocating each computing device in the heterogeneous system to a server and a client;

[0102] Step S242, based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is executed through the server and the client; wherein, the data transmission between the server and the client adopts encrypted transmission or obfuscated transmission.

[0103] In this embodiment, each computing device in the heterogeneous system is allocated to the server and the client, and based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is executed through the server and the client, and the data transmission between the server and the client adopts encrypted transmission or obfuscated transmission, that is, the data information exchanged between the server and the client is always processed intermediate data, not original data, to reduce the risk of data leakage. It should be noted that the key data processing with high security requirements in the model training process is distributed on the client, and some intermediate data processing processes that belong to computationally intensive tasks are executed on the server, so as to ensure the security of the training process and the protection of data privacy, and at the same time effectively utilize the computing power of different devices to improve the overall training efficiency.

[0104] Specifically, the client is responsible for processing the original data and converting it into an intermediate representation. The intermediate representation is encrypted or obfuscated to ensure that it cannot be reversed to the original data. The processed intermediate representation is transmitted to the server, which performs the main calculation work on the intermediate representation and generates further intermediate results. The intermediate results are encrypted or obfuscated and returned to the client. When the client receives the processed intermediate results, it continues to perform the remaining calculation tasks, finally generates the gradients required for training, and completes the model update.

[0105] For example, the local cluster is used as the client and the remote cluster is used as the server. Based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is performed by the server and the client. The embedding layer and the back propagation layer are calculated in parallel in the local cluster, while the remote cluster is used to run the training program. Figure 4As shown in the figure, in the first stage, the local cluster is responsible for processing the original data, converting the original data into an intermediate representation, which is encrypted and transmitted to the server; in the second stage, the server performs the main computing tasks and obtains the intermediate results, which are encrypted and returned to the client; in the third stage, after receiving the processed intermediate results, the client continues to perform the remaining computing tasks, generates the gradients required for training, and completes the model update. In each stage, the local cluster and the remote cluster can perform parallel data processing.

[0106] Through this embodiment, each computing device in the heterogeneous system is allocated to the server and the client, and based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is executed through the server and the client, so as to ensure data privacy and security in a multi-device and multi-center environment.

[0107] The present embodiment is described and illustrated below through preferred embodiments.

[0108] Figure 5 is a flow chart of the model parallel training method for a heterogeneous system of the preferred embodiment. Figure 5 As shown, the model parallel training method of the heterogeneous system includes the following steps:

[0109] Step S510, traversing various preset conditions, where the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device;

[0110] Step S520, under each preset condition, based on the number of computing devices of each node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, the model to be trained is divided into multiple model stages;

[0111] Step S530, mapping processing is performed on each model stage and each computing device to obtain multiple mapping relationships between the model to be trained and each computing device;

[0112] Step S540, under each preset condition, based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies;

[0113] Step S550, generating a corresponding strategy set based on each candidate strategy, processing different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm, and obtaining the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0114] Step S560, determining the minimum execution cost among the execution costs, and outputting the model parallel training strategy corresponding to the minimum execution cost;

[0115] Step S570, allocate each computing device in the heterogeneous system to the server and the client, and execute the training process of the model to be trained through the server and the client based on the model parallel training strategy corresponding to the minimum execution cost; wherein, the data transmission between the server and the client adopts encrypted transmission or obfuscated transmission.

[0116] Through this embodiment, each preset condition is traversed. Under each preset condition, based on the number of computing devices of each node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, the model to be trained is segmented to obtain multiple model stages, and each model stage is mapped with each computing device to obtain multiple mapping relationships between the model to be trained and each computing device. Based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained, multiple preset parallel strategies are combined and enumerated to obtain multiple candidate strategies. Based on each candidate strategy, a corresponding strategy set is generated, and different mapping relationships and candidate strategies in the strategy set are processed through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy, thereby realizing accurate calculation of the execution cost and automatically searching for the optimal model parallel training strategy based on the execution cost.

[0117] Furthermore, the minimum execution cost among the execution costs is determined, the model parallel training strategy corresponding to the minimum execution cost is output, and the computing devices in the heterogeneous system are allocated to the server and the client. Based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is executed through the server and the client; wherein, the data transmission between the server and the client adopts encrypted transmission or obfuscated transmission, thereby realizing efficient parallel training using multiple computing devices in the heterogeneous system, while ensuring the security of the training process and the protection of data privacy.

[0118] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0119] In this embodiment, a model parallel training device for a heterogeneous system is also provided, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0120] Figure 6 is a structural block diagram of the model parallel training device of the heterogeneous system of this embodiment, such as Figure 6 As shown, the device includes: a generation module 10, a planning module 20, an output module 30 and a training module 40;

[0121] A generation module 10 is used to construct multiple mapping relationships between the model to be trained and each computing device under different preset conditions, and generate a set of policies corresponding to the preset conditions; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device;

[0122] The planning module 20 is used to traverse various preset conditions, and under each preset condition, process different mapping relationships and candidate strategies in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0123] An output module 30 is used to determine the minimum execution cost among the execution costs and output the model parallel training strategy corresponding to the minimum execution cost;

[0124] The training module 40 is used to perform model training through each computing device based on the model parallel training strategy corresponding to the minimum execution cost.

[0125] Through the device provided by this embodiment, under each preset condition, multiple mapping relationships between the model to be trained and each computing device are constructed, and a strategy set corresponding to the preset condition is generated; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device; each preset condition is traversed, and based on the preset conditions, different mapping relationships and candidate strategies in the strategy set are processed through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy; further, the minimum execution cost among the execution costs is determined, and the model parallel training strategy corresponding to the minimum execution cost is output. Based on the model parallel training strategy corresponding to the minimum execution cost, model training is performed by each computing device, which solves the problem that a fixed processor is used for training and multiple computing devices in a heterogeneous system cannot be used to achieve efficient parallel training, and realizes efficient parallel training using multiple computing devices in a heterogeneous system, while improving the resource utilization and system throughput of the computing devices.

[0126] In some of the embodiments, the generation module 10 is also used to, under each preset condition, split the model to be trained based on the number of computing devices of each node, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device, to obtain multiple model stages; and map each model stage with each computing device to obtain multiple mapping relationships between the model to be trained and each computing device.

[0127] In some of the embodiments, the generation module 10 is also used to combine and enumerate multiple preset parallel strategies under each preset condition based on the number of computing devices of each node and the pipeline parallel granularity corresponding to the model to be trained to obtain multiple candidate strategies; and generate a corresponding strategy set based on each candidate strategy.

[0128] In some of the embodiments, the planning module 20 is also used to process different mapping relationships and candidate strategies in the strategy set under each preset condition through a dynamic programming algorithm to obtain the current optimal model parallel training strategy; under the current optimal model parallel training strategy, determine the first execution cost of the heterogeneous system executing each model stage in the model to be trained; based on the first execution cost of each model stage in the model to be trained, determine the execution cost of the heterogeneous system under the current optimal model parallel training strategy.

[0129] In some of the embodiments, the planning module 20 is also used to determine the multi-layer networks included in each model stage in the model to be trained; based on the layer execution cost of each layer of the network in the heterogeneous system execution model stage, the memory consumption cost of each layer of the network and the conversion cost between different layers, determine the first execution cost of the heterogeneous system execution model stage.

[0130] In some of the embodiments, the training module 40 is also used to allocate each computing device in the heterogeneous system to the server and the client; based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is executed through the server and the client; wherein, the data transmission between the server and the client adopts encrypted transmission or obfuscated transmission.

[0131] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0132] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0133] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0134] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0135] S1, under different preset conditions, construct multiple mapping relationships between the models to be trained and each computing device, and generate a set of policies corresponding to the preset conditions; the preset conditions include the batch size of the training data, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each computing device;

[0136] S2, traverses each preset condition, and under each preset condition, processes different mapping relationships and candidate strategies in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy;

[0137] S3, determining the minimum execution cost among the execution costs, and outputting the model parallel training strategy corresponding to the minimum execution cost;

[0138] S4, based on the model parallel training strategy corresponding to the minimum execution cost, execute model training through each computing device.

[0139] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0140] In addition, in combination with the model parallel training method for heterogeneous systems provided in the above embodiments, a storage medium may also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the model parallel training methods for heterogeneous systems in the above embodiments is implemented.

[0141] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0142] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0143] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0144] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A model parallel training method for a heterogeneous system, characterized in that: The heterogeneous system includes a plurality of nodes, each of which is distributed with a plurality of computing devices; the method includes: Under each preset condition, multiple mapping relationships between the model to be trained and each of the computing devices are constructed, and a set of policies corresponding to the preset conditions are generated; the preset conditions include a batch size of training data, a pipeline parallel granularity corresponding to the model to be trained, and a preset video memory budget of each of the computing devices; Traversing each of the preset conditions, and based on the preset conditions, processing different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy; Determine the minimum execution cost among the execution costs, and output the model parallel training strategy corresponding to the minimum execution cost; Based on the model parallel training strategy corresponding to the minimum execution cost, model training is performed by each of the computing devices.

2. The model parallel training method for heterogeneous systems according to claim 1, characterized in that: Under each preset condition, multiple mapping relationships between the model to be trained and each of the computing devices are constructed, including: Under each of the preset conditions, based on the number of computing devices of each of the nodes, the pipeline parallel granularity corresponding to the model to be trained, and the preset video memory budget of each of the computing devices, the model to be trained is divided into multiple model stages; Mapping processing is performed on each of the model stages and each of the computing devices to obtain a plurality of mapping relationships between the model to be trained and each of the computing devices.

3. The model parallel training method for heterogeneous systems according to claim 1, characterized in that: Under each preset condition, a strategy set corresponding to the preset condition is generated, including: Under each of the preset conditions, based on the number of computing devices of each of the nodes and the pipeline parallel granularity corresponding to the model to be trained, a plurality of preset parallel strategies are combined and enumerated to obtain a plurality of candidate strategies; Based on each of the candidate strategies, a corresponding strategy set is generated.

4. The model parallel training method for heterogeneous systems according to claim 1, characterized in that: Under each of the preset conditions, the different mapping relationships and each candidate strategy in the strategy set are processed by a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy, including: Under each of the preset conditions, the different mapping relationships and each candidate strategy in the strategy set are processed by the dynamic programming algorithm to obtain the currently optimal model parallel training strategy; Under the currently optimal model parallel training strategy, determining a first execution cost of each model stage in the model to be trained executed by the heterogeneous system; Based on the first execution cost of each model stage in the model to be trained, determine the execution cost of the heterogeneous system under the currently optimal model parallel training strategy.

5. The model parallel training method for heterogeneous systems according to claim 4, characterized in that: The determining a first execution cost of each model stage in the model to be trained executed by the heterogeneous system includes: Determine the multi-layer network included in each model stage in the model to be trained; Based on the layer execution cost of each network layer in the model stage executed by the heterogeneous system, the memory consumption cost of each network layer and the conversion cost between different layers, the first execution cost of the heterogeneous system executing the model stage is determined.

6. The model parallel training method for heterogeneous systems according to claim 4, characterized in that: The layer execution cost includes the forward propagation cost and the backward propagation cost of the model to be trained; wherein determining the backward propagation cost of the model to be trained includes: During the back propagation of the model to be trained, determining the computing time and communication time of the heterogeneous system for executing the current layer of the model to be trained; The computing time and the communication time are calculated according to a preset overlap coefficient to obtain a back propagation cost of the model to be trained; the overlap coefficient is used to indicate the degree of influence of the overlap between the computing operation and the communication operation on the computing time and the communication time.

7. The model parallel training method for heterogeneous systems according to claim 1, characterized in that: The model parallel training strategy corresponding to the minimum execution cost is based on the model parallel training strategy corresponding to the minimum execution cost, and the model training is performed by each of the computing devices, including: Allocating each of the computing devices in the heterogeneous system to a server and a client; Based on the model parallel training strategy corresponding to the minimum execution cost, the training process of the model to be trained is performed through the server and the client; The data transmission between the server and the client is encrypted or obfuscated.

8. A model parallel training device for a heterogeneous system, characterized in that: The heterogeneous system includes a plurality of nodes, each of which is distributed with a plurality of computing devices; the apparatus includes: a generation module, a planning module, an output module and a training module; The generation module is used to construct multiple mapping relationships between the model to be trained and each of the computing devices under different preset conditions, and generate a set of policies corresponding to the preset conditions; the preset conditions include a batch size of training data, a pipeline parallel granularity corresponding to the model to be trained, and a preset video memory budget of each of the computing devices; The planning module is used to traverse each of the preset conditions, and under each of the preset conditions, process different mapping relationships and each candidate strategy in the strategy set through a dynamic programming algorithm to obtain the execution cost of the heterogeneous system under the current optimal model parallel training strategy; The output module is used to determine the minimum execution cost among the execution costs and output the model parallel training strategy corresponding to the minimum execution cost; The training module is used to perform model training through each of the computing devices based on the model parallel training strategy corresponding to the minimum execution cost.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the model parallel training method for a heterogeneous system according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the model parallel training method for a heterogeneous system described in any one of claims 1 to 7 are implemented.

Images