Large model wide-area heterogeneous distributed training method and system based on dual optimization

Abstract

The present invention belongs to the technical field of large model training, and specifically relates to a large model wide-area heterogeneous distributed training method and system based on dual optimization; the large model wide-area heterogeneous distributed training method based on dual optimization includes: obtaining the configuration information of the base model, splitting the heterogeneous data center, and converting the heterogeneous data center into a data center that can complete at most one stage task; using the ant colony algorithm to optimize the initial combination of the split data center to obtain the preliminary parallel group scheme of the base model; based on the optimization of the genetic algorithm, the base model parallel group scheme is obtained, and the model training architecture is generated to complete the large model wide-area heterogeneous distributed training based on dual optimization. Aiming at the difficulties of architecture design, communication cost calculation and difficulty in finding the best parallel group strategy faced by base model training in a real heterogeneous environment, the model training time is reduced while the cost and threshold of large model training are effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of large model training, and specifically relates to a large model wide-area heterogeneous distributed training method and system based on dual optimization. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The rise of large models in the field of artificial intelligence (such as pre-trained pedestal models) has significantly improved the performance of tasks such as intelligent question answering, image generation, and speech recognition. As the number of parameters of models such as Claude, LLaMa, and OpenAI's GPT-3 increases from billions to trillions, the capabilities of the models increase with the increase in parameter scale and have emergent capabilities. However, training these large models has become extremely difficult. In a homogeneous environment, computing resources such as NVIDIA A100 and H100 are not only limited but also expensive. For example, the training of GPT-3 used 10,000 V100 graphics cards and took two months; but economical graphics cards such as NVIDIA TeslaT4 are not competent for the training tasks of pedestal models. Although resources across multiple data centers can be connected through the Internet, different network bandwidth and latency characteristics bring additional communication overhead and parameter synchronization challenges.

[0004] Most base models are based on architectures such as Megatron, Deepspeed, Colossal-AI and Alpa. They use high-speed private networks within data centers for communication and expand the model size through hybrid parallel strategies such as pipeline parallelism, model parallelism and data parallelism without having to consider communication overhead too much. However, centralized homogeneous devices are limited in number and expensive, which raises the training threshold.

[0005] To reduce the cost of training, decentralized base model training can be achieved by combining devices in multiple locations through the Internet. Latency and bandwidth issues in heterogeneous networks complicate training. Distributed parallel training methods based on heterogeneous networks can be used to make it possible to train large language models in a wide area network environment. However, data centers in different regions may have different types of graphics processor GPU devices, which results in the training process being limited by devices with weaker performance.

[0006] Low bandwidth, high latency, and unbalanced computing power in real heterogeneous environments pose complex problems, mainly manifested in:

[0007] 1) General optimization methods cannot fully consider the latency and bandwidth between different regions;

[0008] 2) It is impossible to accurately calculate the communication costs of data parallelism and pipeline parallelism of different devices and measure the computing power balance of the parallel group.

[0009] Therefore, base model training in real heterogeneous environments still faces challenges in architecture design, communication cost calculation, and difficulty in finding the best parallel group strategy. Summary of the invention

[0010] In order to solve the above problems, the present invention proposes a large-model wide-area heterogeneous distributed training method and system based on dual optimization, which combines pipeline model parallelism and data parallelism for decentralized hybrid parallel training of heterogeneous devices and heterogeneous networks to maximize the parallelism of base model training. While significantly reducing the model training time, it effectively reduces the cost and threshold of large-model training, and realizes hybrid parallel training of large models by combining GPU resources in multiple regions over a wide area.

[0011] According to some embodiments, a first solution of the present invention provides a large-model wide-area heterogeneous distributed training method based on dual optimization, which adopts the following technical solutions:

[0012] A large-model wide-area heterogeneous distributed training method based on dual optimization, comprising:

[0013] Get the configuration information of the base model;

[0014] Split the heterogeneous data center based on the configuration information of the base model and convert the heterogeneous data center into a data center that can complete at most one stage task;

[0015] The ant colony algorithm is used to optimize the initial combination of the split data centers and obtain the preliminary parallel group scheme of the base model;

[0016] Based on the preliminary parallel group scheme obtained by genetic algorithm optimization, a parallel group scheme for the base model is obtained;

[0017] Based on the obtained parallel group scheme, a model training architecture is generated to complete the wide-area heterogeneous distributed training of large models based on dual optimization.

[0018] As a further technical limitation, heterogeneous data centers are divided into large data centers, medium data centers and small data centers according to the computing power contained in the heterogeneous data centers; the computing power of the large data center is greater than the computing power required for a stage task, the computing power of the medium data center is equal to the computing power required for a stage task, and the computing power of the small data center is less than the computing power required for a stage task.

[0019] Furthermore, in the process of splitting the heterogeneous data centers, the large data centers are split according to the classification results of the heterogeneous data centers to obtain medium-sized data centers or small data centers, so that the split data centers can complete at most one stage task.

[0020] As a further technical limitation, in the process of initializing the combination optimization of the data centers after the split, the data centers are taken as the basic units and the ant colony algorithm is used to combine the data centers to obtain data parallel groups and pipeline parallel groups with balanced computing resources, that is, to obtain a preliminary parallel group solution of the base model.

[0021] Furthermore, considering the data parallel cost and pipeline parallel cost, the genetic communication optimization algorithm is used to optimize the total data center communication cost of the obtained preliminary parallel group scheme to obtain the base model parallel group scheme.

[0022] As a further technical limitation, based on the obtained parallel group scheme, the arrangement of heterogeneous devices and the communication connection method between the devices during model training are determined to generate a model training architecture.

[0023] According to some embodiments, a second solution of the present invention provides a large-model wide-area heterogeneous distributed training system based on dual optimization, which adopts the following technical solutions:

[0024] A large-model wide-area heterogeneous distributed training system based on dual optimization, comprising:

[0025] An acquisition module, configured to acquire configuration information of a base model;

[0026] A splitting module is configured to split the heterogeneous data center based on the configuration information of the base model, and convert the heterogeneous data center into a data center that can complete at most one stage task;

[0027] A dual optimization module is configured to use an ant colony algorithm to optimize the initial combination of the split data centers to obtain a preliminary parallel group solution for the base model; based on the preliminary parallel group solution obtained by the genetic algorithm optimization, a parallel group solution for the base model is obtained;

[0028] The training module is configured to generate a model training architecture according to the obtained parallel group scheme to complete the wide-area heterogeneous distributed training of large models based on dual optimization.

[0029] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0030] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in the first solution of the present invention.

[0031] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0032] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in the first scheme of the present invention are implemented.

[0033] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0034] A computer program product includes software code, wherein the program in the software code executes the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in the first embodiment of the present invention.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] When a single data center has insufficient resources and cannot complete the base model training, the present invention combines heterogeneous devices in multiple regional data centers to achieve the combination of heterogeneous GPU devices in multiple data centers in a wide area environment, making it possible to build a large-scale base model and effectively reducing the training cost of the base model.

[0037] The present invention uses the ant colony algorithm to initialize and combine data centers at a macro level to form a preliminary parallel group arrangement plan, thereby ensuring the balance of computing resources allocated to different stages and good communication within the data parallel group, so that heterogeneous devices in multiple data centers are fully utilized; existing resources can be fully utilized to train large models.

[0038] The present invention uses genetic algorithms to adjust the arrangement of GPU devices at the microscopic level, and carefully optimizes the communication costs of data parallelism and pipeline parallelism, which greatly reduces the training time of the model, improves the throughput of the system during operation, and further reduces the training cost of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0040] Figure 1This is an architecture diagram of a large-model wide-area heterogeneous distributed training method based on dual optimization in Embodiment 1 of the present invention;

[0041] Figure 2 This is an architectural diagram of the splitting and combining of heterogeneous data centers in the first embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of the model training architecture in Example 1 of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0046] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0047] Embodiment 1

[0048] Embodiment 1 of the present invention introduces a large-model wide-area heterogeneous distributed training method based on dual optimization.

[0049] As the scale of the model gradually increases, limited computing resources have led to restrictions on the training of large models. Joint training of large models by multiple devices in multiple regions has become a key research direction. However, the joint training of devices in multiple regions faces the dual challenges of network heterogeneity and device heterogeneity. Based on this, this embodiment introduces a large model wide-area heterogeneous distributed training method based on dual optimization, such as Figure 1As shown in the figure, for decentralized hybrid parallel training of heterogeneous devices and networks, pipeline model parallelism (PMP) and data parallelism (DP) are combined to maximize the parallelism of base model training; it can coordinate heterogeneous networks and heterogeneous computing devices in different geographical locations with slow connections, generate optimized parallel group solutions based on latency bandwidth and device computing performance, form a dedicated model training architecture, and coordinate multiple data center resources to train base models such as GPT-3, effectively solving the problem of insufficient GPU resources in a single data center, significantly reducing the training time of large models, and effectively lowering the cost and threshold of large model training.

[0050] This embodiment formulates the existing technical problems, and through the splitting of heterogeneous data centers, dual optimization of ant colony algorithm and genetic algorithm, finally generates a parallel group solution and model training architecture of the base model, and completes the distributed training of the base model in a dual heterogeneous environment.

[0051] According to the network connection conditions in the real environment, data centers distributed in different geographical locations have different delays and bandwidths. The devices in each data center are interconnected through a high-speed, low-latency dedicated network. Heterogeneous GPU devices (including NVIDIA A100 and NVIDIA Tesla T4) in multiple data centers are used to collaboratively complete the training of Foundation Models.

[0052] In this embodiment, in order to fully utilize the performance of the device, all layers of the base model are evenly divided into Each stage may contain several transformer blocks, using heterogeneous devices in one or more data centers to process consecutive layers. In heterogeneous training, to execute the pipeline with maximum throughput, all stages in the pipeline must spend roughly the same computing time during training, because the pipeline throughput bottleneck is limited by the slowest stage in addition to the impact of the network.

[0053] for Data Centers ,Each data center contains a different number of heterogeneous GPU devices, such as , indicating the data center contain A100 GPUs and T4 GPU. The video memory of NVIDIA A100 is 40GB and that of NVIDIA Tesla T4 is 16GB. Considering the computing power and video memory size of the device, when calling resources, resources are allocated in the ratio of A100:t4=3:1. If NVIDIA Tesla T4 is used as a basic computing power unit, the number is The computing power contained in the data center is: . The total computing power contained in each data center is: By evenly dividing the layers of the model, ideally, the computing power required for each stage However, in reality, due to the differences in performance and quantity of heterogeneous resources in different data centers, it is impossible to achieve ideal equipment allocation. The subsequent steps will introduce the problem of data center splitting in detail.

[0054] The GPUs in the data center are connected through a high-speed dedicated network, but the network connection between devices in different data centers has high latency and low bandwidth. The cost of training the base model mainly comes from the computing cost and the communication between data centers. The computing cost is inevitable. Due to the influence of the heterogeneous network environment, the communication cost of devices between data centers is the main optimization target. The latency and bandwidth of two data centers represent the latency and bandwidth between two devices in different data centers, which are expressed as and .

[0055] Divide the base model into multiple pipeline stages, using Represents the number of bytes of forward propagation activation values ​​communicated when the pipeline is parallel; at the same time, using data parallelism to speed up training, Indicates the size of the gradient that needs to be exchanged between devices in different batches during data parallelism.

[0056] This embodiment treats the data center as an indivisible unit, and a single large data center still has a lot of remaining computing power after taking on a stage computing task. In order to make full use of existing computing resources and achieve maximum resource utilization, this embodiment logically splits the data center according to actual resource usage.

[0057] In this embodiment, if Figure 2 As shown in the figure, after being split, large data centers are logically divided into "medium-sized data centers" and "small data centers", which further standardizes the computing power of the data centers and facilitates resource allocation in subsequent processes.

[0058] Data centers are classified according to the computing power they contain:

[0059] (1) Large data centers: , indicating that the computing power of the data center is greater than the computing power required for a stage task;

[0060] (2) Medium-sized data center: , indicating that the computing power of the data center just meets the computing power required by a stage task;

[0061] (3) Small data center: , indicating that the computing power of the data center cannot meet the computing power required by a stage task;

[0062] in, Represents a classification constraint, which is used to limit the classification boundaries of three data centers. The value range is .

[0063] A medium-sized data center can undertake a stage task, so it is only necessary to split the large data center into one or more logical "medium-sized data centers" and one logical "small data center". Since the devices in the data center are heterogeneous and their computing power is different, when splitting the "medium-sized data center", the computing power of the split "medium-sized data centers" should be as close as possible. For large data centers The splitting continuously extracts the “medium-sized data center” so that its final remaining computing power is less than , that is, a "small data center", the specific process is as follows:

[0064] 1) If the data center only contains T4 GPUs, you only need to find T4 GPU to make its computing power as close as possible ,Right now ;

[0065] 2) If the data center only contains A100 GPUs, you only need to find A100 GPU to make its computing power as close as possible ,Right now ;

[0066] 3) If the data center contains both T4 and A100 GPUs, find A100 GPU and T4 GPU to make its computing power as close as possible , and logically extract it from the data center, repeating this method until the data center is less than , the data center split is completed; specifically:

[0067] (a) If A100 has sufficient resources, to ensure that A100's resources can be fully utilized, first find A100 GPU to make its computing power as close as possible Next, reduce the number of A100 by 1 and use Block T4 adds computing power. Find the computing power in these two cases as close as possible method, that is ;

[0068] (b) If A100 has limited resources, After the A100 GPU, continue to add T4 GPU to make its computing power as close as possible ,Right now .

[0069] 4) Repeat the above process. After all large data centers are split, logically according to the classification standards, all data centers are medium-sized data centers or small data centers.

[0070] From a macro perspective, we use data centers as the basic unit for combination, and use the ant colony algorithm to find a preliminary solution for the combination of data centers to form preliminary parallel groups (data parallel groups and pipeline parallel groups). The degree of indicates the balance of computing power, ensuring that computing resources are maximized; at the same time, the communication cost between data centers in each data parallel group is taken into account.

[0071] Since a medium-sized data center can execute a stage task alone, while small data centers need to be combined to execute a stage task, in order to improve the utilization of computing resources and reduce the communication overhead caused by the combination of data centers, the ant colony algorithm is used to complete the combination of small data centers.

[0072] All small data centers form a complete graph. Assume that the number of medium-sized data centers after splitting is , the number of small data centers is , with Only ants The specific method of allocating computing resources to each stage and executing an ant colony algorithm iteration is as follows:

[0073] (1) At the initial moment, the pheromone concentration on the path in the complete graph is set to .

[0074] (2) Ants will use probability to determine the concentration of pheromones. Access the data center, .in, Indicates that ants are from the data center Move to the data center The degree of expectation, . For Ants The set of data centers to be visited, initially, have elements. is the pheromone importance factor. The larger its value is, the greater the role of pheromone concentration in the transfer; It is the expected function importance factor. The larger its value is, the greater the role of the heuristic function in the transfer is.

[0075] (3) The data centers visited by the ants represent the computing power used in the corresponding stage. Any two ants cannot visit the same data center repeatedly in the same iteration. Let the ants start from a data center randomly and visit the data centers in the complete graph according to the probability calculated by the state transition function until the total computing power of the data centers visited by the ants is greater than or equal to Then, the next ant continues to visit in this way until When only one ant completes the visit of the entire complete graph, an initial parallel group solution is obtained.

[0076] Repeat steps (1) to (3) to obtain 20 parallel group schemes.

[0077] (4) According to the cost function , the total cost of each parallel group solution is calculated; where, Representing Ants The total computing power of the data centers visited, is the weight of computing power balance, is the weight of the communication cost between data centers, Represents ants Walking through the data center, Represents ants Whether from the data center Move to the data center .

[0078] (5) Update pheromones for all solutions. Ants release pheromones when passing through the connection path of the data center, and the pheromones evaporate over time. The amount of pheromone released depends on the total cost of the parallel group solution. The pheromone update method is: .in, represents the total amount of ant pheromone released, Indicates the volatility of pheromones. Indicates that all ants are in the data center The sum of the pheromone concentrations released on the connecting paths. As the pheromone is constantly updated, the pheromone concentration on the path with the lower cost solution will gradually increase, while the pheromone concentration on other paths will decrease due to volatilization, which ensures the convergence of the algorithm.

[0079] (6) For the 20 generated solutions, use a positive incentive strategy to find the three lowest-cost combination strategies from the generated solutions, rank them by cost, and update the pheromone for the solution path. The update method is: Among them, the ranking .

[0080] (7) At this point, one iteration is completed and we get The initial resource allocation strategy for each stage. A medium-sized data center can The computing power is allocated to each stage, and the two allocation strategies are combined to obtain an ant colony algorithm solution.

[0081] Repeat steps (1) to (7) 200 times to obtain 200 ant colony algorithm solutions, which are used as seeds for the subsequent genetic algorithm. The solution obtained by the ant colony initialization allocation algorithm has the most balanced computing resources, and the data centers in the data parallel group have good communication connections.

[0082] In the macroscopic view, the parallel group solution obtained by the ant colony initialization allocation algorithm has the most balanced computing power allocated to each stage, and the communication between data centers in the data parallel group is also good. However, the total communication cost needs to comprehensively consider the data parallel cost and the pipeline parallel cost. Therefore, in the microscopic view, the genetic communication optimization algorithm is used to optimize the overall communication cost and reduce the training time on the basis of computing power balance, and the final parallel group solution is obtained.

[0083] Since the throughput of both pipeline parallelism and data parallelism is limited by the slowest link, and the network bandwidth and latency between devices are different, it is necessary to constantly adjust the arrangement of devices and optimize the total cost of pipeline parallelism and data parallelism to minimize the total communication cost, thereby increasing the throughput of the device and reducing the training time of the base model. The specific method is as follows:

[0084] 1) Use the solution generated by the ant colony algorithm as the initial population. This can ensure the optimal resource balance and reduce the iteration time of the algorithm.

[0085] 2) Define the fitness function of the genetic algorithm as ,in, represents the communication cost of the data parallel group, represents the communication cost of the pipeline parallel group, There are A kind of pipeline communication; specifically:

[0086] a) Communication cost calculation method in data parallel group: For all devices in the group, calculate the sum of the communication cost of the device with other devices, and take the maximum communication cost of the device in the group as the communication cost of the data parallel group (the communication cost between two devices is calculate).

[0087] b) Calculation method of communication cost in pipeline parallel group: The cost of pipeline parallel group occurs in the communication cost of devices between stages (the communication cost between two devices is expressed as The communication cost is affected by the communication volume between the two devices and the latency and bandwidth of the region where the devices are located. The communication cost of the largest device in the pipeline parallelism between the two stages is regarded as the cost of pipeline parallelism. The goal of algorithm optimization is to obtain a seed (solution) with the largest fitness function value as possible. The larger the fitness function value, the better the quality, which means the lower the communication cost.

[0088] c) Different arrangements of pipeline parallel groups will affect the communication efficiency of the pipeline, and thus affect the overall communication cost of the entire system. This problem can be regarded as an OpenLoopTSP problem. Dynamic programming is used to solve this problem: the pipeline parallel groups are regarded as nodes, and the minimum cost from each node to the future node set is calculated recursively, and the results are stored in the dynamic programming table dp_table to avoid repeated calculations. The path information is stored in trace_table. This information can be used to track and reconstruct the optimal path after the calculation is completed, so as to efficiently find the minimum cost path from the starting node to all nodes, and then find the arrangement with the lowest pipeline parallel cost.

[0089] 3) For the 200 solutions in the population, two solutions are selected as parents each time. , for the solution Producing offspring using both “within-group variation” and “between-group variation” , calculate the fitness functions of the four solutions respectively, and select the two best solutions to replace the two parents. This is actually an evolutionary reversal operation. Only those with improved fitness values ​​after mutation are retained, which ensures the continuous improvement of the population quality. The genetic algorithm can continuously evolve to find the optimal solution to reduce the overall communication cost as much as possible.

[0090] The two variations are as follows:

[0091] (a) Intra-group variation

[0092] In a data parallel group, the types of devices and the data centers where they are located are different. Changing the positions of two devices in a data parallel group with a certain probability will not cause a change in the data parallel cost. However, changes in pipeline parallel devices will cause changes in the pipeline parallel cost.

[0093] (b) Variation between groups

[0094] For two devices of the same type in the two pipeline parallel groups of this solution, their positions are exchanged with a certain probability. After the exchange, the computing power of the data parallel group will not change, ensuring the balance of computing power at different stages. The total cost after the exchange will change.

[0095] 4) Repeat the above steps. After 3000 iterations of genetic algorithm updates, the best parallel group scheme is obtained. Based on the obtained best parallel group scheme, the following is formed: Figure 3 The model training architecture shown is used to efficiently train the base model.

[0096] It should be noted that the parallel group scheme corresponds to the model training architecture. The parallel group scheme represents the arrangement of heterogeneous GPU devices, and the arrangement of GPU devices determines the communication connection method between GPU devices during model training, that is, the model training architecture. When the number of devices in the data center and the latency bandwidth between data centers change, the final parallel group scheme also changes, causing the model training architecture (the arrangement of GPU devices and the communication method) to change as well.

[0097] Based on the obtained parallel group solution, the communication connection method between devices can be further determined to generate the model training architecture. Figure 3 As shown in Figure 1, the model training architecture is also the device arrangement and communication architecture. According to the arrangement of GPU devices in the final parallel group solution, the communication connection between GPU devices is customized, and a GPU device is specified to send the activation value or gradient to the next specified GPU device when the forward or backward propagation is completed.

[0098] When a single data center has insufficient resources to complete the base model training, this embodiment combines heterogeneous devices in multiple regional data centers to achieve the combination of heterogeneous GPU devices in multiple data centers in a wide area environment, making it possible to build a large base model and effectively reducing the training cost of the base model.

[0099] This embodiment uses the ant colony algorithm to initialize and combine data centers at a macro level to form a preliminary parallel group arrangement plan, thereby ensuring the balance of computing resources allocated to different stages and good communication within the data parallel group, so that heterogeneous devices in multiple data centers are fully utilized.

[0100] This embodiment uses a genetic algorithm to adjust the arrangement of GPU devices at the micro level, and carefully optimizes the communication costs of data parallelism and pipeline parallelism, which greatly reduces the training time of the model, improves the throughput of the system during operation, and further reduces the training cost of the model.

[0101] Embodiment 2

[0102] Embodiment 2 of the present invention introduces a large-model wide-area heterogeneous distributed training system based on dual optimization.

[0103] A large-model wide-area heterogeneous distributed training system based on dual optimization, comprising:

[0104] An acquisition module, configured to acquire configuration information of a base model;

[0105] A splitting module is configured to split the heterogeneous data center based on the configuration information of the base model, and convert the heterogeneous data center into a data center that can complete at most one stage task;

[0106] A dual optimization module is configured to use an ant colony algorithm to optimize the initial combination of the split data centers to obtain a preliminary parallel group solution for the base model; based on the preliminary parallel group solution obtained by the genetic algorithm optimization, a parallel group solution for the base model is obtained;

[0107] The training module is configured to generate a model training architecture according to the obtained parallel group scheme to complete the wide-area heterogeneous distributed training of large models based on dual optimization.

[0108] The detailed steps are the same as the large-model wide-area heterogeneous distributed training method based on dual optimization provided in Example 1, and will not be repeated here.

[0109] Embodiment 3

[0110] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0111] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in Embodiment 1 of the present invention.

[0112] The detailed steps are the same as the large-model wide-area heterogeneous distributed training method based on dual optimization provided in Example 1, and will not be repeated here.

[0113] Embodiment 4

[0114] A fourth embodiment of the present invention provides an electronic device.

[0115] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in Embodiment 1 of the present invention are implemented.

[0116] The detailed steps are the same as the large-model wide-area heterogeneous distributed training method based on dual optimization provided in Example 1, and will not be repeated here.

[0117] Embodiment 5

[0118] Embodiment 5 of the present invention provides a computer program product.

[0119] A computer program product includes software code, wherein the program in the software code executes the steps in the large-model wide-area heterogeneous distributed training method based on dual optimization as described in Example 1 of the present invention.

[0120] The detailed steps are the same as the large-model wide-area heterogeneous distributed training method based on dual optimization provided in Example 1, and will not be repeated here.

[0121] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A large-model wide-area heterogeneous distributed training method based on dual optimization, characterized in that: include: Get the configuration information of the base model; Split heterogeneous data centers based on the configuration information of the base model; According to the computing power contained in the heterogeneous data centers, the heterogeneous data centers are divided into large data centers, medium data centers and small data centers; the computing power of the large data center is greater than the computing power required for a stage task, the computing power of the medium data center is equal to the computing power required for a stage task, and the computing power of the small data center is less than the computing power required for a stage task; Split a large data center into one or more logically medium-sized data centers and one logically small data center; for a large data center The splitting of the medium-sized data centers is continuously extracted, so that the final remaining computing power is less than , the specific process is as follows: 1) If the data center only contains T4 GPUs, find T4 GPU to make its computing power as close as possible , ; 2) If the data center only contains A100 GPUs, find A100 GPU to make its computing power as close as possible , ; 3) If the data center contains both T4 and A100 GPUs, find A100 GPU and T4 GPU to make its computing power as close as possible , and logically extract it from the data center, repeating this method until the data center is less than , the data center split is completed; 4) Repeat the above process. After all large data centers are split, all data centers are logically classified as medium-sized data centers or small data centers according to the classification standards. in, It is the ideal computing power for each Stage. , indicating the data center contain A100 GPUs and T4 GPUs; heterogeneous GPU devices in multiple data centers, including NVIDIA A100 and NVIDIA Tesla T4; All small data centers form a complete graph, and each ant allocates computing resources to each stage; The ant colony algorithm is used to optimize the initial combination of the split data centers and obtain the preliminary parallel group scheme of the base model; In the process of the initialization combination optimization of the split data center, the data center is taken as the basic unit, and the ant colony algorithm is used to combine the data centers to obtain the data parallel group and the pipeline parallel group with balanced computing resources, and obtain the preliminary parallel group scheme of the base model; Use ant colony algorithm to complete the combination of small data centers; Based on the preliminary parallel group scheme obtained by genetic algorithm optimization, a parallel group scheme for the base model is obtained; The genetic algorithm also uses "intra-group variation" and "inter-group variation" to generate offspring. Considering the data parallel cost and pipeline parallel cost, the genetic communication optimization algorithm is used to optimize the total communication cost of the data center for the preliminary parallel group scheme, and the base model parallel group scheme is obtained; the arrangement of the equipment is continuously adjusted to optimize the total cost of pipeline parallelism and data parallelism, so that the total communication cost is minimized; Based on the obtained parallel group scheme, a model training architecture is generated to complete the wide-area heterogeneous distributed training of large models based on dual optimization.

2. A large-model wide-area heterogeneous distributed training method based on dual optimization as described in claim 1, characterized in that: Based on the obtained parallel group scheme, determine the arrangement of heterogeneous devices and the communication connection method between devices during model training, and generate a model training architecture.

3. A large-model wide-area heterogeneous distributed training system based on dual optimization, characterized in that: include: An acquisition module, configured to acquire configuration information of a base model; A splitting module, which is configured to split the heterogeneous data center based on the configuration information of the base model; According to the computing power contained in the heterogeneous data centers, the heterogeneous data centers are divided into large data centers, medium data centers and small data centers; the computing power of the large data center is greater than the computing power required for a stage task, the computing power of the medium data center is equal to the computing power required for a stage task, and the computing power of the small data center is less than the computing power required for a stage task; Split a large data center into one or more logically medium-sized data centers and one logically small data center; for a large data center The splitting of the medium-sized data centers is continuously extracted, so that the final remaining computing power is less than , the specific process is as follows: 1) If the data center only contains T4 GPUs, find T4 GPU to make its computing power as close as possible , ; 2) If the data center only contains A100 GPUs, find A100 GPU to make its computing power as close as possible , ; 3) If the data center contains both T4 and A100 GPUs, find A100 GPU and T4 GPU to make its computing power as close as possible , and logically extract it from the data center, repeating this method until the data center is less than , the data center split is completed; 4) Repeat the above process. After all large data centers are split, all data centers are logically classified as medium-sized data centers or small data centers according to the classification standards. in, It is the ideal computing power for each Stage. , indicating the data center contain A100 GPUs and T4 GPUs; heterogeneous GPU devices in multiple data centers, including NVIDIA A100 and NVIDIA Tesla T4; All small data centers form a complete graph, and each ant allocates computing resources to each stage; A dual optimization module is configured to use an ant colony algorithm to optimize the initial combination of the split data centers to obtain a preliminary parallel group solution for the base model; based on the preliminary parallel group solution obtained by the genetic algorithm optimization, a parallel group solution for the base model is obtained; In the process of the initialization combination optimization of the split data center, the data center is taken as the basic unit, and the ant colony algorithm is used to combine the data centers to obtain the data parallel group and the pipeline parallel group with balanced computing resources, and obtain the preliminary parallel group scheme of the base model; Use the ant colony algorithm to complete the combination of small data centers; consider the data parallel cost and pipeline parallel cost, use the genetic communication optimization algorithm to optimize the total cost of data center communication for the preliminary parallel group scheme, and obtain the base model parallel group scheme; The genetic algorithm also uses "intra-group variation" and "inter-group variation" to generate offspring. Constantly adjust the arrangement of devices to optimize the total cost of pipeline parallelism and data parallelism to minimize the total communication cost; The training module is configured to generate a model training architecture according to the obtained parallel group scheme to complete the wide-area heterogeneous distributed training of large models based on dual optimization.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the large-model wide-area heterogeneous distributed training method based on dual optimization are implemented as described in any one of claims 1-2.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the large-model wide-area heterogeneous distributed training method based on dual optimization are implemented as described in any one of claims 1-2.

6. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the large-model wide-area heterogeneous distributed training method based on dual optimization as described in any one of claims 1-2.

Images