Cluster-oriented large model parallel method and device and electronic device

Abstract

The invention relates to a cluster-oriented large model parallelization method and device and an electronic device.The method is applied to the field of large models.The method comprises the steps that operator information of a cluster-oriented large model in a preset microprocessing batch and a preset parallelization mode is obtained, the operator information comprises operator time information and operator memory information of operators in the cluster-oriented large model; the cluster comprises one or more types of accelerators; determining an initial operator parallel configuration strategy of a plurality of assembly lines in the large model based on the operator information, model memory information required by the large model and a memory extreme value of an accelerator; and performing recursion processing on the initial operator parallel configuration strategy according to a preset load balancing mode to obtain a target operator parallel configuration strategy, and running the cluster-oriented large model based on the target operator parallel configuration strategy. According to the method and the device, the utilization efficiency of chip calculation performance during large model parallel configuration is improved, and high efficiency and wide application range of large model training are realized.

Description

Technical Field

[0001] This application relates to the field of large models, and particularly to a large model parallel method, device, and electronic device for a cluster. Background Art

[0002] In recent years, deep learning has achieved remarkable results in fields such as computer vision and natural language processing, driving the rapid development of artificial intelligence technology. As the scale of the model continues to increase, the computing power of a single device can no longer meet the training requirements of large-scale models, and distributed training has become an inevitable choice.

[0003] Currently, popular frameworks for distributed training include Megatron, DeepSpeed, and Colossal-AI, etc. These frameworks provide different optimization strategies for the network layer, but engineers need to spend a lot of time adjusting parameters to obtain the optimal parallel configuration, resulting in low efficiency of the large model parallel configuration; and due to the limited search space covered by the network layer, it cannot meet the current requirements of large model distributed parallel configuration, thereby leading to low utilization efficiency of the chip computing performance when the large model is parallel.

[0004] Regarding the problem that the utilization efficiency of the chip computing performance needs to be improved when the large model for a cluster is parallel in the related technology, no effective solution has been proposed yet. Summary of the Invention

[0005] In this embodiment, a large model parallel method, device, and electronic device for a cluster are provided to solve the problem that the utilization efficiency of the chip computing performance needs to be improved when the large model for a cluster is parallel in the related technology.

[0006] In the first aspect, in this embodiment, a large model parallel method for a cluster is provided, and the method includes:

[0007] Obtain operator information of a large model for a cluster under a preset microprocessing batch and a preset parallel mode, where the operator information includes operator time information and operator memory information of operators in the large model for a cluster; the cluster includes one or more types of accelerators;

[0008] Based on the operator information, the model memory information required by the large model, and the memory maximum and minimum values of the accelerator, determine the initial operator parallel configuration strategy of multiple pipelines in the large model;

[0009] According to a preset load balancing method, perform recursive processing on the initial operator parallel configuration strategy to obtain a target operator parallel configuration strategy, and based on the target operator parallel configuration strategy, run the large model for a cluster.

[0010] In some of these embodiments, obtaining operator information of the cluster-oriented large model under a preset microprocessing batch and a preset parallel mode includes:

[0011] Obtaining an accelerator cluster on which the large model runs; the accelerator cluster includes node-based computing operators and communication operators;

[0012] Under a preset microprocessing batch and a preset parallel mode, determining the computing time information and computing operator memory information of the computing operator, and determining the communication time information of the communication operator; the computing operator memory information includes operator activation values; the operator time information includes computing time information and communication time information.

[0013] In some of these embodiments, determining an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory maximum value of the accelerator includes:

[0014] Determining a first cluster memory ratio corresponding to the pipeline under a preset microprocessing batch and a preset parallel mode according to the memory maximum value of the accelerator and the operator memory information in the operator information;

[0015] Determining a second cluster memory ratio corresponding to the pipeline under a preset microprocessing batch and a preset parallel mode according to the memory maximum value of the accelerator, the operator memory information in the operator information, and the operator activation value; the first cluster memory ratio is not greater than the second cluster memory ratio;

[0016] Based on the first cluster memory ratio and the second cluster memory ratio, determining the initial operator parallel configuration strategy.

[0017] In some of these embodiments, the initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines;

[0018] Performing recursive processing on the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, including:

[0019] Obtaining a first training time of the first pipeline and a second training time of the second pipeline;

[0020] Judging whether the time difference between the first training time and the second training time is less than a preset time threshold;

[0021] If the time difference is not less than the preset time threshold, recursively move the operators of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold, obtaining an intermediate operator parallel configuration strategy;

[0022] Based on the preset load balancing method and memory judgment method, process the intermediate operator parallel configuration strategy to determine the target operator parallel configuration strategy.

[0023] In some embodiments, the processing the intermediate operator parallel configuration strategy based on the preset load balancing method and memory judgment method includes:

[0024] When the time difference between the first training time and the second training time is less than the preset time threshold, determine whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds the preset memory threshold;

[0025] If the parallel memory of the first pipeline exceeds the preset memory threshold, update the microprocessing batch;

[0026] Determine whether the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold;

[0027] If the parallel memory of the first pipeline after updating the microprocessing batch does not exceed the preset memory threshold, process the parallel memory of the second pipeline in sequence.

[0028] In some embodiments, the method further includes:

[0029] If the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold, determine the tensor parallel increase calculation time of the first pipeline and determine the operator activation value recalculation time of the first pipeline;

[0030] When it is determined that the tensor parallel increase calculation time is not greater than the operator activation value recalculation time, update the tensor parallel mode of the first pipeline;

[0031] Based on the preset load balancing strategy, process the updated first pipeline.

[0032] In some embodiments, the method further includes:

[0033] Based on the preset load balancing method and memory judgment method, process all pipelines in the intermediate operator parallel configuration strategy in sequence to obtain the target operator parallel configuration strategy.

[0034] Second aspect, in this embodiment, a large model parallel device for a cluster is provided. The device includes: an information acquisition module, an initialization module, and a parallel module;

[0035] The information acquisition module is used to acquire operator information of the large model for a cluster under a preset microprocessing batch and a preset parallel mode. The operator information includes operator time information and operator memory information of the operators in the large model for a cluster; the cluster includes one or more types of accelerators;

[0036] The initialization module is used to determine an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory threshold of the accelerator;

[0037] The parallel module is used to recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and based on the target operator parallel configuration strategy, run the large model for a cluster.

[0038] Third aspect, in this embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the large model parallel method for a cluster described in the first aspect above is implemented.

[0039] Fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the large model parallel method for a cluster described in the first aspect above is implemented.

[0040] Compared with the related art, a large model parallel method, device, and electronic device provided in this embodiment obtain operator information of the large model for a cluster under a preset microprocessing batch and a preset parallel mode, starting from the data at the operator layer. Compared with the network layer method in the prior art, its application range is wider; according to the obtained operator information, model memory information, and the memory threshold of the accelerator, an initial operator parallel configuration strategy corresponding to multiple pipelines is determined. Furthermore, based on the initial operator parallel configuration strategy, a load balancing judgment is made, and a mobile recursive process is performed on the operator layer in the initial operator parallel configuration strategy, so that the utilization efficiency of the chip computing performance during the large model parallel configuration is higher, and the accuracy, efficiency, and application range of training the large model are higher.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0043] Figure 1 is a hardware structure block diagram of a terminal of a large model parallel method for clusters provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of a large model parallel method for clusters provided by an embodiment of the present application;

[0045] Figure 3 is a flowchart of a large model parallel method for hybrid clusters provided by this specific embodiment;

[0046] Figure 4 is a flowchart of a distributed parallel strategy recursion method provided by this specific embodiment;

[0047] Figure 5 is a structure block diagram of a large model parallel device for clusters provided by an embodiment of the present application. Detailed implementation manners

[0048] To more clearly understand the purpose, technical solution, and advantages of the present application, the present application will be described and explained below with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise defined, technical terms or scientific terms involved in this application shall have the general meanings understood by those with ordinary skills in the technical field to which this application belongs. In this application, words such as "a", "an", "one kind", "the", "these", etc. do not indicate a limitation in quantity, and they can be singular or plural. Terms such as "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. Words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Multiple" involved in this application means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Usually, the character " / " indicates that the objects associated before and after are in an "or" relationship. Terms such as "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific sorting of the objects.

[0050] The method embodiment provided in this embodiment can be executed on a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal for a large model parallel method for clusters provided by an embodiment of this application. As Figure 1 shown, the terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 and a memory 104 for storing data. Among them, the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above terminal. For example, the terminal may further include more or fewer components than those shown in Figure 1 the figure, or have a different configuration from that shown in Figure 1 the figure.

[0051] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the large model parallel method for clusters in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include memories remotely disposed relative to the processor 102, and these remote memories can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. The above-mentioned network includes the wireless network provided by the communication provider of the terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] Currently, when training large models for hybrid accelerator clusters, it usually requires relevant engineers to spend a lot of time adjusting the parameters of different optimization strategies corresponding to different frameworks in the accelerator cluster. In order to reduce the investment in manpower and accelerators, in this embodiment, a large model parallel method is proposed. Based on the basic data of accelerators and network bandwidth in the accelerator cluster, a search algorithm is designed to obtain the optimal parallel strategy configuration of the accelerator cluster during large model parallelism.

[0054] Among them, since hybrid clusters often include different accelerators, such as GPU (Graphics Processing Unit), DCU (Data Center Unit), NPU (Neural Processing Unit), and domestic acceleration cards. These heterogeneous accelerator clusters have different impacts on different parallel strategies and training optimization strategies, and the requirements for engineers in large model training are getting higher and higher. By dividing computing tasks, optimizing communication strategies, and scheduling the computing process, this application can significantly improve the efficiency of distributed training, reduce the complexity of manually designing parallel strategies, and effectively improve the utilization efficiency of the computing performance of accelerator chips during large model parallel configuration.

[0055] In this embodiment, a large model parallel method for a cluster is provided. Figure 2 It is a flowchart of the large model parallel method for a cluster provided by an embodiment of this application. As Figure 2 shown, this process includes the following steps S210 to S230.

[0056] Step S210: Obtain the operator information of the large model for a cluster under a preset microprocessing batch and a preset parallel mode. The operator information includes the operator time information and operator memory information of the operators in the large model for a cluster; the cluster includes one or more types of accelerators.

[0057] In this step, the processor starts from the operator level of the large model and divides and deploys the parallel scheme of the large model according to the characteristics of multiple types of accelerators. First, it is necessary to obtain the operator information of each operator in the large model for a cluster under multiple preset microprocessing batches and a preset parallel mode; the preset microprocessing batch here can be 1, 2, 4, or 8. By adopting different microprocessing batches, the computing efficiency and convergence speed are balanced in different parallel processes; for example, when the preset microprocessing batch is 1, it means that only one sample is used for forward propagation and backward propagation at a time; the specific microprocessing batch is determined according to the actual situation and is not specifically limited here.

[0058] The preset parallel modes include Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP). The above parallel modes are all used for distributed training of deep learning models and are mainly used to solve the problem that a single device cannot accommodate a large model or accelerate the training process. In data parallelism, the entire model is replicated on multiple devices, and each device processes a different subset of data. This means that all devices run exactly the same model but use different parts of the training data. Tensor parallelism is a fine-grained parallel strategy that divides the operations inside a single-layer neural network and executes them on different devices. For example, in a matrix multiplication operation, the input matrix can be divided by columns or rows and assigned to different GPUs for calculation. Pipeline parallelism divides the model into several parts according to the levels and distributes these parts to different devices. Each device is responsible for a part of the forward propagation and backward propagation, and then passes the calculation results in sequence like a pipeline.

[0059] By starting from the operator layer, calculate the operator information of the computing operators and communication operators under different microprocessing batches and different parallel modes, such as the computing time information including forward and backward propagation of the computing operators, the operator memory information of the memory occupied during operation, and the communication time information of different communication operators between different accelerators and accelerator nodes, which is not specifically limited here.

[0060] Step S220: Determine the initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory maximum and minimum values of the accelerator.

[0061] In this step, after obtaining the operator information of each operator under different microprocessing batches and different parallel modes, it is necessary to initialize the pipelines under different microprocessing batches and different tensor parallel modes according to the operator information, the model memory information required in the large model, and the memory information of the accelerators in the cluster. Specifically, according to the operator information, the model memory information required by the large model, and the memory maximum and minimum values of the accelerator, determine the maximum and minimum values of the pipelines in the large model, that is, the initial operator parallel configuration strategy within the range of the pipeline maximum and minimum values, thereby defining the scope of the distributed parallel strategy and trimming the search space of its optimization process. Subsequently, recursively optimize the initial operator parallel configuration strategy to improve the efficiency of distributed parallel strategy configuration and optimization.

[0062] Step S230: Recursively process the initial operator parallel configuration strategy according to the preset load balancing method to obtain the target operator parallel configuration strategy, and run the large model for the cluster based on the target operator parallel configuration strategy.

[0063] In this step, after determining the initial operator parallel configuration strategy, it is necessary to recursively process the initial operator parallel configuration strategy in turn according to the memory-aware method to obtain the optimal target operator parallel configuration strategy. Specifically, by calculating the computation times of two adjacent pipelines corresponding to the initial operator parallel configuration strategy in turn, determine whether the difference between the computation times of the current two adjacent pipelines is within the preset threshold range. If it is within the preset threshold range, calculate the computation time of the initial operator parallel configuration strategy corresponding to the next adjacent pipeline. If the difference between the computation times of the current two adjacent pipelines is not within the preset threshold range, move the computation operator in the pipeline with the larger computation time to the other pipeline, and perform iterative loops until the difference between the computation times of the current two adjacent pipelines is within the preset threshold range to achieve the load balancing of the distributed parallel strategy. After the load balancing recursion for all pipelines is completed, obtain the target operator parallel configuration strategy, and run or train the large model according to this strategy.

[0064] Through the above steps, by obtaining the operator information of the large model for the cluster in a preset microprocessing batch and a preset parallel manner, starting from the data at the operator layer, the applicable range is wider compared to the network layer method in the prior art; according to the obtained operator information, model memory information, and the memory threshold of the accelerator, determine the initial operator parallel configuration strategy corresponding to multiple pipelines, and then, based on the initial operator parallel configuration strategy, perform load balancing judgment and perform mobile recursive processing on the operator layer in the initial operator parallel configuration strategy, so that the utilization efficiency of the chip computing performance during the parallel configuration of the large model is higher and the accuracy is higher.

[0065] In some of the embodiments, obtaining the operator information of the large model for the cluster in a preset microprocessing batch and a preset parallel manner includes: obtaining the accelerator cluster running the large model; the accelerator cluster includes node-based computing operators and communication operators; in the preset microprocessing batch and the preset parallel manner, determine the computing time information and computing operator memory information of the computing operators, and determine the communication time information of the communication operators; the computing operator memory information includes operator activation values; the operator time information includes the computing time information and the communication time information.

[0066] Exemplarily, the computing operators include: Embedding, 1st-layernorm, Attention-qkv, Flash-attention, Attention-dense, Attention-dropout, 2nd-layernorm, MLP-GEMM-1, and MLP-gelu, etc., and the communication operators include: all-gather, reduce-scatter, p2p, allreduce, etc. The types of the computing operators and the communication operators are determined according to the actual parallel requirements of the large model, and the types and quantities of the computing operators and the communication operators are not specifically limited herein. The computing time information of the computing operators includes the computing time information of the forward propagation and backward propagation of the computing operators, and the computing operator memory information of the computing operators includes the input, output, weights, operator activation values, and reserved memory occupied by the computing operators; the communication time information of the communication operators includes time information such as the speed between different accelerators and nodes; exemplarily, the communication time information of the communication operators includes the time information of the communication operators under different communication data sizes, such as the communication data sizes of 128M, 256M, etc.

[0067] In some of these embodiments, based on the operator information, the model memory information required by the large model, and the memory maximum value of the accelerator, an initial operator parallel configuration strategy for multiple pipelines in the large model is determined, including: determining a first cluster memory ratio corresponding to a pipeline under a preset microprocessing batch and a preset parallel manner according to the memory maximum value of the accelerator and the operator memory information in the operator information; determining a second cluster memory ratio corresponding to a pipeline under a preset microprocessing batch and a preset parallel manner according to the memory maximum value of the accelerator, the operator memory information in the operator information, and the operator activation value; the first cluster memory ratio is not greater than the second cluster memory ratio; based on the first cluster memory ratio and the second cluster memory ratio, an initial operator parallel configuration strategy is determined.

[0068] In this embodiment, after obtaining the operator information of the large model under multiple microprocessing batches and multiple parallel manners, according to the operator information, the memory information required by the large model, and the memory maximum value of the accelerator, the first cluster memory ratio of the current pipeline is calculated, and this first cluster memory ratio is the minimum value of the pipelines under different microprocessing batches and different parallel manners; meanwhile, according to the operator information, the memory information required by the large model, the memory maximum value of the accelerator, and the operator activation value of the calculation operator, the second cluster memory ratio of the current pipeline is calculated, and this second cluster memory ratio is the minimum value of the pipelines under different microprocessing batches and different parallel manners.

[0069] Exemplarily, first, according to the obtained operator information, memory information, and memory size of the accelerator of the large model, the first cluster memory ratio and the second cluster memory ratio of the pipeline under different tensor parallel manners and different microprocessing batches in the homogeneous cluster are determined. The calculation formulas for the first cluster memory ratio and the second cluster memory ratio in the homogeneous cluster are as follows:

[0070] ;

[0071] ;

[0072] Among them, represents the maximum video memory of the accelerator, act(o) represents the size of the video memory occupied by the operator activation value of the calculation operator (i.e., the size of the video memory occupied by the presence or absence of the activation value), mem(o) represents the operator memory information in the operator information, that is, the operator video memory size. The specific calculation method is as follows:

[0073] ;

[0074] Among them, represents the operator input memory information of the calculation operator. represents the operator gradient memory information of the calculation operator. Represents the operator weight memory information of the computing operator. Further, the operator optimizer state memory information is determined based on the operator weight memory information, and the ratio of the operator weight memory information to the operator optimizer state memory information is 1:8. Represents the operator optimizer state memory information of the computing operator.

[0075] After calculating the first cluster memory ratio and the second cluster memory ratio of the pipelines under different tensor parallel modes and different microprocessing batches of the homogeneous cluster, that is, after initializing the homogeneous cluster, the large model needs to be initialized with the number of heterogeneous clusters in the way of computing power. Inside different clusters, the maximum and minimum values of the pipelines under different microprocessing batches, data parallelism, and tensor parallelism of the homogeneous cluster are further obtained in the way of the homogeneous cluster, and then the parallel strategy under the heterogeneous cluster is initialized to obtain the initial operator parallel configuration strategy under the heterogeneous cluster.

[0076] In some of these embodiments, the initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines; the first pipeline and the second pipeline here are only used to distinguish two adjacent pipelines schematically. In the actual process, there are multiple pipelines, not limited to the first pipeline and the second pipeline.

[0077] According to the preset load balancing method, the initial operator parallel configuration strategy is recursively processed to obtain the target operator parallel configuration strategy, including: obtaining the first training time of the first pipeline and the second training time of the second pipeline; judging whether the time difference between the first training time and the second training time is less than the preset time threshold; if the time difference is not less than the preset time threshold, recursively move the operators of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold to obtain an intermediate operator parallel configuration strategy; based on the preset load balancing method and memory judgment method, the intermediate operator parallel configuration strategy is processed to determine the target operator parallel configuration strategy.

[0078] In this embodiment, after determining the initial operator parallel configuration strategy, it is necessary to first judge the training time information in the initial operator parallel configuration strategies corresponding to two adjacent pipelines in turn according to the preset load balancing method. For example, calculate the training times corresponding to the first pipeline and the second pipeline, and judge whether the time difference between the first training time and the second training time is less than the preset time threshold, so as to determine whether the load balance is achieved between the first pipeline and the second pipeline. When the time difference is not less than the preset time threshold, it means that the load balance is not achieved between the first pipeline and the second pipeline at this time. Therefore, it is necessary to move the operators in the first pipeline to the second pipeline for cyclic iteration until the time difference is less than the preset time threshold, indicating that the load balance has been achieved between the first pipeline and the second pipeline at this time. Then continue to judge and update the load balance of the second pipeline and the third pipeline adjacent to the second pipeline according to the above load balancing method until the initial operator parallel configuration strategies corresponding to all pipelines reach the load balance. At this time, the target operator parallel configuration strategy is obtained.

[0079] Exemplarily, the formula for calculating the training time T of each pipeline is as follows:

[0080] ;

[0081] Where, represents the calculation time of each pipeline, and there are P in total. represents the communication time between the i-th pipeline and the (i + 1)-th pipeline, B represents the number of microprocessing batches. represents the total calculation time of P pipelines, that is, the time for the large model training iteration.

[0082] The above preset load balancing strategy can be expressed by the following inequality:

[0083] ;

[0084] Where, represents the calculation time of the i-th pipeline. represents the calculation time of the j-th pipeline. The i-th pipeline and the j-th pipeline are adjacent pipelines. When the calculation times of two adjacent pipelines are equal, the training time T of each pipeline is also the smallest, making the time for the large model training iteration also the smallest.

[0085] Furthermore, in the actual processing process, it is impossible to strictly make the calculation time of the adjacent i-th pipeline and the calculation time of the j-th pipeline exactly the same. Therefore, it is necessary to improve it with a preset time threshold, as shown in the following formula:

[0086] ;

[0087] Among them, represents the computing time of the i-th pipeline, represents the computing time of the j-th pipeline, represents a preset time threshold.

[0088] After recursively processing the initial operator parallel configuration strategy through a preset load balancing method, an intermediate operator parallel configuration strategy is obtained; subsequently, it is necessary to combine the memory judgment method and the load balancing method to process the intermediate operator parallel configuration strategy to obtain the target operator parallel configuration strategy.

[0089] In some of these embodiments, processing the intermediate operator parallel configuration strategy based on a preset load balancing method and a memory judgment method includes:

[0090] When the time difference between the first training time and the second training time is less than the preset time threshold, determine whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds the preset memory threshold; if the parallel memory of the first pipeline exceeds the preset memory threshold, update the microprocessing batch; determine whether the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold; if the parallel memory of the first pipeline after updating the microprocessing batch does not exceed the preset memory threshold, sequentially process the parallel memory of the second pipeline.

[0091] If the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold, determine the increased computing time of tensor parallelism of the first pipeline and determine the recomputation time of the operator activation value of the first pipeline; when it is determined that the increased computing time of tensor parallelism is not greater than the recomputation time of the operator activation value, update the tensor parallelism method of the first pipeline; based on the preset load balancing strategy, process the updated first pipeline.

[0092] Among them, when an intermediate operator parallel configuration strategy that meets the load balancing is obtained, it is necessary to calculate whether the parallel configuration strategy corresponding to each pipeline exceeds the parallel memory of the corresponding accelerator. If it exceeds the memory limit, then determine whether the microprocessing batch can be reduced. If the microprocessing batch cannot be reduced, then enter the update of tensor parallelism and recomputation of the operator activation value. If it does not exceed the memory limit, then re-perform the load balancing of the pipeline. If the tensor parallelism of the first pipeline is selected, update and recompute the larger one between the increased computing time and the recomputation time of the operator activation value, and perform load balancing judgment and memory judgment on the updated and recomputed first pipeline until the first pipeline meets the above load balancing and memory judgment methods, and then perform load balancing and memory judgment on the second pipeline adjacent to the first pipeline.

[0093] Based on the preset load balancing method and memory judgment method, after processing all the pipelines in the intermediate operator parallel configuration strategy in sequence, a target operator parallel configuration strategy is obtained, and the accelerators in the hybrid cluster are run according to the target operator parallel configuration strategy to train the large model.

[0094] The following describes and illustrates this embodiment through specific examples.

[0095] Figure 3 It is a flowchart of the large model parallel method for the hybrid cluster provided by this specific embodiment. Refer to Figure 3 This large model parallel method includes the following steps.

[0096] Step S310, obtaining large model and cluster operator information.

[0097] Specifically, based on the cluster and the large model, the forward calculation time, backward calculation time, input, weight, activation value, gradient, memory occupied by the optimizer state, and memory reserved by the framework of each operator under different micro - processing batches in different parallel strategies, such as data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP), are pre - calculated, as well as the speeds and calculation times of different communication operators (all - gather, reduce - scatter, p2p, allreduce, etc.) between different accelerators and nodes. Compared with the model data in the current model parallel method, the model test parameters of the solution provided in this embodiment can reach 10 billion, and for the operator level of the hybrid cluster, multiple factors such as operator activation recomputation, load balancing, and memory limitations can be considered to determine the parallel strategy. Refer to Table 1, which is the operator information of the GPT3 model with a data volume of 7 billion when the micro - processing batch is 1 and both data parallelism and tensor parallelism are 1.

[0098] Table 1 Calculation time and memory information of computational operators

[0099]

[0100] Step S320, initializing the maximum and minimum values of the pipelines under different tensor parallelisms and micro - processing batches.

[0101] Specifically, according to the memory information required by the model obtained in step S310 and the memory size of the accelerator, the maximum and minimum values of the pipelines under different tensor parallelisms and micro - processing batches are initialized. Refer to the solution for determining the first cluster memory ratio (minimum value) and the second cluster memory ratio (maximum value) in the foregoing embodiment, which will not be elaborated here.

[0102] Step S330, recursively obtaining the optimal solution according to the memory - aware distributed parallel strategy.

[0103] Specifically, Figure 4 is a flowchart of the distributed parallel strategy recursive method provided by this specific embodiment. Refer to Figure 4 , this step includes the following details.

[0104] Step Q1: For the initialization parallel configuration of each pipeline, determine whether the load is balanced. If the load is balanced, proceed to the next step. If the load balance is not satisfied, move the operator of the current pipeline to the subsequent pipeline until the load is balanced between the current two adjacent pipelines.

[0105] Exemplarily, the formula for calculating the training time T of each pipeline is as follows:

[0106] ;

[0107] Where, represents the computing time of each pipeline, and there are P in total. represents the communication time between the i-th pipeline and the (i + 1)-th pipeline, B represents the number of microprocessing batches. represents the total computing time of P pipelines, that is, the time for large model training iteration.

[0108] In the actual processing process, it is impossible to strictly make the computing time of the adjacent i-th pipeline and the j-th pipeline exactly the same. Therefore, it is necessary to improve with a preset time threshold. The above preset load balancing strategy can be shown as the following formula:

[0109] ;

[0110] Where, represents the computing time of the i-th pipeline. represents the computing time of the j-th pipeline. represents the preset time threshold.

[0111] Step Q2: Calculate whether each pipeline exceeds the video memory of the corresponding device. If it exceeds the memory limit (OOM), then by determining whether the microprocessing batch can be reduced. If the microprocessing batch can be reduced, then go back to the beginning to determine whether the load is balanced. If the reduction cannot be performed, then enter the next step of tensor parallel and activation value recalculation update.

[0112] Step Q3: Determine whether the updated microprocessing batch exceeds the memory limit. If not, enter the next adjacent pipeline. If so, enter the tensor parallel and operator activation value recalculation update.

[0113] Step Q4, pre-compute the pipeline time for increasing tensor parallelism and recomputing activation values. If the time for increasing tensor parallelism is greater than the pipeline time for recomputing activation values, update tensor parallelism; otherwise, update the operator activation value recomputation. After the updates of tensor parallelism and activation value recomputation are completed, proceed to the load balancing judgment. Determine whether the current pipeline is load balanced and whether the memory limit is exceeded. If both conditions are met, proceed to the calculation of the next pipeline and determine whether the next pipeline is the last one. If so, obtain the operator parallel configuration strategy with the least time from the resource pool including multiple operator parallel configuration strategies according to the above formula for pipeline calculation time, i.e., the optimal configuration, and train the large model.

[0114] In the above solution, to address the problem of low training efficiency caused by the non-uniform computing power, memory, and communication of different accelerator units in a heterogeneous cluster, operator-level automatic parallelism with heterogeneous resource awareness is carried out to improve the training efficiency of large models on a heterogeneous cluster, resulting in a higher MFU (Model FLOPs Utilization), that is, improving the efficiency of using the chip computing performance during the training of large models. At the same time, to address the problem of long search time caused by the expansion of the automatic parallel search space, a heuristic memory-aware search is proposed, and the search process significantly improves the search speed by caching the operator information of profiled (instrumented code). Further, optimization techniques such as flash attention and distributed optimizer are integrated, which is beneficial to improving the efficiency of training large models.

[0115] In this embodiment, a large model parallel device for a cluster is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated here. Terms such as "module", "unit", "sub-unit", etc. hereinafter can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0116] Figure 5 is the structural block diagram of the large model parallel device for a cluster provided by the embodiments of the present application. As Figure 5 shown, the device includes: an information acquisition module 10, an initialization module 20, and a parallel module 30.

[0117] The information acquisition module 10 is configured to acquire operator information of a large model for a cluster under a preset microprocessing batch and a preset parallel manner. The operator information includes the operator time information and operator memory information of the operators in the large model for a cluster; the cluster includes one or more types of accelerators.

[0118] An initialization module 20, configured to determine an initial operator parallel configuration strategy for multiple pipelines in a large model based on operator information, model memory information required by the large model, and the memory threshold of the accelerator.

[0119] A parallel module 30, configured to recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and run a large model for a cluster based on the target operator parallel configuration strategy.

[0120] It should be noted that the above-mentioned each module can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, the above-mentioned each module can be located in the same processor; or the above-mentioned each module can also be located in different processors in any combined form.

[0121] In this embodiment, an electronic device is further provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0122] Optionally, the above-mentioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above-mentioned processor, and the input / output device is connected to the above-mentioned processor.

[0123] Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

[0124] S1. Obtain operator information of a large model for a cluster under a preset microprocessing batch and a preset parallel mode, where the operator information includes operator time information and operator memory information of the operators in the large model for a cluster; the cluster includes one or more types of accelerators.

[0125] S2. Determine an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory threshold of the accelerator.

[0126] S3. Recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and run a large model for a cluster based on the target operator parallel configuration strategy.

[0127] It should be noted that specific examples in this embodiment can refer to the examples described in the above embodiment and optional implementation manners, and will not be elaborated herein.

[0128] In addition, in combination with the large model parallel method for clusters provided in the above embodiments, a storage medium can also be provided in this embodiment to implement it. A computer program is stored on the storage medium; when the computer program is executed by a processor, any one of the large model parallel methods for clusters in the above embodiments is implemented.

[0129] It should be understood that the specific embodiments described here are only used to explain this application, rather than to limit it. According to the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of this application.

[0130] Obviously, the drawings are only some examples or embodiments of this application. For those of ordinary skill in the art, this application can also be applied to other similar situations based on these drawings without creative work. Additionally, it can be understood that although the work done during the development process here may be complex and time-consuming, for those of ordinary skill in the art, certain design, manufacturing, or production changes based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient disclosure of this application.

[0131] The term "embodiment" in this application means that the specific features, structures, or characteristics described in combination with the embodiment may be included in at least one embodiment of this application. The phrase appears in various positions in the specification and does not necessarily mean the same embodiment, nor does it mean being independent or alternative to other embodiments and mutually exclusive. Those of ordinary skill in the art can clearly or implicitly understand that the embodiments described in this application can be combined with other embodiments without conflict.

[0132] The above-described embodiments only represent several implementation manners of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of patent protection. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. A large model parallel method for clusters, characterized in that, The method includes: Obtaining operator information of a cluster-oriented large model under a preset microprocessing batch and a preset parallel mode, where the operator information includes operator time information and operator memory information of operators in the cluster-oriented large model; the cluster includes one or more types of accelerators; Determining an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the maximum and minimum memory values of the accelerator; Performing recursive processing on the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and running the cluster-oriented large model based on the target operator parallel configuration strategy.

2. The parallel method for large models oriented to clusters according to claim 1, wherein, The obtaining of the operator information of the cluster-oriented large model under a preset microprocessing batch and a preset parallel mode includes: Obtaining an accelerator cluster for running the large model; the accelerator cluster includes node-based computing operators and communication operators; Determining the computing time information and computing operator memory information of the computing operators, and determining the communication time information of the communication operators under a preset microprocessing batch and a preset parallel mode; the computing operator memory information includes operator activation values; the operator time information includes computing time information and communication time information.

3. The parallel method for large models oriented to clusters according to claim 2, wherein The determining of the initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the maximum and minimum memory values of the accelerator includes: Determining a first cluster memory ratio corresponding to a pipeline under a preset microprocessing batch and a preset parallel mode according to the maximum and minimum memory values of the accelerator and the operator memory information in the operator information; Determining a second cluster memory ratio corresponding to a pipeline under a preset microprocessing batch and a preset parallel mode according to the maximum and minimum memory values of the accelerator, the operator memory information in the operator information, and the operator activation values; the first cluster memory ratio is not greater than the second cluster memory ratio; Determining the initial operator parallel configuration strategy based on the first cluster memory ratio and the second cluster memory ratio.

4. The parallel method for large models oriented to clusters according to claim 1, characterized in that The initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines; The performing of recursive processing on the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy includes: Obtaining a first training time of the first pipeline and a second training time of the second pipeline; Judging whether the time difference between the first training time and the second training time is less than a preset time threshold; If the time difference is not less than the preset time threshold, recursively moving the operators of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold to obtain an intermediate operator parallel configuration strategy; Processing the intermediate operator parallel configuration strategy based on a preset load balancing method and a memory judgment method to determine the target operator parallel configuration strategy.

5. The parallel method for large models oriented to clusters according to claim 4, characterized in that, Processing the intermediate operator parallel configuration policy based on a preset load balancing method and memory judgment method includes: When the time difference between the first training time and the second training time is less than a preset time threshold, determining whether the parallel memory of the first pipeline of the intermediate operator parallel configuration policy exceeds a preset memory threshold; If the parallel memory of the first pipeline exceeds the preset memory threshold, updating the microprocessing batch; Determining whether the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold; If the parallel memory of the first pipeline after updating the microprocessing batch does not exceed the preset memory threshold, sequentially processing the parallel memory of the second pipeline.

6. The parallel method for large models oriented to clusters according to claim 5, characterized in that The method further includes: If the parallel memory of the first pipeline after updating the microprocessing batch exceeds the preset memory threshold, determining the tensor parallel increase calculation time of the first pipeline and the operator activation value recalculation time of the first pipeline; When it is determined that the tensor parallel increase calculation time is not greater than the operator activation value recalculation time, updating the tensor parallel mode of the first pipeline; Processing the updated first pipeline based on a preset load balancing strategy.

7. The parallel method for large models oriented to clusters according to claim 5, characterized in that The method further includes: After sequentially processing all pipelines in the intermediate operator parallel configuration policy based on a preset load balancing method and memory judgment method, obtaining the target operator parallel configuration policy.

8. A large model parallel device for a cluster, characterized in that The device includes: an information acquisition module, an initialization module, and a parallel module; The information acquisition module is configured to acquire operator information of a large model for a cluster under a preset microprocessing batch and a preset parallel mode, where the operator information includes operator time information and operator memory information of operators in the large model for the cluster; the cluster includes one or more types of accelerators; The initialization module is configured to determine an initial operator parallel configuration policy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory threshold of the accelerator; The parallel module is configured to recursively process the initial operator parallel configuration policy according to a preset load balancing method to obtain a target operator parallel configuration policy, and run the large model for the cluster based on the target operator parallel configuration policy.

9. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for parallelizing a large model for a cluster according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method for parallelizing a large model for a cluster according to any one of claims 1 to 7 are implemented.

Images