Cluster-oriented large model parallel method, device and electronic device

Abstract

The present application relates to a large model parallel method, device and electronic device for clusters, which are applied to the field of large models, wherein the method includes: obtaining operator information of the large model for the cluster under a preset microprocessing batch and a preset parallel mode, the operator information including operator time information and operator memory information of the operator in the large model for the cluster; the cluster includes one or more types of accelerators; based on the operator information, the model memory information required by the large model and the maximum memory value of the accelerator, the initial operator parallel configuration strategy of multiple pipelines in the large model is determined; according to the preset load balancing method, the initial operator parallel configuration strategy is recursively processed to obtain the target operator parallel configuration strategy, and based on the target operator parallel configuration strategy, the large model for the cluster is run. Through this application, the utilization efficiency of the chip computing performance during the parallel configuration of the large model is improved, and the high efficiency and wide applicability of the large model training are achieved.

Description

Technical Field

[0001] The present application relates to the field of large models, and in particular to a cluster-oriented large model parallel method, device and electronic device. Background Art

[0002] In recent years, deep learning has achieved remarkable results in fields such as computer vision and natural language processing, driving the rapid development of artificial intelligence technology. As model size continues to increase, the computing power of a single device can no longer meet the training requirements of large-scale models, making distributed training an inevitable choice.

[0003] Currently, popular frameworks for distributed training include Megatron, DeepSpeed, and Colossal-AI. These frameworks provide different optimization strategies for the network layer, but require engineers to spend a lot of time adjusting parameters to obtain the optimal parallel configuration, resulting in low efficiency in parallel configuration of large models. In addition, due to the limited search space covered by the network layer, it cannot meet the current requirements for distributed parallel configuration of large models, which in turn leads to low utilization efficiency of chip computing performance when large models are run in parallel.

[0004] There is a problem in related technologies that the utilization efficiency of chip computing performance needs to be improved when large cluster-oriented models are run in parallel, and no effective solution has been proposed so far. Summary of the Invention

[0005] In this embodiment, a cluster-oriented large model parallel method, device and electronic device are provided to solve the problem in the related art that the utilization efficiency of chip computing performance needs to be improved when parallelizing cluster-oriented large models.

[0006] First, in this embodiment, a cluster-oriented large model parallel method is provided, the method comprising:

[0007] Obtaining operator information of a cluster-oriented large model in a preset microprocessing batch and a preset parallel mode, wherein the operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators;

[0008] Determining an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, model memory information required by the large model, and the memory maximum of the accelerator;

[0009] According to a preset load balancing method, the initial operator parallel configuration strategy is recursively processed to obtain a target operator parallel configuration strategy, and the cluster-oriented large model is run based on the target operator parallel configuration strategy.

[0010] In some embodiments, obtaining operator information of a cluster-oriented large model in a preset micro-processing batch and a preset parallel mode includes:

[0011] Acquire an accelerator cluster for running the large model; the accelerator cluster includes node-based computing operators and communication operators;

[0012] Under a preset microprocessor batch and a preset parallel mode, the calculation time information and the calculation operator memory information of the calculation operator are determined, and the communication time information of the communication operator is determined; the calculation operator memory information includes the operator activation value; the operator time information includes the calculation time information and the communication time information.

[0013] In some embodiments, determining an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, model memory information required by the large model, and the maximum memory value of the accelerator includes:

[0014] Determining a first cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode according to the maximum memory value of the accelerator and the operator memory information in the operator information;

[0015] determining, based on the maximum memory value of the accelerator, the operator memory information in the operator information, and the operator activation value, a second cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode; wherein the first cluster memory ratio is not greater than the second cluster memory ratio;

[0016] The initial operator parallel configuration strategy is determined based on the first cluster memory ratio and the second cluster memory ratio.

[0017] In some embodiments, the initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines;

[0018] The initial operator parallel configuration strategy is recursively processed according to a preset load balancing method to obtain a target operator parallel configuration strategy, including:

[0019] Obtaining a first training time of the first pipeline and a second training time of the second pipeline;

[0020] Determine whether the time difference between the first training time and the second training time is less than a preset time threshold;

[0021] If the time difference is not less than the preset time threshold, recursively move the operators of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold, thereby obtaining a parallel configuration strategy for intermediate operators;

[0022] Based on a preset load balancing method and a memory judgment method, the intermediate operator parallel configuration strategy is processed to determine the target operator parallel configuration strategy.

[0023] In some embodiments, the processing of the intermediate operator parallel configuration strategy based on a preset load balancing method and a memory determination method includes:

[0024] When the time difference between the first training time and the second training time is less than a preset time threshold, determining whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds a preset memory threshold;

[0025] If the parallel memory of the first pipeline exceeds a preset memory threshold, updating the micro-processing batch;

[0026] determining whether the parallel memory of the first pipeline after the microprocessor batch is updated exceeds a preset memory threshold;

[0027] If the parallel memory of the first pipeline after the updated micro-processing batch does not exceed the preset memory threshold, the parallel memory of the second pipeline is processed in sequence.

[0028] In some embodiments, the method further comprises:

[0029] If the parallel memory of the first pipeline after the updated microprocessor batch exceeds a preset memory threshold, determining a tensor parallel increase calculation time of the first pipeline and determining an operator activation value recalculation time of the first pipeline;

[0030] When it is determined that the tensor parallel increase calculation time is not greater than the operator activation value recalculation time, updating the tensor parallel mode of the first pipeline;

[0031] Based on the preset load balancing strategy, the updated first pipeline is processed.

[0032] In some embodiments, the method further comprises:

[0033] Based on a preset load balancing method and a memory judgment method, all pipelines in the intermediate operator parallel configuration strategy are processed in sequence to obtain the target operator parallel configuration strategy.

[0034] In a second aspect, a large-scale model parallel device for clusters is provided in this embodiment, the device comprising: an information acquisition module, an initialization module, and a parallel module;

[0035] The information acquisition module is used to obtain operator information of a cluster-oriented large model in a preset microprocessing batch and a preset parallel mode, wherein the operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators;

[0036] The initialization module is configured to determine an initial operator parallel configuration strategy for multiple pipelines in the large model based on the operator information, the model memory information required by the large model, and the memory threshold of the accelerator;

[0037] The parallel module is used to recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and run the cluster-oriented large model based on the target operator parallel configuration strategy.

[0038] In a third aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the cluster-oriented large model parallel method described in the first aspect is implemented.

[0039] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the cluster-oriented large model parallel method described in the first aspect is implemented.

[0040] Compared with the related art, a cluster-oriented large model parallel method, device and electronic device provided in this embodiment obtains operator information of a cluster-oriented large model under a preset microprocessing batch and a preset parallel mode, starting from the data of the operator layer. Compared with the network layer method in the prior art, its applicability is wider; based on the obtained operator information, model memory information and the memory threshold of the accelerator, the initial operator parallel configuration strategy corresponding to multiple pipelines is determined, and then based on the initial operator parallel configuration strategy, load balancing judgment is performed, and the operator layer in the initial operator parallel configuration strategy is moved recursively, so that the chip computing performance is more efficiently utilized when the large model is configured in parallel, and the accuracy and efficiency of training the large model are higher and the applicability is wider.

[0041] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of a terminal for a cluster-oriented large model parallel method provided by an embodiment of the present application;

[0044] Figure 2 This is a flow chart of a cluster-oriented large model parallel method provided by an embodiment of the present application;

[0045] Figure 3 This is a flow chart of the large model parallel method for hybrid clusters provided in this specific embodiment;

[0046] Figure 4 is a flow chart of the distributed parallel strategy recursive method provided in this specific embodiment;

[0047] Figure 5 This is a structural block diagram of a cluster-oriented large-model parallel device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0049] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 This is a hardware block diagram of the terminal of the cluster-oriented large model parallel method provided by the embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0051] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the cluster-oriented large model parallel method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0052] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] Currently, training large models for hybrid accelerator clusters typically requires engineers to spend a significant amount of time adjusting parameters for different optimization strategies across different frameworks within the accelerator cluster. To reduce both manpower and accelerator investment, this embodiment proposes a large-model parallelization method. Based on basic data about the accelerators and network bandwidth within the accelerator cluster, a search algorithm is designed to determine the optimal parallelization strategy configuration for the accelerator cluster when running large models in parallel.

[0054] Among them, since hybrid clusters often include different accelerators, such as GPUs (Graphics Processing Units), DCUs (Data Center Units), NPUs (Neural Processing Units), and domestic acceleration cards, these heterogeneous accelerator clusters have different impacts on different parallel strategies and training optimization strategies, and the requirements for engineers training large models are increasingly high. This application can significantly improve the efficiency of distributed training by dividing computing tasks, optimizing communication strategies, and scheduling computing processes, reducing the complexity of manually designed parallel strategies, and effectively improving the utilization efficiency of accelerator chip computing performance when configuring large models in parallel.

[0055] In this embodiment, a large model parallel method for clusters is provided. Figure 2 This is a flow chart of a cluster-oriented large model parallel method provided by an embodiment of the present application. Figure 2 As shown, the process includes the following steps S210 to S230.

[0056] Step S210, obtaining operator information of the cluster-oriented large model under a preset micro-processing batch and a preset parallel mode, the operator information including operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators.

[0057] In this step, the processor starts from the operator level of the large model and divides and deploys the parallel scheme of the large model according to the characteristics of various types of accelerators. First, it is necessary to obtain the operator information of each operator in the cluster-oriented large model under multiple preset microprocessing batches and preset parallel modes; the preset microprocessing batch here can be 1, 2, 4 or 8. By adopting different microprocessing batches, the computing efficiency and convergence speed are balanced in different parallel processing; for example, when the preset microprocessing batch is 1, it means that only one sample is forward propagated and backward propagated at a time; the specific microprocessing batch is determined according to the actual situation and is not specifically limited here.

[0058] Pre-defined parallelism methods include data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP). All of these methods are used for distributed training of deep learning models, primarily to address the problem of large models not being able to fit on a single device or to accelerate the training process. In data parallelism, the entire model is replicated across multiple devices, with each device processing a different subset of the data. This means that all devices run the exact same model but use different portions of the training data. Tensor parallelism is a fine-grained parallelism strategy that splits operations within a single-layer neural network across different devices. For example, in a matrix multiplication operation, the input matrix can be partitioned by columns or rows and assigned to different GPUs for computation. Pipeline parallelism divides the model into several layers and distributes these layers across different devices. Each device performs a portion of the forward and backward propagation passes, and the results are then passed along in a pipeline-like manner.

[0059] Starting from the operator layer, the operator information of the computing operators and communication operators in different microprocessor batches and different parallel modes is calculated, such as the computing time information of the computing operators including forward and backward propagation, the operator memory information of the memory occupied by the operation, and the communication time information of different communication operators between different accelerators and accelerator nodes. No specific restrictions are made here.

[0060] Step S220 , based on the operator information, the model memory information required by the large model, and the maximum memory value of the accelerator, determine the initial operator parallel configuration strategy for multiple pipelines in the large model.

[0061] In this step, after obtaining the operator information of each operator in different microprocessing batches and different parallel modes, it is necessary to initialize the pipelines in different microprocessing batches and different tensor parallel modes based on the operator information, the model memory information required in the large model, and the memory information of the accelerator in the cluster; specifically, based on the operator information, the model memory information required in the large model, and the maximum memory value of the accelerator, determine the maximum and minimum values ​​of the pipeline in the large model, that is, the initial operator parallel configuration strategy within the maximum and minimum value range of the pipeline, and then define the scope of the distributed parallel strategy, and tailor the search space of its optimization process; thereafter, recursively optimize the initial operator parallel configuration strategy to improve the efficiency of distributed parallel strategy configuration and optimization.

[0062] In step S230 , the initial operator parallel configuration strategy is recursively processed according to a preset load balancing method to obtain a target operator parallel configuration strategy, and the cluster-oriented large model is run based on the target operator parallel configuration strategy.

[0063] In this step, after determining the initial operator parallel configuration strategy, it is necessary to recursively perform the initial operator parallel configuration strategy in sequence according to the memory perception method to obtain the optimal target operator parallel configuration strategy. Specifically, by successively calculating the calculation time of the two adjacent pipelines corresponding to the initial operator parallel configuration strategy, it is determined whether the difference in the calculation time of the two adjacent pipelines is within the preset threshold range. If it is within the preset threshold range, the calculation time of the initial operator parallel configuration strategy corresponding to the next adjacent pipeline is calculated; if the difference in the calculation time of the two adjacent pipelines is not within the preset threshold range, the calculation operator in the pipeline with the larger calculation time is moved to another pipeline, and the loop is iterated until the difference in the calculation time of the two adjacent pipelines is within the preset threshold range, thereby achieving load balancing of the distributed parallel strategy. After the load balancing recursion is completed for all pipelines, the target operator parallel configuration strategy is obtained, and the large model is run or trained according to the strategy.

[0064] Through the above steps, by obtaining the operator information of the cluster-oriented large model under the preset micro-processing batch and the preset parallel mode, starting from the data of the operator layer, the application scope is wider than the network layer method in the existing technology; according to the obtained operator information, model memory information and the memory threshold of the accelerator, the initial operator parallel configuration strategy corresponding to multiple pipelines is determined, and then based on the initial operator parallel configuration strategy, load balancing judgment is performed, and the operator layer in the initial operator parallel configuration strategy is moved recursively, so that the chip computing performance is more efficiently utilized and more accurately when the large model is configured in parallel.

[0065] In some embodiments, obtaining operator information of a cluster-oriented large model under a preset microprocessing batch and a preset parallel mode includes: obtaining an accelerator cluster that runs the large model; the accelerator cluster includes node-based computing operators and communication operators; under the preset microprocessing batch and the preset parallel mode, determining the computing time information and computing operator memory information of the computing operator, and determining the communication time information of the communication operator; the computing operator memory information includes an operator activation value; the operator time information includes computing time information and communication time information.

[0066] Exemplarily, computing operators include: embedding, 1st-layernorm, attention-qkv, flash-attention, attention-dense, attention-dropout, 2nd-layernorm, MLP-GEMM-1, and MLP-gelu; communication operators include: all-gather, reduce-scatter, p2p, allreduce, etc. The types of computing and communication operators are determined based on the actual parallel requirements of large models. The types and quantities of computing and communication operators are not specifically limited here. The computational time information of a computing operator includes the computational time information of the forward propagation and backward propagation of the computing operator. The computational operator memory information includes the computing operator's input, output, weight, operator activation value, and reserved memory. The communication time information of a communication operator includes time information such as the speed between different accelerators and nodes. Exemplarily, the communication time information of a communication operator includes the communication operator's time information under different communication data sizes, such as communication data sizes of 128M and 256M.

[0067] In some embodiments, an initial operator parallel configuration strategy for multiple pipelines in a large model is determined based on operator information, model memory information required for the large model, and a maximum memory value of the accelerator, including: determining a first cluster memory ratio corresponding to a preset microprocessing batch and a preset parallel mode of the pipeline according to the maximum memory value of the accelerator and the operator memory information in the operator information; determining a second cluster memory ratio corresponding to a preset microprocessing batch and a preset parallel mode of the pipeline according to the maximum memory value of the accelerator, the operator memory information in the operator information, and the operator activation value; the first cluster memory ratio is not greater than the second cluster memory ratio; and determining an initial operator parallel configuration strategy based on the first cluster memory ratio and the second cluster memory ratio.

[0068] In this embodiment, after obtaining the operator information of the large model under multiple microprocessing batches and multiple parallel modes, the first cluster memory ratio of the current pipeline is calculated based on the operator information, the memory information required by the large model, and the maximum memory value of the accelerator. The first cluster memory ratio is the minimum value of the pipeline under different microprocessing batches and different parallel modes; at the same time, the second cluster memory ratio of the current pipeline is calculated based on the operator information, the memory information required by the large model, the maximum memory value of the accelerator, and the operator activation value of the calculation operator. The second cluster memory ratio is the minimum value of the pipeline under different microprocessing batches and different parallel modes.

[0069] For example, first, based on the operator information of the large model, the required memory information, and the memory size of the accelerator, the first cluster memory ratio and the second cluster memory ratio of the pipeline under different tensor parallelization modes and different microprocessor batches in the homogeneous cluster are determined. The calculation formulas for the first cluster memory ratio and the second cluster memory ratio in the homogeneous cluster are as follows:

[0070] ;

[0071] ;

[0072] in, Indicates the maximum video memory of the accelerator, act(o) indicates the size of the video memory occupied by the operator activation value of the calculation operator (that is, the size of the video memory occupied by the activation value), mem(o) indicates the operator memory information in the operator information, that is, the operator video memory size, The specific calculation method is as follows:

[0073] ;

[0074] in, Indicates the operator input memory information of the calculation operator, Represents the operator gradient memory information of the calculation operator, Indicates the operator weight memory information of the calculation operator. Further, the operator optimizer state memory information is determined based on the operator weight memory information. The ratio of the operator weight memory information to the operator optimizer state memory information is 1:8. Represents the operator optimizer state memory information of the computing operator.

[0075] After calculating the first cluster memory ratio and the second cluster memory ratio of the pipeline under different tensor parallel modes and different microprocessing batches of the homogeneous cluster, that is, after initializing the homogeneous cluster, it is necessary to initialize the multiple pipelines of the large model to the number of heterogeneous clusters according to the computing power method. Within different clusters, the maximum and minimum values ​​of the pipelines under different microprocessing batches, data parallelism and tensor parallelism of the homogeneous cluster are further obtained according to the homogeneous cluster method, and then the parallel strategy under the heterogeneous cluster is initialized to obtain the initial operator parallel configuration strategy under the heterogeneous cluster.

[0076] In some of the embodiments, the initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines; the first pipeline and the second pipeline here are only used to distinguish between two adjacent pipelines. In the actual process, there are multiple pipelines, and they are not limited to the first pipeline and the second pipeline.

[0077] According to a preset load balancing method, the initial operator parallel configuration strategy is recursively processed to obtain a target operator parallel configuration strategy, including: obtaining a first training time of the first pipeline and a second training time of the second pipeline; judging whether the time difference between the first training time and the second training time is less than a preset time threshold; if the time difference is not less than the preset time threshold, recursively moving the operator of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold, thereby obtaining an intermediate operator parallel configuration strategy; based on the preset load balancing method and memory judgment method, the intermediate operator parallel configuration strategy is processed to determine the target operator parallel configuration strategy.

[0078] In this embodiment, after determining the initial operator parallel configuration strategy, it is necessary to first determine the training time information in the initial operator parallel configuration strategies corresponding to two adjacent pipelines in sequence according to a preset load balancing method. For example, the training time corresponding to the first pipeline and the second pipeline is calculated, and it is determined whether the time difference between the first training time and the second training time is less than a preset time threshold, so as to determine whether load balancing is achieved between the first pipeline and the second pipeline. If the time difference is not less than the preset time threshold, it indicates that load balancing has not been achieved between the first pipeline and the second pipeline. Therefore, it is necessary to move the operators in the first pipeline to the second pipeline for loop iteration until the time difference is less than the preset time threshold, indicating that load balancing has been achieved between the first pipeline and the second pipeline. The load balancing judgment and update of the second pipeline and the third pipeline adjacent to the second pipeline continue according to the above load balancing method until the initial operator parallel configuration strategies corresponding to all pipelines are load balanced, at which time the target operator parallel configuration strategy is obtained.

[0079] For example, the formula for calculating the training time T of each pipeline is as follows:

[0080] ;

[0081] in, Indicates the computation time of each pipeline, there are P in total, represents the communication time between the i-th pipeline and the i+1-th pipeline, B represents the number of microprocessor batches, It represents the total computation time of P pipelines, that is, the training iteration time of the large model.

[0082] The above preset load balancing strategy can be expressed by the following inequality:

[0083] ;

[0084] in, represents the computation time of the i-th pipeline, Represents the computation time of the jth pipeline. The i-th pipeline and the j-th pipeline are adjacent pipelines. When the computation time of two adjacent pipelines is equal, the training time T of each pipeline is also minimized, which minimizes the iteration time of large model training.

[0085] Furthermore, in the actual processing process, it is impossible to strictly ensure that the calculation time of the adjacent i-th pipeline is completely consistent with the calculation time of the j-th pipeline. Therefore, it is necessary to use a pre-set time threshold to improve it, as shown in the following formula:

[0086] ;

[0087] in, represents the computation time of the i-th pipeline, represents the computation time of the jth pipeline, Indicates the preset time threshold.

[0088] After the initial operator parallel configuration strategy is recursively processed through the preset load balancing method, the intermediate operator parallel configuration strategy is obtained; then it is necessary to combine the memory judgment method and the load balancing method to process the intermediate operator parallel configuration strategy to obtain the target operator parallel configuration strategy.

[0089] In some embodiments, the intermediate operator parallel configuration strategy is processed based on a preset load balancing method and a memory determination method, including:

[0090] When the time difference between the first training time and the second training time is less than a preset time threshold, determine whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds the preset memory threshold; if the parallel memory of the first pipeline exceeds the preset memory threshold, update the microprocessing batch; determine whether the parallel memory of the first pipeline after the microprocessing batch is updated exceeds the preset memory threshold; if the parallel memory of the first pipeline after the microprocessing batch is updated does not exceed the preset memory threshold, process the parallel memory of the second pipeline in sequence.

[0091] If the parallel memory of the first pipeline after the microprocessor batch is updated exceeds the preset memory threshold, the tensor parallel increase calculation time of the first pipeline is determined, and the operator activation value recalculation time of the first pipeline is determined; when it is judged that the tensor parallel increase calculation time is not greater than the operator activation value recalculation time, the tensor parallel mode of the first pipeline is updated; based on the preset load balancing strategy, the updated first pipeline is processed.

[0092] Among them, after obtaining the intermediate operator parallel configuration strategy that meets the load balancing, it is necessary to calculate whether the parallel configuration strategy corresponding to each pipeline exceeds the parallel memory of the corresponding accelerator. If it exceeds the memory limit, then determine whether the micro-processing batch can be reduced. If the micro-processing batch cannot be reduced, then enter the update of tensor parallelism and operator activation value recalculation. If it does not exceed the memory limit, then re-load balance the pipeline. If the first pipeline tensor parallelism is selected, increase the larger of the calculation time and the operator activation value recalculation time to update and recalculate, and perform load balancing judgment and memory judgment on the first pipeline after the update and recalculation, until the first pipeline meets the above-mentioned load balancing and memory judgment methods, and then perform load balancing and memory judgment on the second pipeline adjacent to the first pipeline.

[0093] Based on the preset load balancing method and memory judgment method, all pipelines in the intermediate operator parallel configuration strategy are processed in sequence to obtain the target operator parallel configuration strategy. The accelerators in the hybrid cluster are run according to the target operator parallel configuration strategy to train the large model.

[0094] The present embodiment is described and illustrated below through specific examples.

[0095] Figure 3 This is a flowchart of the large model parallel method for hybrid clusters provided by this specific embodiment. Figure 3 ,The large model parallel method includes the following steps.

[0096] Step S310: obtaining large model and cluster operator information.

[0097] Specifically, based on clusters and large models, the forward and backward computation times, inputs, weights, activations, gradients, optimizer state usage, and framework reserved memory for each operator in different microprocessor batches are pre-calculated for different parallel strategies, such as data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP). Furthermore, the speed and computation time of different communication operators (all-gather, reduce-scatter, p2p, allreduce, etc.) across different accelerators and nodes are calculated. Compared to current model parallelism approaches, the model test parameters of the solution provided in this embodiment can reach 10 billion. Furthermore, the parallelization strategy can be determined at the operator level for hybrid clusters, taking into account factors such as operator activation recalculation, load balancing, and memory limitations. See Table 1, which shows the operator information for a GPT3 model with 7 billion data points, assuming a microprocessor batch size of 1 and both data and tensor parallelism of 1.

[0098] Table 1 Computation time and memory information of calculation operators

[0099]

[0100] Step S320 , initializing the maximum and minimum values ​​of the pipeline under different tensor parallelism and micro-processing batches.

[0101] Specifically, based on the memory information required by the model and the accelerator's memory size obtained in step S310, the pipeline's maximum and minimum values ​​are initialized for different tensor parallelism and microprocessing batches. The method for determining the first cluster memory ratio (minimum value) and the second cluster memory ratio (maximum value) in the previous embodiment is referenced and will not be explained in detail here.

[0102] Step S330: recursively obtain the optimal solution based on the memory-aware distributed parallel strategy.

[0103] Specifically, Figure 4 This is a flow chart of the distributed parallel strategy recursive method provided by this specific embodiment. Figure 4 , this step includes the following details.

[0104] Step Q1: For each pipeline's initialized parallel configuration, determine whether it is load balanced. If so, proceed to the next step. If not, move the operator of the current pipeline to the next pipeline until the two adjacent pipelines are load balanced.

[0105] For example, the formula for calculating the training time T of each pipeline is as follows:

[0106] ;

[0107] in, Indicates the computation time of each pipeline, there are P in total, represents the communication time between the i-th pipeline and the i+1-th pipeline, B represents the number of microprocessor batches, It represents the total computation time of P pipelines, that is, the training iteration time of the large model.

[0108] In actual processing, it is impossible to strictly ensure that the computation time of the adjacent i-th pipeline is completely consistent with the computation time of the j-th pipeline. Therefore, it is necessary to use a pre-set time threshold to improve it. The above preset load balancing strategy can be expressed as follows:

[0109] ;

[0110] in, represents the computation time of the i-th pipeline, represents the computation time of the jth pipeline, Indicates the preset time threshold.

[0111] In step Q2, we calculate whether each pipeline exceeds the video memory of the corresponding device. If it exceeds the memory limit (OOM), we determine whether the micro-processing batch size can be reduced. If so, we return to the initial stage to determine whether the load is balanced. If not, we proceed to the next step of tensor parallelization and activation value recalculation and update.

[0112] Step Q3, determine whether the updated micro-processing batch exceeds the memory limit. If not, enter the next adjacent pipeline. If so, enter the tensor parallelization and operator activation value recalculation update.

[0113] Step Q4, pre-calculate the pipeline time for increasing tensor parallelism and recalculating activation values. If the time for increasing tensor parallelism is greater than the pipeline time for recalculating activation values, update the tensor parallelism; otherwise, update the operator activation value and recalculate. After the tensor parallelism and activation value recalculation are updated, enter the load balancing judgment to determine whether the current pipeline is load balanced and whether it exceeds the memory limit. If both are met, enter the calculation of the next pipeline and determine whether the next pipeline is the last pipeline. If so, then from the resource pool including multiple operator parallel configuration strategies, according to the above-mentioned pipeline calculation time formula, obtain the operator parallel configuration strategy with the least time, that is, the optimal configuration, and train the large model.

[0114] This solution addresses the problem of low training efficiency caused by uneven computing power, memory, and communication between different accelerator units in heterogeneous clusters. It implements heterogeneous resource-aware operator-level automatic parallelization to improve the training efficiency of large models on heterogeneous clusters, resulting in higher MFU (Model Floating-Point Operation Utilization) performance, thereby increasing the efficient utilization of chip computing power during large model training. Furthermore, to address the current problem of long search times caused by the expansion of the automatic parallel search space, a heuristic memory-aware search is proposed. The search process significantly improves search speed by caching profiled operator information. Furthermore, the integration of optimization techniques such as flash attention and a distributed optimizer further enhances the efficiency of training large models.

[0115] This embodiment also provides a cluster-oriented large-scale model parallel device, which is used to implement the above-mentioned embodiments and preferred implementations. Details that have already been described will not be repeated. The terms "module," "unit," "subunit," etc. used below refer to a combination of software and / or hardware that can implement the predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0116] Figure 5 This is a block diagram of a large-scale parallel cluster-oriented device provided by an embodiment of the present application. Figure 5 As shown, the device includes: an information acquisition module 10, an initialization module 20 and a parallel module 30.

[0117] The information acquisition module 10 is used to obtain operator information of the cluster-oriented large model under a preset microprocessing batch and a preset parallel mode. The operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators.

[0118] The initialization module 20 is used to determine the initial operator parallel configuration strategy of multiple pipelines in the large model based on operator information, model memory information required by the large model, and the memory threshold of the accelerator.

[0119] The parallel module 30 is used to recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain the target operator parallel configuration strategy, and run the cluster-oriented large model based on the target operator parallel configuration strategy.

[0120] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0121] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0122] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0123] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0124] S1, obtaining operator information of a cluster-oriented large model in a preset micro-processing batch and a preset parallel mode, where the operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators.

[0125] S2, based on the operator information, the model memory information required by the large model, and the memory threshold of the accelerator, determines the initial operator parallel configuration strategy for multiple pipelines in the large model.

[0126] S3, according to the preset load balancing method, recursively processes the initial operator parallel configuration strategy to obtain the target operator parallel configuration strategy, and runs the cluster-oriented large model based on the target operator parallel configuration strategy.

[0127] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0128] In addition, in conjunction with the cluster-oriented large model parallelization method provided in the above embodiments, this embodiment may also provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the cluster-oriented large model parallelization methods in the above embodiments.

[0129] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0130] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0131] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0132] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A cluster-oriented large model parallel method, characterized in that: The method comprises: Obtaining operator information of a cluster-oriented large model in a preset microprocessing batch and a preset parallel mode, wherein the operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators; determining, based on the maximum memory value of the accelerator and the operator memory information in the operator information, a first cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode; determining, based on the maximum memory value of the accelerator, the operator memory information in the operator information, and the operator activation value in the computing operator memory information, a second cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode; the first cluster memory ratio is not greater than the second cluster memory ratio; and determining, based on the first cluster memory ratio and the second cluster memory ratio, an initial operator parallel configuration strategy for multiple pipelines in the large model; According to a preset load balancing method, the initial operator parallel configuration strategy is recursively processed to obtain a target operator parallel configuration strategy, and the cluster-oriented large model is run based on the target operator parallel configuration strategy; The initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines; the initial operator parallel configuration strategy is recursively processed according to a preset load balancing method to obtain a target operator parallel configuration strategy, including: obtaining a first training time of the first pipeline and a second training time of the second pipeline; determining whether a time difference between the first training time and the second training time is less than a preset time threshold; if the time difference is not less than the preset time threshold, recursively moving the operator of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than the preset time threshold, thereby obtaining an intermediate operator parallel configuration strategy; When the time difference between the first training time and the second training time is less than a preset time threshold, determine whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds the preset memory threshold; if the parallel memory of the first pipeline exceeds the preset memory threshold, update the microprocessing batch; determine whether the parallel memory of the first pipeline after the microprocessing batch is updated exceeds the preset memory threshold; if the parallel memory of the first pipeline after the microprocessing batch is updated does not exceed the preset memory threshold, process the parallel memory of the second pipeline in sequence.

2. The cluster-oriented large model parallel method according to claim 1, characterized in that: The obtaining of operator information of a cluster-oriented large model in a preset micro-processing batch and a preset parallel mode includes: Acquire an accelerator cluster for running the large model; the accelerator cluster includes node-based computing operators and communication operators; Under a preset microprocessor batch and a preset parallel mode, the calculation time information and the calculation operator memory information of the calculation operator are determined, and the communication time information of the communication operator is determined; the calculation operator memory information includes the operator activation value; the operator time information includes the calculation time information and the communication time information.

3. The cluster-oriented large model parallel method according to claim 1, characterized in that: The method further comprises: If the parallel memory of the first pipeline after the updated microprocessor batch exceeds a preset memory threshold, determining a tensor parallel increase calculation time of the first pipeline and determining an operator activation value recalculation time of the first pipeline; When it is determined that the tensor parallel increase calculation time is not greater than the operator activation value recalculation time, updating the tensor parallel mode of the first pipeline; Based on the preset load balancing strategy, the updated first pipeline is processed.

4. The cluster-oriented large model parallel method according to claim 1, characterized in that: The method further comprises: Based on a preset load balancing method and a memory judgment method, all pipelines in the intermediate operator parallel configuration strategy are processed in sequence to obtain the target operator parallel configuration strategy.

5. A large model parallel device for clusters, characterized by: The device includes: an information acquisition module, an initialization module and a parallel module; The information acquisition module is used to obtain operator information of a cluster-oriented large model in a preset microprocessing batch and a preset parallel mode, wherein the operator information includes operator time information and operator memory information of the operators in the cluster-oriented large model; the cluster includes one or more types of accelerators; The initialization module is configured to determine a first cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode based on the maximum memory value of the accelerator and the operator memory information in the operator information; determine a second cluster memory ratio corresponding to a preset microprocessing batch and a pipeline in a preset parallel mode based on the maximum memory value of the accelerator, the operator memory information in the operator information, and the operator activation value in the calculation operator memory information; the first cluster memory ratio is not greater than the second cluster memory ratio; and determine an initial operator parallel configuration strategy for multiple pipelines in the large model based on the first cluster memory ratio and the second cluster memory ratio; The parallel module is used to recursively process the initial operator parallel configuration strategy according to a preset load balancing method to obtain a target operator parallel configuration strategy, and run the cluster-oriented large model based on the target operator parallel configuration strategy; the initial operator parallel configuration strategy includes a first pipeline and a second pipeline; the first pipeline and the second pipeline are adjacent pipelines; the initial operator parallel configuration strategy is recursively processed according to the preset load balancing method to obtain a target operator parallel configuration strategy, including: obtaining a first training time of the first pipeline and a second training time of the second pipeline; judging whether the time difference between the first training time and the second training time is less than a preset time threshold; if the time difference is not less than the preset time Threshold, recursively move the operators of the first pipeline to the second pipeline until the time difference between the first training time and the second training time is less than a preset time threshold, and obtain an intermediate operator parallel configuration strategy; when the time difference between the first training time and the second training time is less than the preset time threshold, determine whether the parallel memory of the first pipeline of the intermediate operator parallel configuration strategy exceeds a preset memory threshold; if the parallel memory of the first pipeline exceeds the preset memory threshold, update the microprocessor batch; determine whether the parallel memory of the first pipeline after the microprocessor batch is updated exceeds the preset memory threshold; if the parallel memory of the first pipeline after the microprocessor batch is updated does not exceed the preset memory threshold, process the parallel memory of the second pipeline in sequence.

6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the cluster-oriented large model parallel method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cluster-oriented large model parallel method according to any one of claims 1 to 4 are implemented.

Images