A method, device, equipment and medium for determining a distributed parallel solution

Abstract

The present invention discloses a method, device, equipment and medium for determining a distributed parallel scheme, and relates to the technical field of resource allocation. The method includes: obtaining model information of a target large model to be modeled for a distributed system and cluster information of a distributed cluster for deploying the model, determining a search set corresponding to each distributed strategy, obtaining multiple candidate distributed parallel schemes and a single-device calculation graph of the target large model of each scheme, calculating the single-device calculation time of each scheme, obtaining the end-to-end total time of each scheme, and determining the target distributed parallel scheme accordingly. According to the model information of the target large model and the cluster information of the distributed cluster, multiple candidate distributed parallel schemes are determined, and the target distributed parallel scheme with the best performance is determined according to the total time of each scheme through the distributed calculation graph and parallel calculation technology, avoiding iterative testing of each distributed parallel scheme one by one, and improving the efficiency of determining the distributed parallel scheme.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for determining a distributed parallel solution. Background Art

[0002] As the scale of deep learning models increases, traditional single-machine training methods can no longer meet the needs of ultra-large models and tasks with very long contexts. Distributed technology improves computing efficiency and model training speed by processing in parallel on multiple nodes, becoming a key strategy to solve resource bottlenecks and accelerate model iteration and deployment.

[0003] In the prior art, distributed parallel algorithms include tensor parallelism, pipeline parallelism, sequence parallelism, expert parallelism, context parallelism, and data parallelism. In order to obtain a high-performance distributed parallel solution for a certain model, algorithm personnel need to conduct a large number of experiments and tests on all feasible distributed combination solutions in a cluster environment of 1,000 or 10,000 cards to find the optimal parallel strategy, which consumes a lot of manpower and equipment resources and reduces the efficiency of determining the distributed parallel solution. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for determining a distributed parallel solution, so as to efficiently and accurately determine a high-performance distributed parallel solution that meets the requirements among a large number of optional distributed combination solutions.

[0005] According to one aspect of an embodiment of the present invention, a method for determining a distributed parallel solution is provided, comprising:

[0006] Obtain model information of the target large model for distributed system modeling and cluster information of the distributed cluster used to deploy the target large model; determine the search set corresponding to each distributed strategy based on the computing tasks and cluster information matching the target large model, and obtain multiple alternative distributed parallel schemes based on each search set; wherein each alternative distributed parallel scheme includes the number of parallels under at least one distributed strategy; generate a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device calculation time of each single-device calculation graph, wherein the single-device calculation graph includes communication operators; calculate the end-to-end total time corresponding to each alternative distributed parallel scheme based on the single-device calculation time of each single-device calculation graph; determine the target distributed parallel scheme when deploying the target large model in the distributed cluster in each alternative distributed parallel scheme based on the calculated single-device calculation time and end-to-end total time.

[0007] According to another aspect of an embodiment of the present invention, a device for determining a distributed parallel solution is provided, including:

[0008] An information acquisition module is used to obtain model information of a target large model to be modeled in a distributed system, and cluster information of a distributed cluster for deploying the target large model; an alternative distributed parallel scheme determination module is used to determine a search set corresponding to each distributed strategy based on computing tasks and cluster information matching the target large model, and obtain multiple alternative distributed parallel schemes based on each search set; wherein each alternative distributed parallel scheme includes the number of parallels under at least one distributed strategy; a single-device computing time calculation module is used to generate a single-device computing graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device computing time of each single-device computing graph, wherein the single-device computing graph includes a communication operator; a total time calculation module is used to calculate the end-to-end total time corresponding to each alternative distributed parallel scheme based on the single-device computing time of each single-device computing graph; a target distributed parallel scheme determination module is used to determine the target distributed parallel scheme when the target large model is deployed in a distributed cluster in each alternative distributed parallel scheme based on the calculated single-device computing time and the end-to-end total time.

[0009] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0010] at least one processor; and

[0011] a memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for determining a distributed parallel solution described in any embodiment of the present invention.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining a distributed parallel scheme described in any embodiment of the present invention when executed.

[0014] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the steps of the method described in any embodiment of the present invention are implemented.

[0015] The technical solution of the embodiment of the present invention first obtains the model information of the target large model to be modeled in the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, and determines the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and then obtains multiple alternative distributed parallel schemes according to each search set, generates a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculates the single-device calculation time of each single-device calculation graph, and then calculates the end-to-end total time corresponding to each alternative distributed parallel scheme according to the single-device calculation time of each single-device calculation graph, and determines the target distributed parallel scheme when deploying the target large model in the distributed cluster in each alternative distributed parallel scheme according to the calculated single-device calculation time and the end-to-end total time. Through the model information of the target large model and the cluster information of the distributed cluster, multiple alternative distributed parallel schemes are determined, and the alternative distributed parallel schemes are evaluated through distributed computing graphs and parallel computing technology. The target distributed parallel scheme with the best performance is determined according to the total time consumption of each scheme, avoiding iterative testing of each distributed parallel scheme one by one. Automatic screening and decision-making of alternative schemes are achieved based on the cluster device information and the model information of the target large model, which reduces labor costs and equipment resource consumption and improves the efficiency of determining distributed parallel schemes.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a flow chart of a method for determining a distributed parallel solution provided according to Embodiment 1 of the present invention;

[0019] Figure 2 is a flowchart of another method for determining a distributed parallel solution provided according to Embodiment 2 of the present invention;

[0020] Figure 3 It is a specific application scenario diagram applicable to the embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a model training storage content applicable to an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of obtaining a reasonable distributed parallel solution applicable to an embodiment of the present invention;

[0023] Figure 6 It is a schematic diagram of constructing a single-card computing graph applicable to an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of the structure of a distributed parallel solution determination device provided according to Embodiment 3 of the present invention;

[0025] Figure 8 It is a structural schematic diagram of an electronic device for implementing the method for determining a distributed parallel solution according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1 This is a flow chart of a method for determining a distributed parallel solution provided in the first embodiment of the present invention. This embodiment can be applied to the case where a high-performance distributed parallel solution that matches the performance of a set hardware cluster device is efficiently determined when a set large model is deployed in the set hardware cluster device. The method can be executed by a distributed parallel solution determination device, which can be implemented in the form of hardware and / or software and can generally be configured in an electronic device. Figure 1 As shown, the method includes:

[0030] S110: Acquire model information of a target large model for distributed system modeling and cluster information of a distributed cluster for deploying the target large model.

[0031] In the embodiment of the present invention, the target large model can be specifically understood as: a model with a large number of parameters or a very long context for distributed system modeling. Among them, due to the large number of parameters, the structure of the model is often very complex. The very long context means that the text data that the model needs to process is very long, requiring the model to be able to process and parse large amounts of text data, which not only involves the storage and call of a large amount of historical information, but also relates to the accurate grasp of the overall coherence and deep meaning of the text. To this end, the model must rely on sufficient computing resources to support its efficient processing of large amounts of data during training and reasoning.

[0032] The model information of the target large model can be specifically understood as: detailed data and parameters of a series of target large models related to distributed system modeling. The model information can specifically include model structure (such as the number of model layers, the type of each layer (such as convolutional layer, recurrent layer or fully connected layer, etc.) or activation function, etc.), model size (referring to the number of parameters), context length, computational complexity (such as the number of floating-point operations required for the model to perform a forward or backward propagation), model input data format and dimension or model output format and dimension.

[0033] The target large model can be applied in the fields of image classification, target detection, face recognition, text classification, machine translation or speech recognition, etc. The target large model can be a set pre-trained model.

[0034] In the embodiment of the present invention, the distributed cluster can be specifically understood as: a group of computers (nodes) connected through a network, which work together to complete computing tasks together to improve processing power, storage capacity and reliability, while reducing costs and improving computing efficiency. These computers can be physical machines or virtual machines, which share resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, storage and network bandwidth, and can run distributed applications. Among them, GPU can also be called GPU card or card, which is a hardware accelerator for performing parallel computing, and is particularly suitable for large-scale matrix operations in deep learning. When the model is very large or the data to be processed is very large, such a large-scale cluster (such as a cluster of thousands of cards or tens of thousands of cards) is required to provide the required computing power. The cluster information of the distributed cluster can be specifically understood as: detailed data describing all nodes in the distributed system and their mutual relationships. The cluster information can specifically be: cluster hardware equipment specifications, interconnection information between devices, storage capacity, geographical location of nodes, operating system and version, etc. Cluster information is the basis for distributed system modeling and the premise that the distributed system can meet business needs and provide high-quality services.

[0035] In an optional implementation of this embodiment, the distributed cluster may include multiple cabinets, each cabinet includes multiple servers, and each server includes multiple computing cards (also referred to as computing devices). The above servers and computing cards are connected to each other using a set topological connection method (for example, a ring or a network) to establish a communication connection.

[0036] Among them, after the target large model is deployed in the distributed cluster, the model training task or model reasoning task can be executed accordingly. The above-mentioned model training task or model reasoning task can be collectively referred to as computing tasks.

[0037] S120. Determine a search set corresponding to each distributed strategy based on computing tasks and cluster information that match the target large model, and obtain multiple candidate distributed parallel solutions based on each search set.

[0038] Each candidate distributed parallel scheme includes the number of parallel operations under at least one distributed strategy.

[0039] In the embodiment of the present invention, the distributed strategy can be specifically understood as: a method of executing computing tasks simultaneously on multiple processors or multiple computer systems to improve processing speed and efficiency. The distributed strategy can specifically include: tensor parallel strategy, pipeline parallel strategy, sequence parallel strategy, expert parallel strategy, context parallel strategy, data parallel strategy and dynamic adjustment strategy.

[0040] Tensor parallelism refers to a parallel method that splits tensors (multidimensional data structures) among multiple processors. For example, in deep learning, the weights of a large neural network model can be distributed to different GPUs or processors (also called computing cards or computing devices). Each processor only stores and computes a portion of the tensor, and then updates it synchronously through communication operations. Pipeline parallelism refers to breaking down a computing task into a series of stages, each of which can be executed in parallel on different processors, similar to an assembly line in industrial production, where each stage completes its task and passes the result to the next stage. For example, in deep learning, different layers of a model can be processed in pipeline parallelism. Sequential parallelism refers to executing multiple parallel tasks in a specific order, where each task is completed before the next task can start. For example, different computing stages are executed sequentially on different processors. Expert parallelism refers to assigning different models or different parts of a model to different processors or machines. Each processor or machine is responsible for processing a part of the model and can be trained or inferred independently. Context parallelism refers to running the same algorithm or task in parallel in different execution contexts or environments, that is, running the same code on different processors, but each processor has its own data set or parameters. Context parallelism allows other processors to perform calculations while waiting for a task to complete, which can improve resource utilization. Data parallelism refers to splitting large data sets into small blocks and then processing these data blocks simultaneously on multiple processors. For example, in deep learning, models are trained in parallel on multiple GPUs, with each GPU processing different data samples but sharing the same model parameters. During training, each GPU synchronizes their model parameters to ensure that the model on all processors remains consistent. Dynamic adjustment strategy refers to adjusting computing resources and task allocation based on real-time feedback through adaptive algorithms.

[0041] In an embodiment of the present invention, the search set can be specifically understood as: a series of optional distributed parallel strategies, such as a tensor parallel search set, a pipeline parallel search set, a sequence parallel search set, an expert parallel search set, a context parallel search set or a data parallel search set. The search set contains all parallel strategies under the distributed strategy, and each parallel strategy has its corresponding allowed number of parallels. The search sets are orthogonalized to obtain multiple parallel combination schemes, namely, alternative distributed parallel schemes. Among them, orthogonal means that distributed parallel strategies are independently selected from each search set, and they are combined to form a larger distributed parallel search space. The distributed parallel search space contains all possible parallel strategy combinations, and these parallel strategy combinations constitute multiple alternative distributed parallel schemes, from which the optimal or approximately optimal parallel execution scheme can be found. Since the alternative distributed parallel schemes are obtained by orthogonalization of the search sets, each alternative distributed parallel scheme contains at least one parallel number set under a distributed strategy.

[0042] It is understandable that the above-mentioned alternative distributed parallel solutions may fully occupy all computing devices in the distributed cluster, or may only occupy part (for example, 50% or 80%) of the computing devices in the distributed cluster.

[0043] S130, generating a single-device calculation graph of the target large model under each alternative distributed parallel solution, and calculating the single-device calculation time of each single-device calculation graph.

[0044] Among them, the single-device computing graph contains communication operators.

[0045] In an embodiment of the present invention, a single-device computational graph can be specifically understood as: a computational graph constructed and executed on a single computing device. Among them, a computational graph is a graph representation method that describes a network structure and a computing process, and is composed of nodes (representing operations or calculations) and edges (representing data flows or tensors). Among them, a single-device computational graph contains communication operators. A communication operator refers to an operator that is specifically responsible for exchanging data between different computing nodes, ensuring that data processed in parallel on multiple computing devices can be correctly synchronized and aggregated. Single-device computing time refers to the time required to complete a computing task on a single computing node (computing device), which can be used to evaluate system performance, optimize algorithms, and arrange task scheduling.

[0046] S140. Calculate the end-to-end total time corresponding to each candidate distributed parallel solution according to the single-device computing time of each single-device computing graph.

[0047] In the embodiment of the present invention, the end-to-end total time can be specifically understood as: the total time from the start to the end of the computing process, including the time of all steps and stages. In a distributed system, computing tasks are usually decomposed and executed in parallel on multiple computing nodes, and the computing graph of each node may contain different tasks. By adding up the single-device computing time of all single-device computing graphs, and adding the time for data transmission, aggregation, and post-processing, the end-to-end total time of the entire distributed computing task can be obtained.

[0048] S150. According to the calculated calculation time of each single device and the end-to-end total time, determine a target distributed parallel scheme when deploying the target large model in a distributed cluster from among various alternative distributed parallel schemes.

[0049] Specifically, through the model information of the target large model to be modeled in the distributed system (such as the structure, size and context length of the model) and the cluster information of the distributed cluster used to deploy the target large model (such as the cluster hardware device specifications and the interconnection information between devices), a distributed strategy search set (such as a tensor parallel strategy search set, a pipeline parallel strategy search set, a sequence parallel strategy search set, an expert parallel strategy search set, a context parallel strategy search set and a data parallel strategy search set) that meets the hardware performance requirements and is suitable for the target large model is determined, orthogonal operations are performed on each search set to obtain multiple alternative distributed parallel schemes, and a single-device calculation graph corresponding to each alternative distributed parallel scheme is constructed, the corresponding single-device calculation time is calculated and summed to obtain the end-to-end total time corresponding to each alternative distributed parallel scheme, and according to the calculated single-device calculation time and end-to-end total time, a distributed parallel scheme that meets the high performance conditions is screened out from each alternative distributed parallel scheme as the target distributed parallel scheme when the target large model is deployed in the distributed cluster, thereby realizing the determination of the distributed parallel scheme.

[0050] Among them, the above-mentioned high-performance conditions can be preset according to actual needs. For example, the total end-to-end time can be minimized, the performance of a single computing device can be optimal, or the number of total computing devices in a distributed cluster can be minimized, etc. This embodiment does not limit this.

[0051] The technical solution of the embodiment of the present invention first obtains the model information of the target large model to be modeled in the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, and determines the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and then obtains multiple alternative distributed parallel schemes according to each search set, generates a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculates the single-device calculation time of each single-device calculation graph, and then calculates the end-to-end total time corresponding to each alternative distributed parallel scheme according to the single-device calculation time of each single-device calculation graph, and determines the target distributed parallel scheme when deploying the target large model in the distributed cluster in each alternative distributed parallel scheme according to the calculated single-device calculation time and the end-to-end total time. According to the model information of the target large model and the cluster information of the distributed cluster, multiple alternative distributed parallel schemes are determined, and the alternative distributed parallel schemes are screened and evaluated through distributed computing graphs and parallel computing technology. The target distributed parallel scheme with the best performance is determined according to the total time consumption of each scheme, and automatic screening and decision-making of the alternative schemes are realized based on the cluster device information and the model information of the target large model, avoiding time-consuming iterative testing of each distributed parallel scheme one by one, reducing labor costs and equipment resource consumption, and improving the efficiency of determining distributed parallel schemes.

[0052] Furthermore, in an embodiment of the present invention, a method for determining a distributed parallel solution may further include:

[0053] According to the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster, the weight-related storage requirements and activation-related storage requirements matching the target large model are calculated;

[0054] Accordingly, after obtaining multiple candidate distributed parallel solutions according to each search set, it also includes:

[0055] According to the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated respectively; among each alternative distributed parallel scheme, the invalid distributed parallel scheme whose single-device storage requirement is greater than the single-device standard storage capacity in the distributed cluster is filtered out.

[0056] In an embodiment of the present invention, the weight-related storage requirements can be specifically understood as: storage resources required to store and manage model parameters. Among them, model parameters may include: weights and biases, model parameters are learned during the training process, and are used in the reasoning and training of the model. Among them, for the model training task, the weight size, gradient size, and optimizer size are calculated and added to obtain the weight-related storage requirements. The weight size can be determined by multiplying the number of parameters by their data types to determine the memory space they occupy. For example, a parameter of type float32 occupies 4 bytes (32 bits), while a parameter of type int8 occupies only 1 byte. The gradient is the derivative of the loss function with respect to the weight and is used to update the weight during the training process. The gradient size is usually the same as the weight size because the gradient is also a floating point number and the number is the same as the weight. The optimizer is an algorithm used to update the model weights, and the weights are adjusted by calculating the gradient to improve the performance of the model. The optimizer not only stores the updated value of the weight, but may also store additional state information for each weight, such as additional states such as momentum and the square of the gradient. Among them, the optimizer size is related to the type of optimizer used and the number of additional states stored. For simple optimizers, such as SGD (Stochastic Gradient Descent), its size may be the same as the weight size. For complex optimizers, such as Adam (Adaptive Moment Estimation), its size may be about twice the weight size because of the need to store additional states. For model reasoning tasks (i.e., model inference or model execution), the model does not need to store gradients and optimizers, that is, the weight-related storage requirements are equal to the weight size.

[0057] In the embodiment of the present invention, the activation-related storage requirement can be specifically understood as: in order to perform effective gradient calculation, weight update or model recovery, it is necessary to store the storage resources required for the data generated by the intermediate layer (activation layer) of some models in the memory. The activation-related storage requirement can be divided into activation storage size and temporary storage size. Among them, in the reverse propagation process of the model, in order to calculate the gradient of each layer, the activation value during the forward propagation is required. Therefore, in order to perform reverse propagation, the activation value of each layer during the forward propagation must be saved. The activation storage size refers to the sum of the activation value sizes during the forward propagation that need to be saved for reverse calculation. The temporary storage size refers to the maximum possible size of all temporary activation values ​​that need to be saved in order to maintain the continuity of the calculation during the execution of the calculation graph. For example, at a certain point in the calculation graph, when multiple operators require a large number of intermediate activation values, the maximum value of memory usage may be reached. Among them, the intermediate activation value refers to each operator in the calculation graph that may generate temporary activation values ​​during execution, which may no longer be needed after the current operator completes the calculation, but may be used as input in subsequent operators in the graph. For model training tasks, the activation storage size is equal to the sum of the forward activation sizes that need to be saved for reverse calculations. The temporary storage size is the maximum activation size of all operators in the computation graph. For model inference tasks, the activation values ​​in the inference process are only used for the current forward calculation and no activation storage is required; during the model inference process, some operations may require intermediate results, but these results are no longer needed in subsequent calculations, so temporary variables can be stored in temporary storage and released after use. In addition, for model inference tasks, the KV cache (key value-cache) storage size can also be included. In some model architectures, such as the Transformer model (a self-attention mechanism model), KV cache can be used to improve efficiency during inference. The KV cache stores the key (Key) and value (Value) information in the self-attention mechanism, which is reused in each step of the generated sequence instead of being recalculated each time. By caching this information, the amount of calculation can be reduced and the inference speed can be accelerated, but it will also take up a certain amount of storage space.

[0058] In an optional implementation of this embodiment, in the model training stage, weight-related storage may specifically include: weight storage, gradient storage and optimizer storage, and activation-related storage may specifically include: activation storage and temporary storage; in the model inference stage, weight-related storage may only include weight storage, and activation-related storage may specifically include: activation storage (the amount of activation storage data is much smaller than that in the model training stage), temporary storage and KV cache storage.

[0059] In the embodiment of the present invention, the single-device storage requirement can be specifically understood as: the storage capacity required by a single computing device when executing a task, including but not limited to weight-related storage requirements and activation-related storage requirements.

[0060] Specifically, when designing a distributed solution, it is crucial to comprehensively consider the deployment scenario information on the distributed cluster and the model information of the target large model to calculate the single-device storage requirements and screen the alternative distributed solutions to achieve performance optimization, reasonable resource allocation, cost-effectiveness maximization, system reliability and fault tolerance. By calculating the weight-related storage requirements and activation-related storage requirements in the two task scenarios respectively, and according to each search set, a variety of alternative distributed parallel solutions are obtained, and then the single-device standard storage capacity in the distributed cluster and the single-device storage requirements corresponding to each alternative distributed parallel solution are evaluated respectively. By screening out invalid alternative distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster, not only the number of single-device calculation graphs corresponding to each alternative solution and the total time-consuming calculation amount are reduced, but also the time-consuming of the overall calculation task is reduced, and the efficiency of determining the distributed parallel solution is improved.

[0061] Optionally, in an embodiment of the present invention, evaluating the single-device storage requirements corresponding to each candidate distributed parallel solution respectively according to the weight-related storage requirements and the activation-related storage requirements may include:

[0062] Obtain the current tensor parallel number, current pipeline parallel number, and current context parallel number in the target alternative distributed parallel scheme currently being processed; calculate the single-device weight-related storage requirement based on the weight-related storage requirement, the current tensor parallel number, and the current pipeline parallel number; calculate the single-device activation-related storage requirement based on the activation-related storage requirement, the current tensor parallel number, and the current context parallel number; calculate the single-device storage requirement corresponding to the target alternative distributed parallel scheme based on the single-device weight-related storage requirement and the single-device activation-related storage requirement.

[0063] Specifically, in the process of respectively evaluating the single-device standard storage capacity in the distributed cluster and the single-device storage requirements corresponding to each alternative distributed parallel scheme, the calculation method of the single-device storage requirement corresponding to the target alternative distributed parallel scheme (referring to the alternative distributed parallel scheme to be evaluated) can be: first, obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target alternative distributed parallel scheme currently being processed, and then calculate the single-device weight-related storage requirement based on the weight-related storage requirement, the current tensor parallel number, and the current pipeline parallel number, that is, the single-device weight-related storage requirement is calculated in the storage module required by the model. The weight-related storage requirement is divided by the product of the current tensor parallelism number and the current pipeline parallelism number, that is, the weight-related storage requirement of a single device = weight-related storage requirement / (current tensor parallelism number × current pipeline parallelism number); according to the activation-related storage requirement, the current tensor parallelism number and the current context parallelism number, the activation-related storage requirement of a single device is calculated, that is, the activation-related storage requirement of a single device is the activation-related storage requirement calculated in the storage module required by the model divided by the product of the current tensor parallelism number and the current context parallelism number, that is, the activation-related storage requirement of a single device = activation-related storage requirement / (current tensor parallelism number × current context parallelism number).

[0064] Among them, the tensor parallel number, pipeline parallel number, and context parallel number can all be obtained from the computing tasks and cluster information that match the target large model. In the process of evaluating the alternative distributed parallel solutions, the current tensor parallel number, the current pipeline parallel number, and the current context parallel number can be directly obtained. Finally, based on the single-device weight-related storage requirements and the single-device activation-related storage requirements, the single-device storage requirements corresponding to the target alternative distributed parallel solution are calculated and used for comparison with the single-device standard storage capacity in the distributed cluster, and invalid alternative distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity are screened out.

[0065] Optionally, a method for determining a distributed parallel solution, the distributed strategies may include: pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism, and expert parallelism.

[0066] Specifically, pipeline parallelism improves resource utilization through pipeline processing; tensor parallelism parallelizes computationally intensive layers by splitting tensors; sequence parallelism is suitable for application scenarios that process long sequence data, such as time series or long texts; context parallelism allows different tasks or model instances to run independently on different devices; data parallelism is suitable for large data set application scenarios, and accelerates training by replicating models on multiple devices; expert parallelism distributes different experts to different devices in hybrid expert models to improve the sparsity and efficiency of the model. By comprehensively utilizing multiple parallel technologies such as pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism, and expert parallelism, each technology can play its unique advantages in specific application scenarios, thereby improving the determination rate of distributed parallel solutions, optimizing resource utilization, enhancing the scalability and flexibility of distributed parallel solutions, and improving the reliability of distributed parallel solutions.

[0067] In an optional implementation of this embodiment, the alternative distributed parallel scheme may specifically include: the number of tensor parallelism, the number of sequence parallelism, the number of context parallelism, the number of data parallelism, and the number of expert parallelism.

[0068] Optionally, according to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, including at least one of the following:

[0069] According to the number of high-bandwidth interconnected devices (which can also be understood as computing devices that meet high-bandwidth interconnection requirements) supported by the distributed cluster in the cluster information, determine the tensor parallel search set corresponding to the tensor parallelism; according to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, determine the pipeline parallel search set corresponding to the pipeline parallelism; extract the hardware information in the cluster information, and determine the maximum number of context parallelism based on the hardware information; according to the computing tasks matching the target large model, determine the context length information; according to the maximum number of context parallelism and the context length information, determine the context parallel search set corresponding to the context parallelism; according to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallelism; according to the total number of cluster devices in the cluster information, the determined tensor parallel search set, pipeline parallel search set and context parallel search set, calculate the maximum number of data parallelism, and according to the maximum number of data parallelism, determine the data parallel search set corresponding to the data parallelism.

[0070] Specifically, since the search set is orthogonally obtained from various search sets (including tensor parallel search set, pipeline parallel search set, sequence parallel search set, expert parallel search set, context parallel search set and data parallel search set), the search set includes at least one of the above sets.

[0071] Among them, the tensor parallel search set can be defined as a set of powers of 2 in the range of 1 to the number of high-bandwidth interconnection cards (interface cards that provide high-speed data transmission between high-performance computing devices) according to the number of high-bandwidth interconnection devices supported by the distributed cluster. For example, if the number of high-bandwidth interconnection devices is 32, the tensor parallel search set is 1, 2, 4, 8, 16, 32. Alternatively, if the number of high-bandwidth interconnection devices is small, the set of all integers in the range of 1 to the number of high-bandwidth interconnection devices can be defined as the tensor parallel search set. For example, if the number of high-bandwidth interconnection devices is 8, the tensor parallel search set is 1, 2, 3, 4, 5, 6, 7, 8.

[0072] The pipeline parallel search set can be an integer starting from 1, and supports setting some prior restrictions, including but not limited to setting the maximum value of pipeline parallelism or setting the number of pipeline parallelism to be divisible by the number of model layers. For example, when the number of model layers is 96, if the maximum value of pipeline parallelism is set to 8 and the number of pipeline parallelism is set to be divisible by the number of model layers, the pipeline parallel search set can be 1, 2, 4, 6, 8.

[0073] Sequence parallelism needs to be used together with tensor parallelism and the number of sequence parallelism is equal to the number of tensor parallelism, so there is no need to define a sequence parallel search set.

[0074] The context parallel search set can be defined as an integer in the range of 1 to the maximum context parallel number. Specifically, the context parallel search set can be: a set including 1 and the maximum context parallel number, and satisfying the restriction that the context parallel number in the search set is divisible by the context size (that is, context length information). The context length information is the context length of the computing task input into the target large model.

[0075] There are two ways of expert parallelism. One is that all experts are distributed on different devices, and the other is that some of all experts are deployed on the same device. If all experts are distributed on different devices, the number of experts in parallel is equal to the total number of model experts; if some of all experts are deployed on the same device, the number of experts in parallel is equal to 1. Among them, experts can be understood as specialized sub-models, which can be an independent neural network, usually a feedforward network, or a more complex network structure. Each expert can generally participate in solving problems in the field of his expertise through a "gated network".

[0076] The maximum number of data parallelism can be calculated based on the total number of cluster devices. The maximum number of data parallelism is equal to the total number of cluster devices divided by the number of devices in one data parallel group. Among them, the number of devices in one data parallel group is equal to the product of the number of tensor parallelism, the number of pipeline parallelism, and the number of context parallelism. The search set of data parallelism is an integer (integer or power of 2) in the range from 1 to the maximum number of data parallelism, which can include 1 and the maximum number of data parallelism.

[0077] Embodiment 2

[0078] Figure 2 A flow chart of a method for determining a distributed parallel scheme provided in Embodiment 2 of the present invention. This embodiment is a refinement of the single-device calculation graph for generating a target large model under each alternative distributed parallel scheme in the above-mentioned embodiment, specifically including: obtaining the current tensor parallel number, the current pipeline parallel number and the current context parallel number in the target alternative distributed parallel scheme currently being processed; according to the current pipeline parallel number and the preset load distribution strategy, converting the original calculation graph corresponding to the target large model into a pipeline calculation graph on each pipeline; according to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, calculating a single-device calculation operator graph; according to the parallel segmentation dimension of the input and output tensors of each calculation operator in the single-device calculation operator graph, deriving the communication operator to be inserted in the single-device calculation operator graph and the insertion position of the communication operator; according to the insertion position, inserting the communication operator into the single-device calculation operator graph to obtain the single-device calculation graph under the target alternative distributed parallel scheme.

[0079] Correspondingly, such as Figure 2 As shown, the method includes:

[0080] S210: Obtain model information of a target large model for distributed system modeling and cluster information of a distributed cluster for deploying the target large model.

[0081] S220. Determine a search set corresponding to each distributed strategy based on computing tasks and cluster information that match the target large model, and obtain multiple candidate distributed parallel solutions based on each search set.

[0082] In this embodiment, after determining the search set corresponding to each distributed strategy and the relationship between the parallel numbers under different distributed strategies, all alternative distributed parallel schemes can be determined by permutations and combinations, and each alternative distributed parallel scheme includes the parallel number under each distributed strategy.

[0083] S230, obtaining the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel solution currently being processed.

[0084] S240. According to the current number of parallel pipelines and a preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline.

[0085] In the embodiment of the present invention, the load distribution strategy can be specifically understood as: a method of allocating tasks to different computing nodes in the system to optimize resource utilization, improve system performance, and ensure efficient completion of tasks. An effective load distribution strategy can balance the workload of each node, avoid overloading some nodes while other nodes are idle, thereby improving overall computing efficiency and throughput. For example, polling and random allocation, which are easy to implement but may not consider the actual load of the node, resulting in uneven load; least connection and least load strategies, which can allocate tasks according to the current load of the node to avoid overload; weighted polling and hash allocation, which ensure load balancing and session consistency by considering the weight of the node or the key attributes of the task.

[0086] Typically, the load distribution strategy can be a load balancing strategy that evenly distributes each computational operator in the original computational graph to each pipeline that matches the number of parallel operations of the current pipeline. The computational operator can be understood as an operator used to implement the set operation logic, such as addition, square operation, or convolution operation.

[0087] In a specific example, if the original computation graph contains 10 sequentially connected computational operators and the current pipeline parallel number is 2, the first 5 computational operators in the original computation graph can be divided into a pipeline computation graph on one pipeline, and the last 5 computational operators in the original computation graph can be divided into a pipeline computation graph on another pipeline.

[0088] On the basis of the above embodiments, considering that expert parallelism can enable experts who are "good at" the set field to perform the set type of calculation, the number of expert parallelism can be further combined to update the pipeline calculation graph on each pipeline. Alternatively, considering that the actual impact of the number of expert parallelism on the pipeline calculation graph is small, in order to simplify the calculation, the impact of the number of expert parallelism on the pipeline calculation graph can also be ignored.

[0089] S250. Calculate a single-device computing operator graph according to the pipeline computing graph, the current tensor parallel number, and the current context parallel number.

[0090] In the embodiment of the present invention, the single-device computing operator graph can be specifically understood as: a computing graph constructed and executed on a single computing device, in which the computing operators contained therein are mainly used for tensor calculations.

[0091] In this embodiment, considering that sequence parallelism is used together with tensor parallelism, and the number of sequence parallelism is the same as the number of tensor parallelism, the number of sequence parallelism is no longer introduced here. Specifically, tensor (sequence) parallelism and context parallelism are mainly used to realize the splitting of activations or weights in the tensor dimension. This operation reduces the size of the input and output tensors of the computational operator. Only local calculation results will be calculated on each single device, thereby reducing the amount of calculation on a single device. However, these local calculation results need to be transmitted and aggregated between multiple devices to obtain the final complete calculation result. Therefore, it is necessary to introduce a communication operator in the obtained single-device computational operator graph.

[0092] In this embodiment, the calculated single-device computing operator graph includes the input and output tensor sizes of each computing operator.

[0093] In this embodiment, the parallel splitting dimension of the input and output tensors of each computational operator in the single-device computational operator graph can be derived by combining the tensor size of the input and output tensors of each computational operator in the original computational graph (which can be obtained through prior knowledge or derived based on its actual input tensor), the operator semantics of each computational operator, and the pre-annotated splitting description information.

[0094] In a specific example, if the current tensor (sequence) parallel number is determined to be 8, the current context parallel number is 1, the computational operator currently being processed is the matmul operator for implementing matrix multiplication, and it is pre-specified that the tensor (sequence) is split on the first dimension (starting from the 0th dimension) of the matmul operator. When the original input tensors of the matmul operator are A[b, s, h] and W1[h, h], it can be deduced that the parallel split dimension of the input tensor A of the matmul operator is [b, s / 8, h], and the parallel split dimension of the input tensor W1 is [h, h / 8]. Furthermore, combined with the semantics of the matmul operator, it can be deduced that the parallel split dimension of the output tensor B of the matmul operator is [b, s / 8, h / 8].

[0095] It should be noted that data parallelism is equivalent to allocating a complete computing task to computing devices in different data parallel groups for parallel computing. It does not split the target large model and will not change the single-device computing graph, but it will introduce communication time between different computing devices. Therefore, when calculating the single-device computing operator graph, the current number of data parallelism is not taken into account.

[0096] S260. According to the parallel segmentation dimension of the input and output tensors of each computing operator in the single-device computing operator graph, derive the communication operator that needs to be inserted in the single-device computing operator graph and the insertion position of the communication operator.

[0097] Specifically, a single-device computation graph refers to a computational process built on a computing device, which includes operators that perform computational tasks and possible communication operations. In such a computation graph, computational operators and communication operators work together to complete the forward and backward propagation of the model. After obtaining the model information of the target large model to be modeled in the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, determining the search set corresponding to each distributed strategy, and obtaining multiple alternative distributed parallel schemes, obtain the current tensor parallel number, current pipeline parallel number, and current context parallel number in the target alternative distributed parallel scheme currently being processed, and first analyze the original large model computation graph, understand the structure of the model, data flow, and computational requirements, and identify the nodes (such as convolutional layers, fully connected layers, and activation functions, etc.) and edges (data flows) in the graph. Then, according to the current number of pipeline parallelism and the preset load distribution strategy, the original calculation graph is decomposed into multiple pipeline stages, each stage executes a part of the computing tasks, and the uniform distribution of the model layers or computing tasks is achieved. The point-to-point communication operators for sending and receiving are added to convert the original calculation graph into a calculation graph on the pipeline to achieve parallel execution of multiple operations and improve throughput and resource utilization. Then, according to the pipeline calculation graph, the current number of tensor parallelism, and the current number of context parallelism, the original pipeline calculation graph is analyzed to understand the structure and data flow of the model, and the parallel execution strategy of the model is determined according to the number of context parallelism and tensor parallelism. The pipeline calculation graph is integrated into parts suitable for execution on a single device, and optimizations such as operator fusion and constant folding are performed to improve execution efficiency. The optimized calculation graph is mapped to a single device, and a single-device computing operator graph optimized for the device is generated. According to this operator graph, the computing tasks are scheduled and executed on the device.

[0098] Specifically, according to the operator tensor parallel splitting dimension, determine the splitting method of the input and output tensors of each computing operator, for example, whether it is split by row, column or block, understand the role of each operator in tensor parallelism, and how it operates the split tensor, and check whether the input and output tensor layouts between adjacent operators match. If the output tensor layout of the previous operator is inconsistent with the input tensor layout of the next operator, then it is necessary to insert a communication operator to rearrange the tensor. According to the tensor splitting method and the layout compatibility between operators, determine which communication operators are needed to synchronize or reorganize data, insert the communication operator between operators with incompatible layouts to ensure that the data is correctly synchronized or reorganized before continuing processing, and realize automatic derivation of the inserted tensor parallel communication operator type and the insertion position of the communication operator, such as the full protocol communication operator, the protocol distribution communication operator and the full aggregation communication operator, and insert the communication operator into the single-device computing operator graph to obtain the single-device computing graph.

[0099] As mentioned above, after tensor parallelization of a computing operator, if the complete output tensor of the computing operator needs to be obtained, it is necessary to perform collective communication on the split output tensors on all computing devices where the computing operator is deployed, and then, it is necessary to add a matching communication operator after the computing operator. In various embodiments of the present invention, the inventor generates the communication operator insertion logic shown in Table 1 by analyzing the tensor conversion logic under different splitting strategies.

[0100] Table 1

[0101]

[0102] Among them, N stands for Not Split. That is, the tensor is not split and is a complete copy on all computing devices. G(i) stands for Group Split, which is used to divide the tensor into different groups. The tensor is complete on all computing devices in the group to which it belongs and does not exist on other groups. i indicates that the tensor belongs to the i-th group. D(i) stands for Dim Split. The tensor is equally divided on all computing devices along the i-th dimension, and each computing device stores a tensor of size 1 / n (n is the total size of dimension i). V is value Split. The size of the tensor on each computing device is the same as the original tensor, but the sum of the tensor values ​​on all computing devices is equal to the original tensor. Among them, the splitting semantics corresponding to pipeline parallelism includes group splitting; the splitting semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include dimension splitting or value splitting; the splitting semantics corresponding to data parallelism includes dimension splitting; the splitting semantics corresponding to expert parallelism includes dimension splitting, group splitting or data splitting.

[0103] In a specific example, if it is determined that in the single-device computing operator graph, the parallel split dimension of the output tensor of the previous computing operator belongs to D(i), and the parallel split dimension of the input tensor of the next computing operator belongs to N, it means that an All-Gather communication operator needs to be inserted between the previous computing operator and the next computing operator.

[0104] S270. Insert the communication operator into the single-device computing operator graph according to the insertion position to obtain a single-device computing graph under the target candidate distributed parallel solution.

[0105] Through the above settings, a single-device computing graph including a communication operator can be obtained. The communication operator may include: a communication operator for realizing pipeline parallelism or a communication operator for realizing tensor parallelism. Furthermore, the communication operator may also include a communication operator for realizing expert parallelism, etc.

[0106] S280, calculating the single-device computing time of each single-device computing graph, and calculating the end-to-end total time corresponding to each alternative distributed parallel solution based on the single-device computing time of each single-device computing graph.

[0107] S290. According to the calculated calculation time of each single device and the end-to-end total time, determine the target distributed parallel scheme when deploying the target large model in the distributed cluster from among the alternative distributed parallel schemes.

[0108] Specifically, the corresponding single-device computing time is calculated and added up to obtain the end-to-end total time corresponding to each alternative distributed parallel solution; based on the calculated single-device computing time and end-to-end total time, a distributed solution that meets actual needs is screened out from each alternative distributed parallel solution, for example, a distributed parallel solution with optimal performance (minimum end-to-end total time or minimum single-device time, etc.) and minimum computing device consumption, is used as the target distributed parallel solution when the target large model is deployed in a distributed cluster, thereby realizing the determination of the distributed parallel solution.

[0109] The technical solution of the embodiment of the present invention first obtains the model information of the target large model to be modeled for the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, and determines the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and then obtains multiple alternative distributed parallel schemes according to each search set, obtains and converts the original calculation graph corresponding to the target large model into a pipeline calculation graph on each pipeline according to the current tensor parallel number, the current pipeline parallel number and the current context parallel number in the target alternative distributed parallel scheme currently being processed, and combines with the preset load distribution strategy, calculates the single-device computing operator graph according to the current tensor parallel number and the current context parallel number, and derives the communication operator to be inserted in the single-device computing operator graph according to the parallel segmentation dimension of the input and output tensors of each computing operator in the graph, and the insertion position of the communication operator, and inserts it into the single-device computing operator graph to obtain the single-device computing graph under the target alternative distributed parallel scheme, calculates the corresponding single-device computing time and then obtains the end-to-end total time, and determines the target distributed parallel scheme accordingly. According to the model information of the target large model and the cluster information of the distributed cluster, multiple alternative distributed parallel schemes are determined. Through the distributed computing graph and parallel computing technology, the original computing graph is converted into a pipeline computing graph and then a single device computing graph is constructed. Through pipeline parallelism, different parts of the model can be distributed to different devices, so that each device is processing data, reducing the idle time of the device, improving the overall utilization of resources, speeding up the training speed of the model, and better balancing the load of each device. Finally, the target distributed parallel scheme with the best performance is determined based on the total time consumption of each scheme, avoiding time-consuming iterative testing of each distributed parallel scheme one by one, and realizing automatic screening and decision-making of alternative schemes based on cluster device information and model information of the target large model, reducing labor costs and equipment resource consumption, and improving the efficiency of determining distributed parallel schemes.

[0110] Optionally, in the embodiment of the present invention, calculating the single-device computing time consumption of each single-device computing graph may include:

[0111] Obtain the target single-device calculation graph currently being processed; calculate the first-category calculation time of each computing operator and the second-category calculation time of each communication operator in the target single-device calculation graph according to the cluster information of the distributed cluster; calculate the single-device calculation time of the target single-device calculation graph according to the deployment scenario information of the target large model on the distributed cluster, the first-category calculation time and the second-category calculation time.

[0112] Specifically, the single-device computing time of the target single-device computing graph can be calculated through the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time. Among them, the first type of computing time refers to the time consumption of each computing operator in the target single-device computing graph, and the second type of computing time refers to the time consumption of each communication operator in the target single-device computing graph. The time consumption of computing operators and communication operators can be collected and analyzed through the performance analysis tools provided by the deep learning framework, such as the MindInsight tool of the MindSpore framework, to obtain detailed time consumption statistics of operators including the time consumption of computing operators and communication operators.

[0113] Optionally, in an embodiment of the present invention, the first type of computing time consumption of each computing operator in the target single-device computing graph is calculated according to the cluster information of the distributed cluster, including:

[0114] Obtain the target computing operator currently being processed in the target single-device computing graph; calculate the input and output memory access time of the target computing operator according to all input tensor sizes and all output tensor sizes of the target computing operator, as well as the device memory access bandwidth and memory access efficiency; calculate the computation time of the target computing operator according to the floating-point operations of the target computing operator, as well as the number of computing operations per second of the device and the device computing efficiency; calculate the first category computation time of the target computing operator according to the predetermined operator kernel program startup time, the input and output memory access time of the target computing operator, and the computation time of the target computing operator.

[0115] Specifically, the target computing operator currently being processed is obtained in the target single-device computing graph; according to all input tensor sizes and all output tensor sizes of the target computing operator, as well as the device memory access bandwidth and memory access efficiency, the input and output memory access time of the target computing operator is calculated.

[0116] That is, the input and output memory access time of the target computing operator = (the size of all input tensors of the operator + the size of all output tensors of the operator) / (device memory access bandwidth × memory access efficiency).

[0117] Calculate the computation time of the target computing operator based on the floating-point operations of the target computing operator, the number of computing operations per second of the device, and the computing efficiency of the device.

[0118] That is, the computation time of the target computing operator = the number of floating-point operations of the operator / (the number of computing operations per second of the device × computing efficiency.

[0119] Among them, the device memory access bandwidth, memory access efficiency, number of computing operations per second of the device and computing efficiency match the hardware specifications of the computing device and can be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0120] Finally, the first type of computing time of the target computing operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target computing operator, and the computing time of the target computing operator.

[0121] That is, the first-category computing time of the target computing operator = the operator kernel program startup time + max(the input and output memory access time of the target computing operator, the computing time of the target computing operator), where the operator kernel program startup time is the inherent time of the corresponding device, and max(·) is the maximum value function.

[0122] Optionally, in an embodiment of the present invention, calculating the second type of computing time consumption of each communication operator in the target single-device computing graph according to the cluster information of the distributed cluster may include:

[0123] In the target single-device computation graph, obtain the target communication operator currently being processed, and obtain the parallel type of the target communication operator; if the parallel type is tensor parallelism, calculate the second type of computational time consumption of the target communication operator according to the device interconnection mode, the number of tensor parallelisms, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator; if the parallel type is pipeline parallelism, calculate the second type of computational time consumption of the target communication operator according to the pipeline parallel communication operator communication volume, the pipeline parallel communication bandwidth, and the pipeline parallel communication efficiency.

[0124] In an optional implementation of this embodiment, generally speaking, there are two types of device interconnection between computing devices: full connection or ring connection. According to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth and the operator type of the target communication operator, a corresponding time consumption formula is selected to calculate the second type of computing time of the tensor parallel target communication operator.

[0125] In a specific example, when the device interconnection mode in the distributed cluster is fully connected, the calculation formula for the second type of computing time consumption of the target communication operator with the parallel type of tensor parallelism = tensor parallel communication operator communication volume / (tensor parallel communication bandwidth × tensor parallel communication efficiency).

[0126] Among them, the communication volume of tensor parallel communication operators = the original data volume of the operator × (computing device-1) / total number of computing devices. The original data volume of the operator can be understood as the original data volume before tensor parallelization according to the tensor parallel number. The original data volume is compatible with the operator type of the target communication operator, the total number of computing devices is consistent with the tensor parallel number, and the tensor parallel communication bandwidth and tensor parallel communication efficiency match the hardware specifications of the computing device, which can be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0127] In another optional implementation of this embodiment, the calculation formula for the second type of calculation time consumption of the target communication operator whose parallel type is pipeline parallel=pipeline parallel communication operator communication volume / (pipeline parallel communication bandwidth×pipeline parallel communication efficiency).

[0128] The communication volume of pipeline parallel communication operators can be understood as the data volume of a single card. For example, if the amount of data processed by each computing device is A, then the communication volume of pipeline parallel communication operators is also A. The pipeline parallel communication bandwidth and pipeline parallel communication efficiency can also be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0129] Furthermore, the method may further include: if the parallel type is expert parallelism, the second type of computing time consumption of the target communication operator may be calculated according to the expert parallel communication operator communication volume, the expert parallel communication bandwidth and the expert parallel communication efficiency.

[0130] Optionally, in an embodiment of the present invention, calculating the single-device computing time of the target single-device computing graph according to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time may include:

[0131] If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth and data parallel communication efficiency, and the single-device calculation time of the target single-device calculation graph is calculated based on the first type of calculation time, the second type of calculation time and the gradient reduction time; if the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device calculation time of the target single-device calculation graph is calculated based on the first type of calculation time and the second type of calculation time.

[0132] Specifically, for the deployment scenario information of the target large model on the distributed cluster, the model training scenario, the single device computing time of the target single device computing graph includes not only the first type of computing time and the second type of computing time, but also the gradient reduction time. Among them, the gradient reduction time can be calculated based on the model gradient reduction communication volume, data parallel communication bandwidth and data parallel communication efficiency.

[0133] That is, the calculation formula for gradient reduction time = model gradient reduction communication volume / (data parallel communication bandwidth × data parallel communication efficiency).

[0134] The gradient reduction communication volume of the model refers to the amount of gradient information of all computational operators. The data parallel communication bandwidth and data parallel communication efficiency can also be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0135] Specific application scenarios

[0136] For ease of understanding, a specific application scenario to which each embodiment of the present invention is applicable is now described. In this application scenario, specific operation steps for efficiently generating a 6D (6 types of distributed strategies) distributed strategy when mapping a specific large model to a specific distributed cluster are specifically described.

[0137] Specifically, Figure 3 As shown in the figure, the 6D distributed parallel generation method of the large model is broken down into six functional modules as shown in the figure below, namely, the storage calculation required for the large model, the definition of the distributed parallel search space, the acquisition of a reasonable combination of distributed parallel solutions, the construction of a single-card calculation graph (that is, a single-device calculation graph), the end-to-end total time calculation, and the generation of the best distributed solution.

[0138] 1. Storage and computing required for large models

[0139] in, Figure 4 A schematic diagram of a model training storage content applicable to an embodiment of the present invention. As mentioned above, after a specific large model is deployed in a specific distributed cluster, the model training task or model reasoning task can be executed accordingly. Different task types may not be suitable for the best distributed solution, because the above six functional modules can be executed according to the actual task type. Specifically, for different task types, the calculation content of the storage calculation required by the large model is not consistent. In this specific application scenario, taking the model training task as an example, the storage calculation required by the large model is introduced.

[0140] Specifically, Figure 4 As shown, the content of model training storage may include: weight-related storage (whose size is equivalent to the weight-related storage requirement in the previous text) and activation-related storage (whose size is equivalent to the activation-related storage requirement in the previous text). Among them, weight-related storage includes weight storage, gradient storage, and optimizer storage. Activation-related storage includes activation storage and temporary storage. Based on the above specific storage content, the amount of data in the weight-related storage during model training storage and the amount of calculation in the activation-related storage can be calculated accordingly.

[0141] 2. Distributed Parallel Search Space Definition

[0142] In this application scenario, the entire distributed parallel search space is orthogonally obtained by tensor parallel search set, pipeline parallel search set, top-bottom parallel search set and data parallel search set. The definitions of each search set are as follows:

[0143] (1) Tensor parallel search set: Based on the number of high-bandwidth interconnect devices supported by the cluster, the tensor parallel search set is defined as a set of powers of 2 ranging from 1 to the number of high-bandwidth interconnect card devices. For example, if the number of high-bandwidth interconnect devices is 32, the tensor parallel search set is 1, 2, 4, 8, 16, and 32.

[0144] (2) Pipeline parallel search set: The pipeline parallel search set can be an integer starting from 1. The present invention supports setting some a priori restrictions, such as setting the maximum value of pipeline parallelism, and setting the number of pipeline parallelism to be divisible by the number of model layers. Taking GPT175B as an example, the number of layers of the model is 96. If the maximum value of pipeline parallelism is set to 8 and the number of pipeline parallelism is set to be divisible by the number of model layers, the pipeline parallel search set is 1, 2, 4, 6, 8.

[0145] (3) Sequence parallel search set: Sequence parallelism needs to be used together with tensor parallelism and the number of sequence parallelism is equal to the number of tensor parallelism, so there is no need to define a sequence parallel search set.

[0146] (4) Context parallel search set: The context parallel search set is defined as a set of integers ranging from 1 to the maximum context parallel number, including 1 and the maximum context parallel number, and satisfying the restriction that the context parallel number is divisible by the context size.

[0147] (5) Number of experts in parallel: There are two ways to parallelize experts. One is that all experts are distributed on different devices, and the other is that some of all experts are on the same device. If all experts are distributed on different devices, the number of experts in parallel is equal to the number of model experts; if some of all experts are on the same device, the number of experts in parallel is equal to 1.

[0148] (6) Data parallel search set: The maximum number of data parallels is calculated based on the total number of cluster devices. The maximum number of data parallels is equal to the total number of cluster devices divided by the number of devices in one data parallel group, where the number of devices in one data parallel group = the number of tensor parallels × the number of pipeline parallels × the number of context parallels. The data parallel search set is an integer (integer or power of 2) in the range from 1 to the maximum number of data parallels, and can be 1 and the maximum number of data parallels.

[0149] 3. Obtain a reasonable combination of distributed parallel solutions

[0150] Figure 5 FIG. 1 is a schematic diagram of obtaining a reasonable distributed parallel solution applicable to an embodiment of the present invention. Figure 5 As shown, the steps to obtain a reasonable distributed parallel solution combination are as follows:

[0151] (1) Traverse each distributed parallel solution defined in the distributed search space.

[0152] (2) Still taking the model training task as an example, the single-device storage calculation required by the distributed parallel solution = single-device weight-related storage size + single-device activation-related storage size. The single-device weight-related storage size is the weight-related storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the pipeline parallel number; the single-device activation-related storage size is the activation storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the context parallel number. The single-device temporary storage size is the temporary storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the context parallel number.

[0153] (3) For each distributed parallel solution, a single device storage condition check is performed. If the single device storage required by the distributed parallel solution is less than or equal to the single device storage specification multiplied by the storage utilization rate, then it is a reasonable distributed parallel solution; otherwise, it is an invalid parallel solution.

[0154] 4. Build a single card calculation graph

[0155] Figure 5 This is a schematic diagram of constructing a single-card calculation graph applicable to an embodiment of the present invention. The original calculation graph is first converted into a pipeline calculation graph, and then the single-card calculation graph is constructed. Figure 5 As shown, the specific method of constructing a single-card calculation graph is described as follows:

[0156] (1) Build a pipeline computation graph.

[0157] Evenly distribute strategies or computing tasks according to the number of model layers, evenly distribute strategies and the number of pipeline parallelism, and add point-to-point communication operators for sending and receiving to convert the original calculation graph into a calculation graph on the pipeline.

[0158] (2) Construct a computational graph for a single device.

[0159] A computational operator graph for a single device is obtained from a pipeline computation graph, the number of context parallelism, and the number of tensor parallelism. The type of inserted tensor parallel communication operators is automatically derived based on the operator tensor parallel segmentation dimension, such as the all-reduce communication operator, the reduce-scatter communication operator, and the all-gather communication operator. The computational graph for a single device is obtained, including computational operators and communication operators.

[0160] 5. End-to-end total time calculation method

[0161] First, calculate the time consumed by computing the graph on a single device, and then reversely calculate the end-to-end total time consumed based on the pipeline parallel scheduling strategy, such as a forward and a reverse scheduling strategy. The method for computing the graph on a single device is as follows:

[0162] The time consumption of a single device calculation graph includes the time consumption of computing operators, tensor parallel communication operators, and pipeline parallel communication. If it is a model training task, it also includes the time consumption of gradient reduction communication.

[0163] The time consumption of the single-device calculation graph is calculated based on the single-device calculation graph and the parallel scheduling method. If the communication operator and the calculation operator are executed serially, the single-device calculation graph time consumption = the sum of the calculation operator time consumption + the sum of the tensor parallel communication operator time consumption + the pipeline parallel communication time consumption + the gradient reduction time consumption. If some tensor parallel communication operators are executed in parallel with some calculation operators, the single-device calculation graph time consumption needs to first take the maximum value of the communication time consumption and the calculation operator time consumption of the partial parallel execution, and then sum it with the other time consumption. The calculation formula for each time consumption is as follows:

[0164] (1) Computational single operator time-consuming calculation formula.

[0165] The input and output memory access time of the target computing operator = (the size of all input tensors of the operator + the size of all output tensors of the operator) / (device memory access bandwidth × memory access efficiency).

[0166] The computation time of the target computing operator = the number of floating-point operations of the operator / (the number of computing operations per second of the device × computing efficiency).

[0167] The first type of computing time of the target computing operator = the operator kernel program startup time + max (the input and output memory access time of the target computing operator, the computing time of the target computing operator), where the operator kernel program startup time is the inherent time of the corresponding device.

[0168] (2) Formula for calculating the time consumption of a single operator in tensor parallel communication.

[0169] There are two types of device interconnection: full connection and ring connection. Select the corresponding time consumption formula to calculate the communication time consumption based on the device interconnection mode, the number of tensor parallel operations, the tensor parallel interconnection bandwidth, and the communication operator type. For example, the time consumption calculation formula for the fully connected communication operator = the communication volume of the tensor parallel communication operator ( / (tensor parallel communication bandwidth × tensor parallel communication efficiency).

[0170] (3) Formula for calculating the time consumption of a single operator in pipeline parallel communication.

[0171] The time consumption calculation formula of a single operator in pipeline parallel communication = pipeline parallel communication operator communication volume / (pipeline parallel communication bandwidth × pipeline parallel communication efficiency).

[0172] (4) Gradient reduction time-consuming calculation formula.

[0173] Gradient reduction time calculation formula = Model gradient reduction communication volume / (data parallel communication bandwidth × data parallel communication efficiency)

[0174] 6. Determine the best distributed parallel solution

[0175] The present invention supports multiple performance indicators such as minimum end-to-end total time consumption, best single device performance or minimum total number of devices, and outputs the corresponding optimal distributed parallel solution.

[0176] The embodiment of the present invention proposes a method for determining a distributed parallel solution. According to the structure of the model, the model size, the context length, the specifications of the cluster hardware equipment, and the interconnection between the equipment, a distributed parallel solution with the minimum end-to-end total time consumption, the best single device performance, or the minimum total number of equipment is determined, so as to avoid algorithm personnel from iteratively running the computing task one by one on all feasible distributed combination solutions in a cluster environment of 1,000 or 10,000 cards, and comparing and analyzing the results, and realizing automatic screening and decision-making of backup solutions based on the cluster equipment information and the model information of the target large model, thereby reducing the consumption of manpower and equipment resources and improving the efficiency of determining distributed parallel solutions.

[0177] Embodiment 3

[0178] Figure 7 This is a schematic diagram of a structure of a device for determining a distributed parallel solution provided in Embodiment 4 of the present invention. Figure 7 As shown, the device is configured in an electronic device, and specifically includes: an information acquisition module 710, an alternative distributed parallel solution determination module 720, a single device computing time calculation module 730, a total time calculation module 740 and a target distributed parallel solution determination module 750.

[0179] The information acquisition module 710 is used to acquire model information of a target large model to be modeled for a distributed system, and cluster information of a distributed cluster for deploying the target large model;

[0180] The candidate distributed parallel scheme determination module 720 is used to determine the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and obtain multiple candidate distributed parallel schemes according to each search set; wherein each candidate distributed parallel scheme includes the parallel number under at least one distributed strategy;

[0181] A single-device computing time calculation module 730 is used to generate a single-device computing graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device computing time of each single-device computing graph, wherein the single-device computing graph includes a communication operator;

[0182] A total time consumption calculation module 740 is used to calculate the end-to-end total time consumption corresponding to each candidate distributed parallel solution according to the single device calculation consumption of each single device calculation graph;

[0183] The target distributed parallel scheme determination module 750 is used to determine the target distributed parallel scheme when deploying the target large model in the distributed cluster from among the alternative distributed parallel schemes according to the calculated calculation time of each single device and the end-to-end total time.

[0184] The technical solution of the embodiment of the present invention first obtains the model information of the target large model to be modeled in the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, and determines the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and then obtains multiple alternative distributed parallel schemes according to each search set, generates a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculates the single-device calculation time of each single-device calculation graph, and then calculates the end-to-end total time corresponding to each alternative distributed parallel scheme according to the single-device calculation time of each single-device calculation graph, and determines the target distributed parallel scheme when deploying the target large model in the distributed cluster in each alternative distributed parallel scheme according to the calculated single-device calculation time and the end-to-end total time. According to the model information of the target large model and the cluster information of the distributed cluster, multiple alternative distributed parallel schemes are determined, and the alternative distributed parallel schemes are screened and evaluated through distributed computing graphs and parallel computing technology. The target distributed parallel scheme with the best performance is determined according to the total time consumption of each scheme, thereby avoiding time-consuming iterative testing of each distributed parallel scheme one by one. Automatic screening and decision-making of alternative schemes are achieved based on cluster device information and model information of the target large model, which reduces labor costs and equipment resource consumption and improves the efficiency of determining distributed parallel schemes.

[0185] Furthermore, in an embodiment of the present invention, a storage requirement calculation module is also included, which is used to calculate the weight-related storage requirements and activation-related storage requirements matching the target large model based on the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster.

[0186] In the embodiment of the present invention, the candidate distributed parallel solution determination module 720 is specifically used to:

[0187] According to the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated respectively; among each alternative distributed parallel scheme, the invalid distributed parallel scheme whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster is filtered out.

[0188] Optionally, the alternative distributed parallel solution determination module 720 includes a parallel number acquisition unit, a single device weight-related storage requirement calculation unit, a single device activation-related storage requirement calculation unit, and a single device storage requirement calculation unit.

[0189] A parallel number acquisition unit, used to acquire the current tensor parallel number, the current pipeline parallel number and the current context parallel number in the target candidate distributed parallel scheme currently being processed;

[0190] A single device weight-related storage requirement calculation unit, used to calculate the single device weight-related storage requirement according to the weight-related storage requirement, the current tensor parallel number and the current pipeline parallel number;

[0191] A single device activation-related storage requirement calculation unit, used for calculating the single device activation-related storage requirement according to the activation-related storage requirement, the current tensor parallel number and the current context parallel number;

[0192] The single-device storage requirement calculation unit is used to calculate the single-device storage requirement corresponding to the target alternative distributed parallel solution according to the single-device weight-related storage requirement and the single-device activation-related storage requirement.

[0193] In the embodiment of the present invention, the distributed strategies include: pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism.

[0194] In the embodiment of the present invention, a search set corresponding to each distributed strategy is determined according to the computing tasks and cluster information matching the target large model, including at least one of the following:

[0195] Determine a tensor parallel search set corresponding to the tensor parallel according to the number of high-bandwidth interconnect devices supported by the distributed cluster in the cluster information;

[0196] According to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, a pipeline parallel search set corresponding to the pipeline parallelism is determined;

[0197] Extracting hardware information from cluster information, and determining the maximum number of context parallelisms based on the hardware information; determining context length information based on computing tasks matching the target large model; determining a context parallel search set corresponding to the context parallelism based on the maximum number of context parallelisms and the context length information;

[0198] According to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallel;

[0199] According to the total number of cluster devices in the cluster information, the determined tensor parallel search set, pipeline parallel search set and context parallel search set, the maximum number of data parallelism is calculated, and according to the maximum number of data parallelism, the data parallel search set corresponding to the data parallelism is determined.

[0200] In the embodiment of the present invention, the single device computing time consumption calculation module 730 is specifically used for:

[0201] Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed;

[0202] According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline;

[0203] According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated;

[0204] According to the parallel segmentation dimension of the input and output tensors of each computing operator in the single-device computing operator graph, the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived;

[0205] According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

[0206] Furthermore, the single device computing time consumption calculation module 730 is also specifically used for:

[0207] Get the target single device calculation graph currently being processed;

[0208] According to the cluster information of the distributed cluster, the first type of computing time of each computing operator and the second type of computing time of each communication operator in the target single device computing graph are calculated;

[0209] According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single-device computing time of the target single-device computing graph is calculated.

[0210] Optionally, the single-device computing time calculation module 730 includes a target computing operator acquisition unit, an input and output memory access time calculation unit, a target computing operator computing time calculation unit and a first type computing time calculation unit.

[0211] A target computing operator acquisition unit, used to acquire a currently processed target computing operator in a target single-device computing graph;

[0212] An input and output memory access time calculation unit, used to calculate the input and output memory access time of the target computing operator according to all input tensor sizes and all output tensor sizes of the target computing operator, as well as the device memory access bandwidth and memory access efficiency;

[0213] The target computing operator's computing time calculation unit is used to calculate the computing time of the target computing operator according to the floating-point operation number of the target computing operator, the number of computing operations per second of the device, and the computing efficiency of the device;

[0214] The first type of computing time calculation unit is used to calculate the first type of computing time of the target computing operator according to the predetermined operator core program startup time, the input and output memory access time of the target computing operator and the computing time of the target computing operator.

[0215] Optionally, the single-device computing time consumption calculation module 730 includes a parallel type acquisition unit and a second type computing time consumption calculation unit.

[0216] A parallel type acquisition unit, used to acquire the target communication operator currently being processed in the target single-device calculation graph, and acquire the parallel type of the target communication operator;

[0217] The second type of computing time calculation unit is used to calculate the second type of computing time of the target communication operator according to the device interconnection method, the number of tensor parallels, the tensor parallel interconnection bandwidth and the operator type of the target communication operator if the parallel type is tensor parallel; if the parallel type is pipeline parallel, the second type of computing time of the target communication operator is calculated according to the communication volume of the pipeline parallel communication operator, the pipeline parallel communication bandwidth and the pipeline parallel communication efficiency.

[0218] Optionally, the single-device computing time calculation module 730 includes a single-device computing time unit, which is used to calculate the gradient reduction time according to the model gradient reduction communication volume, data parallel communication bandwidth and data parallel communication efficiency if the deployment scenario information of the target large model on the distributed cluster is a model training scenario, and calculate the single-device computing time of the target single-device computing graph according to the first category computing time, the second category computing time and the gradient reduction time; if the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, then the single-device computing time of the target single-device computing graph is calculated according to the first category computing time and the second category computing time.

[0219] The distributed parallel scheme determination device provided in the embodiment of the present invention can execute the distributed parallel scheme determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0220] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0221] Embodiment 4

[0222] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0223] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0224] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network device, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as the determination method of the distributed parallel scheme, that is:

[0226] Obtaining model information of a target large model for distributed system modeling and cluster information of a distributed cluster for deploying the target large model;

[0227] According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, and according to each search set, multiple candidate distributed parallel schemes are obtained; wherein each candidate distributed parallel scheme contains the parallel number under at least one distributed strategy;

[0228] Generate a single-device computation graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device computation time of each single-device computation graph, wherein the single-device computation graph contains communication operators;

[0229] According to the single-device computing time of each single-device computing graph, calculate the end-to-end total time corresponding to each alternative distributed parallel solution;

[0230] According to the calculated computing time of each single device and the end-to-end total computing time, a target distributed parallel scheme is determined from various alternative distributed parallel schemes when deploying the target large model in a distributed cluster.

[0231] In some embodiments, the method for determining a distributed parallel solution may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for determining a distributed parallel solution described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for determining a distributed parallel solution in any other appropriate manner (e.g., by means of firmware).

[0232] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0234] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0235] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0236] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0237] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0238] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0239] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for determining a distributed parallel solution, characterized in that: include: Obtaining model information of a target large model for distributed system modeling and cluster information of a distributed cluster for deploying the target large model; According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, and according to each search set, multiple candidate distributed parallel schemes are obtained; wherein each candidate distributed parallel scheme includes the parallel number under at least one distributed strategy; Generate a single-device computation graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device computation time of each single-device computation graph, wherein the single-device computation graph contains communication operators; According to the single-device computing time of each single-device computing graph, calculate the end-to-end total time corresponding to each alternative distributed parallel solution; According to the calculated computing time of each single device and the end-to-end total computing time, determine the target distributed parallel scheme when deploying the target large model in the distributed cluster from among the alternative distributed parallel schemes; Among them, the single-device calculation graph of the target large model under each alternative distributed parallel solution is generated, including: Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed; According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline; According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated; According to the parallel segmentation dimension of the input and output tensors of each computing operator in the single-device computing operator graph, the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived; According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

2. The method according to claim 1, characterized in that Also includes: According to the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster, the weight-related storage requirements and activation-related storage requirements matching the target large model are calculated; Accordingly, after obtaining multiple candidate distributed parallel solutions according to each search set, it also includes: According to the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated respectively; Among the candidate distributed parallel solutions, invalid distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster are filtered out.

3. The method according to claim 2, characterized in that Based on the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated separately, including: Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed; The weight-related storage requirements of a single device are calculated based on the weight-related storage requirements, the current tensor parallel number, and the current pipeline parallel number. The activation-related storage requirements of a single device are calculated based on the activation-related storage requirements, the current tensor parallel number, and the current context parallel number. According to the storage requirements related to the weight of a single device and the storage requirements related to the activation of a single device, the storage requirements of a single device corresponding to the target alternative distributed parallel solution are calculated.

4. The method according to any one of claims 1 to 3, characterized in that: The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism.

5. The method according to claim 4, characterized in that According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, including at least one of the following: Determine a tensor parallel search set corresponding to the tensor parallel according to the number of high-bandwidth interconnect devices supported by the distributed cluster in the cluster information; According to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, a pipeline parallel search set corresponding to the pipeline parallelism is determined; Extracting hardware information from the cluster information, and determining the maximum number of context parallelisms according to the hardware information; Determine context length information based on a computational task that matches a target large model; Determine a context parallel search set corresponding to the context parallelism according to the maximum context parallelism quantity and context length information; According to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallel; According to the total number of cluster devices in the cluster information, the determined tensor parallel search set, pipeline parallel search set and context parallel search set, the maximum number of data parallelism is calculated, and according to the maximum number of data parallelism, the data parallel search set corresponding to the data parallelism is determined.

6. The method according to claim 1, characterized in that Calculate the single-device computing time of each single-device computing graph, including: Get the target single device calculation graph currently being processed; According to the cluster information of the distributed cluster, the first type of computing time of each computing operator and the second type of computing time of each communication operator in the target single device computing graph are calculated; According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single-device computing time of the target single-device computing graph is calculated.

7. The method according to claim 6, characterized in that According to the cluster information of the distributed cluster, the first type of computing time consumption of each computing operator in the target single device computing graph is calculated, including: Get the target computing operator currently being processed in the target single-device computing graph; Calculate the input and output memory access time of the target computing operator based on the total input tensor size and total output tensor size of the target computing operator, as well as the device memory access bandwidth and memory access efficiency; Calculate the computation time of the target computing operator based on the floating-point operations of the target computing operator, the number of computing operations per second of the device, and the computing efficiency of the device; The first type of computing time of the target computing operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target computing operator, and the computing time of the target computing operator.

8. The method according to claim 6, characterized in that According to the cluster information of the distributed cluster, the second type of computing time of each communication operator in the target single device computing graph is calculated, including: Obtain the target communication operator currently being processed in the target single-device computation graph, and obtain the parallel type of the target communication operator; If the parallel type is tensor parallel, the second type of computing time of the target communication operator is calculated according to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator; If the parallel type is pipeline parallelism, the second type of computing time consumption of the target communication operator is calculated according to the communication volume of the pipeline parallel communication operator, the pipeline parallel communication bandwidth and the pipeline parallel communication efficiency.

9. The method according to claim 6, characterized in that According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single device computing time of the target single device computing graph is calculated, including: If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth, and data parallel communication efficiency, and the single-device computing time of the target single-device computing graph is calculated based on the first-category computing time, the second-category computing time, and the gradient reduction time. If the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device computing time of the target single-device computing graph is calculated based on the first type of computing time and the second type of computing time.

10. A device for determining a distributed parallel solution, characterized in that: include: An information acquisition module, used to acquire model information of a target large model for distributed system modeling, and cluster information of a distributed cluster for deploying the target large model; The candidate distributed parallel scheme determination module is used to determine the search set corresponding to each distributed strategy according to the computing tasks and cluster information matching the target large model, and obtain multiple candidate distributed parallel schemes according to each search set; wherein each candidate distributed parallel scheme contains the parallel number under at least one distributed strategy; A single-device computing time calculation module is used to generate a single-device computing graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device computing time of each single-device computing graph, wherein the single-device computing graph contains communication operators; A total time consumption calculation module is used to calculate the end-to-end total time consumption corresponding to each alternative distributed parallel solution according to the single-device calculation time consumption of each single-device calculation graph; A target distributed parallel scheme determination module is used to determine a target distributed parallel scheme when deploying the target large model in a distributed cluster from among various candidate distributed parallel schemes according to the calculated calculation time of each single device and the end-to-end total time; Among them, the single device computing time calculation module is specifically used for: Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed; According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline; According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated; According to the parallel segmentation dimension of the input and output tensors of each computing operator in the single-device computing operator graph, the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived; According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for determining a distributed parallel solution according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining a distributed parallel solution according to any one of claims 1 to 9 when the processor executes the computer instructions.

Images