Cost evaluation method, device, equipment, medium and product for distributed system

Abstract

The present invention discloses a cost evaluation method, device, equipment, medium and product for a distributed system. The method includes: obtaining the original calculation graph of a target large model and the target distributed parallel scheme used when deploying the target large model in a target distributed cluster; dividing the original calculation graph according to the target distributed parallel scheme to obtain a single-device calculation graph; determining the operator-level cost of the operator in the single-card calculation graph according to the annotation information of the operator in the single-device calculation graph and the architecture information of the target distributed cluster; calculating the device-level cost according to the operator-level cost, and combining each device-level cost according to each distributed strategy to obtain the cluster-level cost; obtaining the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme according to the operator-level, device-level and cluster-level costs. The technical solution of the embodiment of the present invention can efficiently and accurately evaluate the system performance analysis result in the distributed modeling process.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cost evaluation method, device, equipment, medium and product for a distributed system. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC) and large language models (LLM), deep learning models are undergoing rapid iteration and updating, which is accompanied by the continuous expansion of model scale and the demand for larger clusters.

[0003] In order to cope with the distributed deployment of large models, effective distributed system modeling is often required. In the distributed system modeling process, effectively calculating various costs (typically, time-consuming) for different large models under different distributed parallel strategies is the basis for modeling analysis and distributed system performance evaluation.

[0004] At present, how to efficiently and accurately evaluate the cost of distributed systems is an important issue that needs to be solved. Summary of the invention

[0005] Embodiments of the present invention provide a cost evaluation method, apparatus, device, medium and product for a distributed system, so as to provide a new way of cost evaluation for a distributed system, which effectively reduces the evaluation time and cost while ensuring the accuracy of the cost evaluation.

[0006] According to one aspect of an embodiment of the present invention, a cost evaluation method for a distributed system is provided, comprising:

[0007] Obtaining an original computation graph of a target large model and a target distributed parallel scheme used when deploying the target large model in a target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-marked with basic description information of operators;

[0008] The original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator;

[0009] According to the annotation information of each operator in the single-device computing graph and the architecture information of the target distributed cluster, the pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-card computing graph;

[0010] According to the operator-level cost of each operator, the device-level cost corresponding to each single-device computation graph is calculated, and the device-level costs are combined according to each distributed strategy to obtain the cluster-level cost;

[0011] According to the operator-level cost, device-level cost and cluster-level cost, the system performance analysis results are obtained when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme.

[0012] According to another aspect of an embodiment of the present invention, a cost evaluation device for a distributed system is provided, including:

[0013] A parallel scheme acquisition module is used to acquire the original computation graph of the target large model and the target distributed parallel scheme used when the target large model is deployed in the target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-annotated with basic description information of the operators;

[0014] A single-device computation graph acquisition module, used to segment the original computation graph according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator;

[0015] The operator-level cost acquisition module is used to call the pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-card calculation graph;

[0016] The cluster-level cost acquisition module is used to calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain the cluster-level cost;

[0017] The performance analysis result acquisition module is used to obtain the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution based on the operator-level cost, device-level cost and cluster-level cost.

[0018] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cost evaluation method for a distributed system described in any embodiment of the present invention.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cost evaluation method for a distributed system described in any embodiment of the present invention when executed.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in any embodiment of the present invention are implemented.

[0024] The technical solution of the embodiment of the present invention is to obtain the original calculation graph of the target large model and the target distributed parallel scheme used when deploying the target large model in the target distributed cluster; divide the original calculation graph according to the target distributed parallel scheme to obtain at least one single-device calculation graph for execution in a single computing device; call a pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-card calculation graph; calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy. , get the cluster-level cost; according to the operator-level cost, device-level cost and cluster-level cost, obtain the implementation method of the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme, and provide a new method for distributed system cost evaluation, which can be based on the target distributed parallel scheme used when a specific target large model is deployed in the target distributed cluster, and efficiently and accurately evaluate the operator-level cost, device-level cost and cluster-level cost in distributed modeling, and then obtain accurate system performance analysis results, so as to achieve a better balance in evaluation accuracy, evaluation time and evaluation cost in the process of system performance analysis.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flow chart of a cost evaluation method for a distributed system provided according to Embodiment 1 of the present invention;

[0028] Figure 2 is a flowchart of another cost evaluation method for a distributed system provided according to Embodiment 2 of the present invention;

[0029] Figure 3 It is a structural diagram of a cost evaluation platform of a distributed system to which the present invention is applicable;

[0030] Figure 4 It is a structural diagram of a basic operator cost model applicable to an embodiment of the present invention;

[0031] Figure 5 It is a flow chart of adding optimization strategy pre-processing and optimization strategy post-processing to the cost evaluation process of a distributed system applicable to an embodiment of the present invention;

[0032] Figure 6 This is an overall flow chart for calculating the cluster-level cost applicable to the embodiment of the present invention;

[0033] Figure 7 It is a schematic diagram of a single-device calculation graph generation process applicable to an embodiment of the present invention;

[0034] Figure 8 is a schematic diagram of the structure of a cost evaluation device for a distributed system provided according to Embodiment 3 of the present invention;

[0035] Fig. 9 It is a schematic diagram of the structure of an electronic device for implementing the cost evaluation method of a distributed system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment 1

[0039] Figure 1 This is a flow chart of a cost evaluation method for a distributed system provided in the first embodiment of the present invention. This embodiment is applicable to the case where a system performance analysis is performed on the deployment effect when a specific target large model is deployed in a specific distributed parallel scheme under a specific distributed cluster. The method can be executed by a cost evaluation device for a distributed system, which can be implemented in the form of hardware and / or software and can generally be configured in an electronic device with data processing functions. Figure 1 As shown, the method includes:

[0040] S110, obtaining the original calculation graph of the target large model and the target distributed parallel solution used when deploying the target large model in the target distributed cluster.

[0041] In the embodiment of the present invention, the target large model can be specifically understood as: a model with a large number of parameters or a very long context for distributed system modeling. Among them, due to the large number of parameters, the structure of the model is often very complex. The very long context means that the text data that the model needs to process is very long, requiring the model to be able to process and parse large amounts of text data, which not only involves the storage and call of a large amount of historical information, but also relates to the accurate grasp of the overall coherence and deep meaning of the text. To this end, the model must rely on sufficient computing resources to support its efficient processing of large amounts of data during training and reasoning.

[0042] The target large model may be a deep learning model applied in the fields of image classification, target detection, face recognition, text classification, machine translation or speech recognition, etc. The target large model may be a set pre-trained model.

[0043] Among them, by parsing the target source code file of the target large model, the original calculation graph matching the target large model can be obtained. Specifically, the target source code file is generally obtained by setting a programming language, for example, it is written based on the pytorch open source framework. In the original calculation graph, there are computational operators (also called basic operators) obtained by interconnecting by setting connection relationships. The computational operator can be understood as an operator used to implement the set operation logic, for example, a square operation operator or a convolution operation operator, etc.

[0044] In this embodiment, the original calculation graph of the target large model is pre-annotated with basic description information of operators (typically, computational operators). The basic description information may include: semantic information of the operator (typically, the operation logic of the operator), operator name, operator type, data type of the operator's input and output data, data format of the operator's input and output data, and storage location of the operator's input and output data (for example, L3 or L4), etc. Furthermore, the basic description information of the operator may also include the operator's input and output tensor dimension information, etc.

[0045] In the embodiment of the present invention, the distributed cluster can be specifically understood as: a group of computers (nodes) connected through a network, which work together to complete computing tasks to improve processing power, storage capacity and reliability, while reducing costs and improving computing efficiency. These computers can be physical machines or virtual machines, which share resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, storage and network bandwidth, and can run distributed applications. Among them, GPUs can also be called GPU cards or cards. They are hardware accelerators that perform parallel computing and are particularly suitable for large-scale matrix operations in deep learning. When the model is very large, or the data to be processed is very large, such a large-scale cluster (such as a cluster of thousands or tens of thousands of cards) is needed to provide the required computing power.

[0046] In an optional implementation of this embodiment, the target distributed cluster may include multiple cabinets, each cabinet includes multiple servers, and each server includes multiple computing cards (also referred to as computing devices). The above servers and computing cards are connected to each other using a set topological connection method (for example, a ring or a network) to establish a communication connection.

[0047] Among them, after the target large model is deployed in the target distributed cluster, the model training task or model reasoning task can be executed accordingly. The above-mentioned model training task or model reasoning task can be collectively referred to as computing tasks.

[0048] The target distributed parallel scheme includes the number of parallelism under at least one distributed strategy. The distributed strategy may include: pipeline parallelism (PP), tensor parallelism (TP), sequence parallelism (SP), context parallelism (CP), data parallelism (DP) and expert parallelism (EP).

[0049] Accordingly, the target distributed parallel scheme may specifically include: one or more of the pipeline parallel number, tensor parallel number, sequence parallel number, context parallel number, data parallel number and expert parallel number.

[0050] In this embodiment, the target distributed parallel solution can be understood as a distributed parallel solution that can achieve optimal performance (for example, the least total time consumption, the least time consumption of a single computing device, or the least total computing devices used) after the target large model is deployed in the target distributed cluster.

[0051] S120. Split the original computation graph according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator.

[0052] Specifically, a single-device computation graph refers to a computational process built on a computing device, which includes operators that perform computational tasks and possible communication operations. In such a computation graph, computational operators and communication operators work together to complete the forward and backward propagation of the model.

[0053] Communication operators are operators that are specifically responsible for exchanging data between different computing nodes, ensuring that data processed in parallel on multiple computing devices can be correctly synchronized and aggregated. Single-device computing time refers to the time required to complete a computing task on a single computing node, which can be used to evaluate system performance, optimize algorithms, and schedule tasks. The types of communication operators can include: All-Reduce communication operator, Reduce-Scatter communication operator, All-Gather communication operator, or point-to-point communication operator (P2P), etc.

[0054] It is understandable that by deploying the target large model according to the target distributed parallel scheme, it is often necessary to distribute the various calculations required to be performed by the target large model to different computing devices in different servers located in different cabinets for execution. Furthermore, the original calculation graph needs to be split into single-device calculation graphs executed by each single computing device according to the target distributed parallel scheme. This further involves the situation where different copies of data, or part of a piece of data, are split into different computing devices in the target distributed cluster for execution. Therefore, in order to obtain complete data processing results, it is necessary to insert a communication operator for realizing the data aggregation function at a specific position between the basic operators.

[0055] That is, the final single-device computation graph includes both the basic operators in the original computation graph and the communication operators subsequently inserted for realizing data exchange.

[0056] S130. According to the annotation information of each operator in the single-device computing graph and the architecture information of the target distributed cluster, a pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-card computing graph.

[0057] Among them, the annotation information of each operator in the single-device calculation graph can be understood as the pre-annotated basic description information of each basic operator (computational operator) in the original calculation graph. The architecture information of the target distributed cluster (also called cluster information) can include: cluster hardware device specifications, interconnection information between devices, storage capacity, geographical location of nodes, operating system and version, etc. Cluster information is the basis of distributed system modeling and the prerequisite for distributed systems to meet business needs and provide high-quality services.

[0058] Furthermore, the architecture information may also include the read and write bandwidth and bandwidth utilization of different storage locations, as well as the computing resource parallel mode of the target distributed cluster (for example, 2D and 1D parallelism), etc., which is not limited in this embodiment.

[0059] In this embodiment, operator cost calculation models corresponding to different operators can be pre-built through various modeling methods. Among them, different types of operators can correspond to different operator cost calculation models. For example, the Matmul basic operator used to implement matrix multiplication has a corresponding operator cost calculation model, and the All-Gather communication operator has a corresponding operator cost calculation model.

[0060] In an optional implementation of this embodiment, each operator cost calculation model takes one or more annotation information and one or more architecture information of the corresponding operator as input, and calculates the cost as output, where the output is also called the operator-level cost.

[0061] S140. Calculate the device-level cost corresponding to each single-device computation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain a cluster-level cost.

[0062] In this embodiment, by accumulating and summing the operator-level costs of each operator included in each single-device calculation graph, the device-level cost corresponding to each single-device calculation graph can be obtained. In addition, by accumulating and summing the device-level costs corresponding to each single-device calculation graph, the cluster-level cost corresponding to the target distributed cluster can be preliminarily obtained. Further, considering the distributed strategy defined in the target distributed parallel scheme, typically data parallelism, pipeline parallelism or expert parallelism, new cluster-level costs will also be introduced. Then, considering the device-level cost corresponding to each single-device calculation graph and the various distributed strategies defined in the target distributed parallel scheme, the cluster-level cost is calculated.

[0063] S150. Obtain system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution based on the operator-level cost, the device-level cost and the cluster-level cost.

[0064] In this embodiment, after calculating the operator-level cost of each operator in each single-device calculation graph, the device-level cost corresponding to each single-device calculation graph, and the cluster-level cost when the target large model is deployed in the target distributed cluster, different types of statistical indicator values ​​can be obtained according to multiple statistical dimensions, for example, the operator cost statistics of each target operator (typically, operators that the user is interested in or focuses on) obtained in each computing device, the operator cost statistics of each type of communication operator obtained in each computing device, etc.

[0065] Based on the obtained statistical indicator values, the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution can be obtained.

[0066] Through the above settings, the performance of the same target large model under distributed clusters of different hardware architectures can be compared horizontally, and then the architecture of the distributed cluster with the best performance for deploying the target large model can be explored. After the target large model is deployed on the distributed cluster with the best performance, the computing tasks matching the target large model can be executed efficiently and accurately, meeting the high-performance execution requirements of the target large model.

[0067] The technical solution of the embodiment of the present invention is to obtain the original calculation graph of the target large model and the target distributed parallel scheme used when deploying the target large model in the target distributed cluster; divide the original calculation graph according to the target distributed parallel scheme to obtain at least one single-device calculation graph for execution in a single computing device; call a pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-card calculation graph; calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy. , get the cluster-level cost; according to the operator-level cost, device-level cost and cluster-level cost, obtain the implementation method of the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme, and provide a new method for distributed system cost evaluation, which can be based on the target distributed parallel scheme used when a specific target large model is deployed in the target distributed cluster, and efficiently and accurately evaluate the operator-level cost, device-level cost and cluster-level cost in distributed modeling, and then obtain accurate system performance analysis results, so as to achieve a better balance in evaluation accuracy, evaluation time and evaluation cost in the process of system performance analysis.

[0068] Based on the above embodiments, obtaining a target distributed parallel solution used when deploying the target large model in the target distributed cluster may include:

[0069] Obtain model information of the target large model and cluster information of the distributed cluster; determine the search set corresponding to each distributed strategy based on the computing tasks and cluster information matching the target large model, and obtain multiple alternative distributed parallel schemes based on each search set; wherein each alternative distributed parallel scheme includes the number of parallels under at least one distributed strategy; generate a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device calculation time of each single-device calculation graph; calculate the end-to-end total time corresponding to each alternative distributed parallel scheme based on the single-device calculation time of each single-device calculation graph; determine the target distributed parallel scheme when deploying the target large model in the distributed cluster in each alternative distributed parallel scheme based on the calculated single-device calculation time and end-to-end total time.

[0070] In an embodiment of the present invention, the search set can be specifically understood as: a series of optional distributed parallel strategies, such as a tensor parallel search set, a pipeline parallel search set, a sequence parallel search set, an expert parallel search set, a context parallel search set or a data parallel search set. The search set contains all parallel strategies under the distributed strategy, and each parallel strategy has its corresponding allowed number of parallels. The search sets are orthogonalized to obtain multiple parallel combination schemes, namely, alternative distributed parallel schemes. Among them, orthogonal means that distributed parallel strategies are independently selected from each search set, and they are combined to form a larger distributed parallel search space. The distributed parallel search space contains all possible parallel strategy combinations, and these parallel strategy combinations constitute multiple alternative distributed parallel schemes, from which the optimal or approximately optimal parallel execution scheme can be found. Since the alternative distributed parallel schemes are obtained by orthogonalization of the search sets, each alternative distributed parallel scheme contains at least one parallel number set under a distributed strategy.

[0071] It is understandable that the above-mentioned alternative distributed parallel solutions may fully occupy all computing devices in the distributed cluster, or may only occupy part (for example, 50% or 80%) of the computing devices in the distributed cluster. Single-device computing time refers to the time required to complete a computing task on a single computing node (computing device), which can be used to evaluate system performance, optimize algorithms, and arrange task scheduling. The end-to-end total time can be specifically understood as: the total time from the beginning to the end of the computing process, including the time of all steps and stages. In a distributed system, computing tasks are usually decomposed and executed in parallel on multiple computing nodes, and the computing graph of each node may contain different tasks. By adding up the single-device computing time of all single-device computing graphs, and adding the time for data transmission, aggregation, and post-processing, the end-to-end total time of the entire distributed computing task can be obtained.

[0072] Specifically, through the model information of the target large model to be modeled in the distributed system (such as the structure, size and context length of the model) and the cluster information of the distributed cluster used to deploy the target large model (such as the cluster hardware device specifications and the interconnection information between devices), a distributed strategy search set (such as a tensor parallel strategy search set, a pipeline parallel strategy search set, a sequence parallel strategy search set, an expert parallel strategy search set, a context parallel strategy search set and a data parallel strategy search set) that meets the hardware performance requirements and is suitable for the target large model is determined, orthogonal operations are performed on each search set to obtain multiple alternative distributed parallel schemes, and a single-device calculation graph corresponding to each alternative distributed parallel scheme is constructed, the corresponding single-device calculation time is calculated and summed to obtain the end-to-end total time corresponding to each alternative distributed parallel scheme, and according to the calculated single-device calculation time and end-to-end total time, a distributed parallel scheme that meets the high performance conditions is screened out from each alternative distributed parallel scheme as the target distributed parallel scheme when the target large model is deployed in the distributed cluster, thereby realizing the determination of the distributed parallel scheme.

[0073] Among them, the above-mentioned high-performance conditions can be preset according to actual needs. For example, the total end-to-end time can be minimized, the performance of a single computing device can be optimal, or the number of total computing devices in a distributed cluster can be minimized, etc. This embodiment does not limit this.

[0074] According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, including at least one of the following:

[0075] Determine a tensor parallel search set corresponding to the tensor parallel according to the number of high-bandwidth interconnect devices supported by the distributed cluster in the cluster information;

[0076] According to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, a pipeline parallel search set corresponding to the pipeline parallelism is determined;

[0077] Extracting hardware information from cluster information, and determining the maximum number of context parallelisms based on the hardware information; determining context length information based on computing tasks matching the target large model; determining a context parallel search set corresponding to the context parallelism based on the maximum number of context parallelisms and the context length information;

[0078] According to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallel;

[0079] According to the total number of cluster devices in the cluster information, the determined tensor parallel search set, pipeline parallel search set and context parallel search set, the maximum number of data parallelism is calculated, and according to the maximum number of data parallelism, the data parallel search set corresponding to the data parallelism is determined.

[0080] Furthermore, generating a single-device computation graph of the target large model under each alternative distributed parallel solution may include:

[0081] Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed;

[0082] According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline;

[0083] According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated;

[0084] According to the parallel segmentation dimension of the input and output tensors of each computing operator in the single-device computing operator graph, the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived;

[0085] According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

[0086] Furthermore, calculating the single-device computing time of each single-device computing graph may include:

[0087] Get the target single device calculation graph currently being processed;

[0088] According to the cluster information of the distributed cluster, the first type of computing time of each computing operator and the second type of computing time of each communication operator in the target single device computing graph are calculated;

[0089] According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single-device computing time of the target single-device computing graph is calculated.

[0090] Further, according to the cluster information of the distributed cluster, the first type of computing time consumption of each computing operator in the target single-device computing graph is calculated, which may include:

[0091] Get the target computing operator currently being processed in the target single-device computing graph;

[0092] Calculate the input and output memory access time of the target computing operator based on the total input tensor size and total output tensor size of the target computing operator, as well as the device memory access bandwidth and memory access efficiency;

[0093] Calculate the computation time of the target computing operator based on the floating-point operations of the target computing operator, the number of computing operations per second of the device, and the computing efficiency of the device;

[0094] The first type of computing time of the target computing operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target computing operator, and the computing time of the target computing operator.

[0095] Further, according to the cluster information of the distributed cluster, the second type of computing time consumption of each communication operator in the target single-device computing graph is calculated, which may include:

[0096] Obtain the target communication operator currently being processed in the target single-device computation graph, and obtain the parallel type of the target communication operator;

[0097] If the parallel type is tensor parallel, the second type of computing time of the target communication operator is calculated according to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator;

[0098] If the parallel type is pipeline parallelism, the second type of computing time consumption of the target communication operator is calculated according to the communication volume of the pipeline parallel communication operator, the pipeline parallel communication bandwidth and the pipeline parallel communication efficiency.

[0099] Further, according to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time consumption and the second type of computing time consumption, the single device computing time consumption of the target single device computing graph is calculated, which may include:

[0100] If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth, and data parallel communication efficiency, and the single-device computing time of the target single-device computing graph is calculated based on the first-category computing time, the second-category computing time, and the gradient reduction time.

[0101] If the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device computing time of the target single-device computing graph is calculated based on the first type of computing time and the second type of computing time.

[0102] Based on the above embodiments, before the original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, the method may further include:

[0103] The original calculation graph is pre-processed according to the optimization strategy.

[0104] Based on the above embodiments, before obtaining the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution according to the operator-level cost, the device-level cost and the cluster-level cost, the following may also be included:

[0105] An optimization strategy post-processing is performed on at least one of the operator-level cost, the device-level cost, and the cluster-level cost.

[0106] Specifically, the optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy.

[0107] The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

[0108] Specifically, there are many algorithms and engineering optimization strategies and methods in the computing process of AI applications (for example, training, reasoning or fine-tuning), such as: operator fusion to improve operator computing efficiency, low-bit quantization to reduce data volume, kv-cache or GQA to improve reasoning efficiency, and Lora or QLora to reduce the amount and difficulty of fine-tuning (also known as fine-tuning). In order to model the impact of various optimization strategies on system costs under given hardware and cluster architectures, the architecture of optimization strategy pre-processing and optimization strategy post-processing is proposed in various embodiments of the present invention.

[0109] The optimization strategy pre-processing mainly adjusts the calculation graph before cost calculation, and the optimization strategy post-processing mainly further optimizes the costs at all levels. To achieve the relative independence of optimization processing and cost calculation, multiple pre- and post-processing can be superimposed on the cost calculation, which simplifies the modeling of multiple optimization strategies and makes it easy to verify and compare the impact of a certain optimization strategy on the system. These optimization strategies have different principles and complexity. Some optimization strategies only require pre-processing (such as operator fusion, kv-cache or GQA) or post-processing (such as various pipeline parallel optimizations), while others require both pre- and post-processing to work together (such as low-bit quantization).

[0110] Embodiment 2

[0111] Figure 2A flowchart of another cost evaluation method for a distributed system provided in Embodiment 2 of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, the operations of "slicing the original computation graph according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device" and "combining the device-level costs according to the distributed strategies to obtain the cluster-level cost" are concretized.

[0112] Correspondingly, such as Figure 2 As shown, the method may include:

[0113] S210, obtaining the original calculation graph of the target large model and the target distributed parallel solution used when deploying the target large model in the target distributed cluster.

[0114] The target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-marked with basic description information of the operators.

[0115] S220. Split the original computation graph according to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device.

[0116] In this embodiment, four segmentation semantics can be predefined, namely, no segmentation, segmentation by group, segmentation by dimension, and segmentation by value;

[0117] Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

[0118] Among them, not split (Not Split), denoted as N. The tensor is not split, which means that it is a complete copy on all computing devices. Group Split (Group Split), denoted as G(i), divides the tensor into different groups. The tensor is complete on all computing devices (also called computing cards) in the group to which it belongs, and does not exist on other groups. i indicates that the tensor belongs to the i-th group. Dim Split (Dim Split), denoted as D(i). The tensor is equally divided on all computing devices along the i-th dimension, and each computing device stores a tensor of size 1 / n (n is the total size of data on dimension i). Value Split (ValueSplit), denoted as V. The size of the tensor on each computing device card is the same as the original tensor size, but the sum of the tensor values ​​on all computing devices is equal to the original tensor.

[0119] Generally speaking, pipeline parallelism or expert parallelism involves dividing the basic operators contained in the original calculation graph into groups. At this time, it involves dividing the basic operators contained in the original calculation graph into different single-device calculation graphs. At this time, the original calculation graph can be split and processed according to at least one of the pipeline parallel number and expert parallel number in the target distributed parallel scheme.

[0120] In a specific example, if the original calculation graph contains four basic operators connected in sequence, and the number of pipeline parallelism is 2, the original calculation graph can be split into two single-device calculation graphs (in this case, they can also be called pipeline calculation graphs). The first single-device calculation graph contains the first two basic operators, and the second single-device calculation graph contains the last two basic operators.

[0121] S230. According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, mark the parallel split dimensions of the input and output tensors of each basic operator in the single-device calculation graph, and add communication operators in each single-device calculation graph based on the marking results.

[0122] Furthermore, the tensor parallelism, sequence parallelism and context parallelism involve splitting the input activation values ​​or model weights of the basic operators in the tensor dimension, and then the complete input and output tensors and model weights in a single-device calculation graph can be further divided into multiple single-device calculation graphs. It can be understood that the above-mentioned splitting method will not change the organization of the model operators contained in each single-device calculation graph, but will change the number of tensors contained in each single-device calculation graph.

[0123] That is, if the number of parallel tensors corresponding to a pipeline calculation graph obtained by S220 is 8, it is determined that the pipeline calculation graph needs to be executed in parallel on 8 computing devices, and each of the above computing devices corresponds to a single-device calculation graph with the same structure as the pipeline calculation graph. The difference is that the input and output tensors of each basic operator in each single-device calculation graph are cut into 1 / 8 of the original compared with the pipeline calculation graph.

[0124] In a specific example, if the current tensor (sequence) parallel number is determined to be 8, the current context parallel number is 1, the computational operator currently being processed is the matmul operator for implementing matrix multiplication, and it is pre-specified that the tensor (sequence) is split on the first dimension (starting from the 0th dimension) of the matmul operator. When the original input tensors of the matmul operator are A[b, s, h] and W1[h, h], it can be deduced that the parallel split dimension of the input tensor A of the matmul operator is [b, s / 8, h], and the parallel split dimension of the input tensor W1 is [h, h / 8]. Furthermore, combined with the semantics of the matmul operator, it can be deduced that the parallel split dimension of the output tensor B of the matmul operator is [b, s / 8, h / 8].

[0125] As mentioned above, after tensor parallelization of a computing operator, if the complete output tensor of the computing operator needs to be obtained, it is necessary to communicate the split output tensors on all computing devices where the computing operator is deployed, and then, it is necessary to add a matching communication operator after the computing operator. In various embodiments of the present invention, the inventor generates the communication operator insertion logic shown in Table 1 by analyzing the tensor conversion logic under different splitting strategies.

[0126] Table 1

[0127]

[0128] In a specific example, if it is determined that in the single-device computing operator graph, the parallel split dimension of the output tensor of the previous computing operator belongs to D(i), and the parallel split dimension of the input tensor of the next computing operator belongs to N, it means that an All-Gather communication operator needs to be inserted between the previous computing operator and the next computing operator.

[0129] S240. According to the annotation information of each operator in the single-device computing graph and the architecture information of the target distributed cluster, a pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-card computing graph.

[0130] In an optional implementation of this embodiment, the operator cost calculation model is constructed using at least one of the following methods:

[0131] Disassemble the large-grained operators to obtain the combination of ATen operators;

[0132] Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators;

[0133] Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results;

[0134] The operator cost calculation model includes: a basic operator cost calculation model and a communication operator cost calculation model.

[0135] S250. Calculate the device-level cost corresponding to each single-device computation graph according to the operator-level cost of each operator.

[0136] S260. Calculate the cluster communication cost between computing devices deployed with different single-device computing graphs based on at least one of the data parallel number, the pipeline parallel number, and the expert parallel number.

[0137] In this embodiment, considering that data parallelism is to distribute computing tasks to computing devices of different data parallel groups for parallel computing, data parallelism will not split the target large model, nor will it change the single device computing graph, but it will introduce new communication costs. Therefore, the cluster communication cost can be directly accumulated and calculated based on the number of data parallelism. Generally speaking, the larger the number of data parallelism, the higher the value of the cluster communication cost introduced. Therefore, a corresponding computing model can be constructed to calculate the cluster communication cost that matches the number of data parallelism.

[0138] Similarly, the number of pipeline parallelism and the number of expert parallelism can affect or determine the basic operators included in each single-device calculation graph, but will not introduce communication operators in the single-device calculation graph. The communication cost introduced can also be obtained by directly accumulating the cluster communication cost. Similarly, the larger the number of pipeline parallelism and the number of expert parallelism, the higher the value of the cluster communication cost introduced. Therefore, a corresponding calculation model can be constructed to calculate the cluster communication cost that matches the number of pipeline parallelism or the number of expert parallelism.

[0139] S270: Calculate a cluster-level cost according to each device-level cost and the cluster communication cost.

[0140] S280. According to the operator-level cost, the device-level cost, and the cluster-level cost, obtain the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution.

[0141] In an optional implementation of this embodiment, according to the operator-level cost, the device-level cost, and the cluster-level cost, a system performance analysis result is obtained when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme, including:

[0142] The operator-level cost of each operator, the IO (input and output) cost and computational cost of each operator in the forward calculation phase, the backward BPI (backward propagation input gradient) calculation phase, and the backward BPK (backward propagation kernel gradient) calculation phase, and the judgment result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results;

[0143] The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results;

[0144] The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

[0145] On the basis of the above embodiments, according to the operator-level cost, the device-level cost and the cluster-level cost, obtaining the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme may also include:

[0146] Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators;

[0147] The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

[0148] The target operator may be an operator that the modeling user is interested in or a pre-set core operator, etc. The computing power type may include: 2D or 1D, etc.

[0149] The technical solution of the embodiment of the present invention provides a new method for evaluating the cost of a distributed system. It can efficiently and accurately evaluate the operator-level cost, device-level cost and cluster-level cost in distributed modeling based on the target distributed parallel scheme used when a specific target large model is deployed in a target distributed cluster, and then obtain accurate system performance analysis results to meet people's growing demand for convenient and personalized cost evaluation of distributed systems.

[0150] On the basis of the above embodiments, after obtaining the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution according to the operator-level cost, the device-level cost and the cluster-level cost, the following may also be included:

[0151] In response to a target data item selected by a user in the system performance analysis result through a human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

[0152] Specific application scenarios

[0153] For easier understanding, the embodiments of the present invention are first described in the form of a specific application scenario. The solutions of the embodiments of the present invention can be executed by a cost evaluation platform. Figure 3 The schematic diagram shows the structure of a cost evaluation platform of a distributed system to which the present invention is applicable.

[0154] Specifically, Figure 3 As shown, the present invention breaks down the cost evaluation platform into four main parts, namely: operator cost modeling, optimization strategy processing (including optimization strategy preprocessing and optimization strategy postprocessing), Roofline Model (model) Cost calculation and system Cost analysis. The above four main parts are described below.

[0155] 1. The invention integrates the following inputs to model the operator cost

[0156] (1) aten operator decomposition: The benchmark construction system used for distributed system modeling supports abstract granularity scaling, and the operator granularity in the resulting distributed computing graph is generally larger than the kernel operator granularity during actual runtime (such as the pytorch aten operator). In each embodiment of the present invention, an operator decomposition tool is provided, which can export the pytorch module and subgraph to Fx Graph, decompose large-grained distributed operators (such as RoPE, flash-attention, etc.) into a combination of aten basic operators, and then form a cost model of large-grained operators based on the known cost of aten operators.

[0157] (2) Algorithm implementation: When building a forward-looking model, new operators without code implementation may be encountered. In response to this situation, the embodiments of the present invention provide a modeling programming interface that can easily build an operator cost model based on the algorithm logic.

[0158] (3) System measured data: Based on the calculation of the basic operator cost according to theoretical values ​​such as the amount of computation and the amount of data transported, the embodiments of the present invention further introduce the operator cost model of the system measured data correction theory. For example, the operator cost model is calibrated using the performance data such as computing power utilization, effective bandwidth, and kernel startup time measured on different platforms.

[0159] (4) Hardware simulation data: For hardware architectures that are still in the R&D stage, the theoretical model can use the performance data of the simulation platform to calibrate the operator cost.

[0160] (5) SoC (System on Chip) modeling results: In the early stage of product development, such as the architecture exploration stage, hardware, software and simulation platforms are not available. The basic operator cost model of the present invention can refer to the results of other SoC level modeling for performance calibration.

[0161] Accordingly, in Figure 4 A schematic diagram of the structure of a basic operator cost model applicable to an embodiment of the present invention is shown in FIG. Figure 4 As shown, the basic operator Cost model includes three parts: forward Cost, reverse Bpi Cost and reverse Bpk Cost. In this embodiment, forward and reverse cost modeling is performed for each basic operator, rather than modeling the forward and reverse operators separately, so that the cost calculation does not depend on the reverse graph, and it is more convenient to apply optimization strategies. For example, a configuration option can be used to control the parallel optimization of reverse Bpk cost and reverse communication without changing the reverse calculation graph.

[0162] The forward cost, reverse Bpi cost and reverse Bpk cost are composed of three parts: kernel startup time (that is, kernel startup cost), data transfer cost and calculation cost. Data transfer and calculation can be done in parallel, so: total cost = kernel startup cost + Max (data transfer cost, calculation cost), where Max (.) represents the maximum value operation.

[0163] The data transfer cost is the sum of the read and write costs of the input and output data. The amount of data is determined by the input and output Tensor shape, data type, and segmentation scheme, and the total transfer time is determined by the amount of data, the read and write bandwidth of the data storage location (such as L3 DeviceMemory, L4 Cxl memory), and the bandwidth utilization:

[0164] Data transfer time = L3 data volume / (L3 bandwidth * L3 bandwidth utilization) + L4 data volume / (L4 bandwidth * L4 bandwidth utilization)

[0165] This cost evaluation platform can also support the separate settings of read bandwidth and write bandwidth, as well as their respective utilization rates, to model the read-write imbalanced architecture.

[0166] The computational cost is divided into four types: 2D, 1D, SFU, and RNG, depending on the computing resources used. Simple basic operators have only one type of computational cost, while complex basic operators (such as Flash-Attention) may include multiple or all types of costs. This cost evaluation platform counts the costs of different computing resources separately, which can better compare the performance indicators of different architectures under different computing power ratios. For hardware platforms that support parallel operation of different computing resources (such as 2D / 1D parallel operation), parallel cost conversion can also be conveniently performed. In addition to the inherent computational cost, there may also be additional computational costs, such as the cost of scale and cast calculations added after the quantization solution is applied.

[0167] In a single-device computing graph, in addition to basic operators, communication operators are further included, wherein the communication operator modeling is described as follows.

[0168] Optionally, the communication volume of commonly used communication operators is shown in Table 2. (n represents the number of computing cards (that is, the number of computing devices), and T represents the original Tensor size)

[0169] Table 2

[0170]

[0171] Specifically, communication operator Cost = control delay + data volume / (interconnection bandwidth * bandwidth utilization)

[0172] Control delay and bandwidth utilization are determined by factors such as communication protocol (for example, RDMA, Infinband, or RoCE), network topology (for example, Full-Mesh or Ring), network hardware, and algorithm efficiency, and can be obtained through system measurement or simulation.

[0173] 2. Optimize strategy processing

[0174] Among them, Figure 5 A flow chart showing a method of adding optimization strategy pre-processing and optimization strategy post-processing to a cost evaluation process of a distributed system applicable to an embodiment of the present invention is shown in FIG.

[0175] like Figure 5 As shown in the figure, this cost evaluation platform supports modeling of various deep learning optimization strategies, such as operator fusion, low-bit quantization, MQA / GQA, and recalculation. The processing of optimization strategies is generally divided into two parts: (1) preprocessing the computational graph; (2) correcting and converting the cost.

[0176] Taking low-bit quantization as an example, through the computational graph control capability of the modeling system, the non-quantized computational graph imported by the benchmark can be quantized through a simple configuration file, including: (1) modifying the input and output Tensor types of the operators that need to be quantized; (2) setting quantization parameters and selecting the corresponding quantization cost model; (3) inserting quantization (or dequantization) operators or integrating quantization (or dequantization) calculations with other operators.

[0177] Specifically, for quantization-aware training (QAT), after completing the cost calculation of the entire model, it is also necessary to attach the calculation cost of quantizing the model parameters for each training iteration.

[0178] 3. Roofline Model Cost calculation, also known as system cost calculation

[0179] Among them, Figure 6 The overall flow chart of calculating the cluster-level cost applicable to the embodiment of the present invention is shown in FIG. Figure 7 It is a schematic diagram of a single-device computation graph generation process applicable to an embodiment of the present invention.

[0180] like Figure 6 and Figure 7 As shown in the figure, the system cost calculation includes two scheduling processes:

[0181] (1) Cluster scheduling: According to the input distribution scheme, the original computation graph is divided into different pipeline groups, data groups or expert groups according to the pipeline parallelism, data parallelism or expert parallelism logic to obtain a single-device computation graph executed on a single computing device.

[0182] (2) Operator scheduling for a single computing device: Based on the logical relationship between operators (including tensor parallel, sequence parallel, or context parallel communication operators) in the single-device computing graph, a logical ID is generated for each operator, and the cost of each operator is calculated in sequence according to the logical ID. Cost optimization can be performed as needed during the process. For example, different cost models can be selected for each operator cost calculation based on configurations such as quantization and fusion.

[0183] Among them, basic operators and communication operators with the same logic id (that is, no logic dependency) can be optimized in parallel. Finally, according to the cluster scheduling logic, the cost of a single computing device and the communication cost of pipeline parallelism, data parallelism or expert parallelism are combined to calculate the cost of the entire cluster.

[0184] 4. System Cost Analysis

[0185] Specifically, the embodiments of the present invention can provide the following system cost analysis capabilities:

[0186] (1) Low-cost comparative experiment capability. From the benchmark import method to the distributed search strategy and the modeling of the roofline cost model, the entire cost evaluation platform is rationally abstracted around distributed modeling, and can complete modeling calculations without distributed cluster hardware. The cost evaluation platform provides a highly customized and parallel computing framework. Using the control variable method, it can complete thousands of end-to-end cost calculations in minutes, and compare and analyze the impact of key parameter changes on system performance.

[0187] (2) Detailed data statistics function. The modeling system provides cost data statistics at three levels: operator, single computing device, and cluster:

[0188] a) Operator-level cost: Focus on the data IO cost and computing cost of the forward, reverse BPI, and reverse BPK parts of a single operator to determine whether the operator performance is IO-intensive or computing-intensive. When calculating the operator cost, the cost of different computing power types (2D, 1D, SFU, or RNG, etc.) and the kenerl startup cost are also counted.

[0189] b) Single-device level cost: traverse the single-device computation graph on a single computing device after distributed segmentation, combine the single operator cost and the communication cost according to the specified parallel and optimization strategy, and obtain the total cost on a computing device. While calculating the single-device cost, the following cost statistics are performed: different computing power costs (2D, 1D, SFU, RNG, etc.), the costs of core operators and user-focused operators (typically, dropout, linear, and flash-attention operators, etc.) and single-device level communication costs (typically, tensor parallelism or context parallelism, etc.)

[0190] c) Cluster-level cost: Combine the single-device cost according to the distributed parallel strategy, apply the optimization strategy, and get the cluster cost. At the same time, perform cluster performance statistics, including: different computing power costs (2D, 1D, SFU, and RNG, etc.), core operator and user-focused operator costs, and cluster-level communication costs (pipeline parallelism, expert parallelism, and data parallelism, etc.)

[0191] 3. Efficient data analysis and display functions.

[0192] a) The cost evaluation platform provides tools to summarize and analyze batch end-to-end costs, including: changes in core parameters (such as architecture, number of batches, number of sequences, model scale, and number of TP / CP / PP), system performance indicators (such as total time, throughput, TPS, effective computing power, and effective bandwidth), the proportion of various computing power costs (for example, 2D / 1D / SFU / RNG), the proportion of various communication costs (for example, TP / CP / PP / DP / EP), the cost proportion of IO-intensive operators and compute-intensive operators, and the cost proportion of operators that users are concerned about (for example, the cost proportion of operators quantized by FP8).

[0193] b) The cost evaluation platform provides an online data display tool that can filter data and draw various charts through a graphical interface to assist users in data analysis.

[0194] Embodiment 3

[0195] Figure 8 This is a schematic diagram of the structure of a cost evaluation device for a distributed system provided in Embodiment 3 of the present invention. Figure 8 As shown, the apparatus includes: a parallel solution acquisition module 810, a single device calculation graph acquisition module 820, an operator level cost acquisition module 830, a cluster level cost acquisition module 840 and a performance analysis result acquisition module 850, wherein:

[0196] The parallel scheme acquisition module 810 is used to acquire the original computation graph of the target large model and the target distributed parallel scheme used when the target large model is deployed in the target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-annotated with basic description information of the operators;

[0197] A single-device computation graph acquisition module 820 is used to segment the original computation graph according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator;

[0198] The operator-level cost acquisition module 830 is used to call a pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-card calculation graph;

[0199] The cluster-level cost acquisition module 840 is used to calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain the cluster-level cost;

[0200] The performance analysis result acquisition module 850 is used to obtain the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution based on the operator level cost, the device level cost and the cluster level cost.

[0201] The technical solution of the embodiment of the present invention is to obtain the original calculation graph of the target large model and the target distributed parallel scheme used when deploying the target large model in the target distributed cluster; divide the original calculation graph according to the target distributed parallel scheme to obtain at least one single-device calculation graph for execution in a single computing device; call a pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-card calculation graph; calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy. , get the cluster-level cost; according to the operator-level cost, device-level cost and cluster-level cost, obtain the implementation method of the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme, and provide a new method for distributed system cost evaluation, which can be based on the target distributed parallel scheme used when a specific target large model is deployed in the target distributed cluster, and efficiently and accurately evaluate the operator-level cost, device-level cost and cluster-level cost in distributed modeling, and then obtain accurate system performance analysis results, so as to achieve a better balance in evaluation accuracy, evaluation time and evaluation cost in the process of system performance analysis.

[0202] On the basis of the above embodiments, an optimization strategy pre-processing module may also be included, which is used to:

[0203] Before the original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, the original computation graph is pre-processed with an optimization strategy; and / or

[0204] It may also include an optimization strategy post-processing module for:

[0205] Before obtaining the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme based on the operator-level cost, the device-level cost and the cluster-level cost, at least one of the operator-level cost, the device-level cost and the cluster-level cost is post-processed with an optimization strategy.

[0206] On the basis of the above embodiments, the optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy;

[0207] The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

[0208] Based on the above embodiments, the parallel solution acquisition module 810 can be specifically used for:

[0209] Obtain model information of the target large model and cluster information of the distributed cluster;

[0210] According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, and according to each search set, multiple candidate distributed parallel schemes are obtained; wherein each candidate distributed parallel scheme contains the parallel number under at least one distributed strategy;

[0211] Generate a single-device calculation graph of the target large model under each alternative distributed parallel solution, and calculate the single-device calculation time of each single-device calculation graph;

[0212] According to the single-device computing time of each single-device computing graph, calculate the end-to-end total time corresponding to each alternative distributed parallel solution;

[0213] According to the calculated computing time of each single device and the end-to-end total computing time, a target distributed parallel scheme is determined from various alternative distributed parallel schemes when deploying the target large model in a distributed cluster.

[0214] Based on the above embodiments, the distributed strategy may include: pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism.

[0215] On the basis of the above embodiments, an operator cost calculation model construction module may also be included, which is used to construct the operator cost calculation model using at least one of the following methods:

[0216] Disassemble the large-grained operators to obtain the combination of ATen operators;

[0217] Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators;

[0218] Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results;

[0219] The operator cost calculation model includes: a basic operator cost calculation model and a communication operator cost calculation model.

[0220] Based on the above embodiments, the single-device calculation graph acquisition module 820 can be specifically used for:

[0221] According to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme, the original computation graph is split and processed to obtain at least one single-device computation graph for execution in a single computing device;

[0222] According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, the parallel splitting dimensions of the input and output tensors of each basic operator are marked in the single-device calculation graph, and communication operators are added to each single-device calculation graph based on the marking results.

[0223] Based on the above embodiments, the cluster-level cost acquisition module 840 can be specifically used for:

[0224] Calculate the cluster communication cost between computing devices deployed with different single-device computing graphs according to at least one of the data parallel number, the pipeline parallel number, and the expert parallel number;

[0225] The cluster-level cost is calculated based on the device-level cost and the cluster communication cost.

[0226] Based on the above embodiments, the performance analysis result acquisition module 850 can be specifically used for:

[0227] The operator-level cost of each operator, the IO cost and computation cost of each operator in the forward computation phase, the reverse BPI computation phase, and the reverse BPK computation phase, and the determination result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results;

[0228] The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results;

[0229] The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

[0230] Based on the above embodiments, the performance analysis result acquisition module 850 may be further specifically used for:

[0231] Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators;

[0232] The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

[0233] Based on the above embodiments, a visual display module may also be included, which is used for:

[0234] After obtaining the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme based on the operator-level cost, the device-level cost and the cluster-level cost, in response to the target data item selected by the user in the system performance analysis results through the human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

[0235] The cost evaluation device for a distributed system provided in an embodiment of the present invention can execute the cost evaluation method for a distributed system provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0236] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0237] Embodiment 4

[0238] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0239] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0240] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0241] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as executing a cost evaluation method for a distributed system as described in any embodiment of the present invention.

[0242] That is, obtain the original calculation graph of the target large model and the target distributed parallel scheme used when deploying the target large model in the target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original calculation graph is pre-annotated with basic description information of the operators; divide the original calculation graph according to the target distributed parallel scheme to obtain at least one single-device calculation graph for execution in a single computing device, wherein the single-device calculation graph includes communication operators; according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster, call the pre-built operator cost calculation model to determine the operator-level cost of each operator in the single-card calculation graph; according to the operator-level cost of each operator, calculate the device-level cost corresponding to each single-device calculation graph, and combine each device-level cost according to each distributed strategy to obtain the cluster-level cost; according to the operator-level cost, the device-level cost and the cluster-level cost, obtain the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme.

[0243] In some embodiments, a cost evaluation method for a distributed system as described in any embodiment of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the cost evaluation method for a distributed system as described in any embodiment of the present invention described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute a cost evaluation method for a distributed system as described in any embodiment of the present invention by any other appropriate means (for example, by means of firmware).

[0244] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0245] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0246] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0247] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0248] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0249] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0250] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0251] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cost evaluation method for a distributed system, characterized in that: include: Obtaining an original computation graph of a target large model and a target distributed parallel scheme used when deploying the target large model in a target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-marked with basic description information of operators; The original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator; According to the annotation information of each operator in the single-device computing graph and the architecture information of the target distributed cluster, the pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-device computing graph; According to the operator-level cost of each operator, the device-level cost corresponding to each single-device computation graph is calculated, and the device-level costs are combined according to each distributed strategy to obtain the cluster-level cost; According to the operator-level cost, device-level cost and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution; The system performance analysis results include: operator dimension performance analysis results, device dimension performance analysis results and system dimension performance analysis results.

2. The method according to claim 1, characterized in that Before the original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, the method further includes: Performing optimization strategy pre-processing on the original computation graph; and / or Before obtaining the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme according to the operator-level cost, the device-level cost and the cluster-level cost, it also includes: An optimization strategy post-processing is performed on at least one of the operator-level cost, the device-level cost, and the cluster-level cost.

3. The method according to claim 2, characterized in that The optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy; The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

4. The method according to claim 1, characterized in that: Get the target distributed parallel solution used when deploying the target large model on the target distributed cluster, including: Obtain model information of the target large model and cluster information of the distributed cluster; According to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, and according to each search set, multiple candidate distributed parallel schemes are obtained; wherein each candidate distributed parallel scheme contains the parallel number under at least one distributed strategy; Generate a single-device calculation graph of the target large model under each alternative distributed parallel solution, and calculate the single-device calculation time of each single-device calculation graph; According to the single-device computing time of each single-device computing graph, calculate the end-to-end total time corresponding to each alternative distributed parallel solution; According to the calculated computing time of each single device and the end-to-end total computing time, a target distributed parallel scheme is determined from various alternative distributed parallel schemes when deploying the target large model in a distributed cluster.

5. The method according to any one of claims 1 to 4, characterized in that: The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism.

6. The method according to any one of claims 1 to 4, characterized in that: The operator cost calculation model is constructed using at least one of the following methods: Disassemble the large-grained operators to obtain the combination of ATen operators; Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators; Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results; The operator cost calculation model includes: a basic operator cost calculation model and a communication operator cost calculation model.

7. The method according to claim 5, characterized in that The original computation graph is segmented according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, including: According to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme, the original computation graph is split and processed to obtain at least one single-device computation graph for execution in a single computing device; According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, the parallel splitting dimensions of the input and output tensors of each basic operator are marked in the single-device calculation graph, and communication operators are added to each single-device calculation graph based on the marking results.

8. The method according to claim 7, characterized in that Combine the device-level costs according to the distribution strategies to get the cluster-level costs, including: Calculate the cluster communication cost between computing devices deployed with different single-device computing graphs according to at least one of the data parallel number, the pipeline parallel number, and the expert parallel number; The cluster-level cost is calculated based on the device-level cost and the cluster communication cost.

9. The method according to claim 1, characterized in that: Based on the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution, including: The operator-level cost of each operator, the IO cost and computation cost of each operator in the forward computation phase, the reverse BPI computation phase, and the reverse BPK computation phase, and the determination result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results; The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results; The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

10. The method according to claim 9, characterized in that According to the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution, including: Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators; The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

11. The method according to claim 10, characterized in that: After obtaining the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme based on the operator-level cost, the device-level cost and the cluster-level cost, it also includes: In response to a target data item selected by a user in the system performance analysis result through a human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

12. A cost evaluation device for a distributed system, characterized in that: include: A parallel scheme acquisition module is used to acquire the original computation graph of the target large model and the target distributed parallel scheme used when the target large model is deployed in the target distributed cluster, wherein the target distributed parallel scheme includes the number of parallel operations under at least one distributed strategy, and the original computation graph is pre-annotated with basic description information of the operators; A single-device computation graph acquisition module, used to segment the original computation graph according to the target distributed parallel scheme to obtain at least one single-device computation graph for execution in a single computing device, wherein the single-device computation graph includes a communication operator; The operator-level cost acquisition module is used to call the pre-built operator cost calculation model according to the annotation information of each operator in the single-device calculation graph and the architecture information of the target distributed cluster to determine the operator-level cost of each operator in the single-device calculation graph; The cluster-level cost acquisition module is used to calculate the device-level cost corresponding to each single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain the cluster-level cost; A performance analysis result acquisition module is used to acquire the system performance analysis result when the target large model is deployed in the target distributed cluster according to the target distributed parallel scheme based on the operator-level cost, the device-level cost and the cluster-level cost; The system performance analysis results include: operator dimension performance analysis results, device dimension performance analysis results and system dimension performance analysis results.

13. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cost evaluation method for a distributed system according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cost evaluation method for a distributed system according to any one of claims 1 to 11 when executed.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the cost evaluation method for a distributed system according to any one of claims 1 to 11 is implemented.

Images