Distributed system modeling method, device, equipment, medium and program product

Abstract

The present invention discloses a distributed system modeling method, apparatus, equipment, medium and program product. The method comprises: obtaining benchmark description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model; determining a target distributed parallel scheme when deploying the target large model in a distributed cluster from multiple alternative distributed parallel schemes according to the benchmark description information and the cluster information; and obtaining system performance analysis results when deploying the target large model in a distributed cluster according to the target distributed parallel scheme from three dimensions of operator-level cost, device-level cost and cluster-level cost according to the target distributed parallel scheme. The technical solution of the embodiment of the present invention provides a standardized distributed modeling solution that meets various distributed deployment and performance evaluation requirements.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed system modeling method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC) and large language models (LLM), deep learning models are undergoing rapid iteration and updating, which is accompanied by the continuous expansion of model scale and the demand for larger clusters.

[0003] Therefore, how to effectively conduct distributed system modeling to cope with the ever-increasing demand for distributed deployment of large models is an important issue that needs to be solved. Summary of the invention

[0004] The embodiments of the present invention provide a distributed system modeling method, apparatus, device, medium and program product to provide a standardized distributed modeling solution to meet various distributed deployment and performance evaluation requirements.

[0005] According to one aspect of an embodiment of the present invention, a distributed system modeling method is provided, including:

[0006] Obtaining benchmark description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model;

[0007] The benchmark description information includes operator information of each model operator in the target large model, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of communication operators between each model operator;

[0008] According to the benchmark description information and the cluster information, a target distributed parallel scheme is determined from a plurality of candidate distributed parallel schemes when the target large model is deployed in the distributed cluster; wherein each candidate distributed parallel scheme includes a parallel number under at least one distributed strategy;

[0009] According to the target distributed parallel scheme, the system performance analysis results are obtained from three dimensions: operator-level cost, device-level cost, and cluster-level cost when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme.

[0010] According to another aspect of an embodiment of the present invention, there is provided a distributed system modeling device, comprising:

[0011] A basic information acquisition module, used to acquire benchmark test description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model;

[0012] The benchmark description information includes operator information of each model operator in the target large model, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of communication operators between each model operator;

[0013] A distributed scheme determination module is used to determine a target distributed parallel scheme when deploying a target large model in a distributed cluster from a plurality of candidate distributed parallel schemes according to the benchmark description information and the cluster information; wherein each candidate distributed parallel scheme includes a parallel number under at least one distributed strategy;

[0014] The system performance analysis result acquisition module is used to obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme from three dimensions: operator-level cost, device-level cost and cluster-level cost.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed system modeling method described in any embodiment of the present invention.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed system modeling method described in any embodiment of the present invention when executed.

[0020] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the distributed system modeling method as described in any embodiment of the present invention are implemented.

[0021] The technical solution of the embodiment of the present invention obtains benchmark description information of the target large model to be distributed modeled and cluster information of the distributed cluster used to deploy the target large model; determines the target distributed parallel scheme when the target large model is deployed in the distributed cluster from multiple alternative distributed parallel schemes according to the benchmark description information and the cluster information; obtains the technical means of the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme from three dimensions of operator-level cost, device-level cost and cluster-level cost according to the target distributed parallel scheme, decomposes the entire distributed modeling process into independent parts such as the construction of benchmark description information, the exploration of the target distributed parallel scheme and the evaluation of distributed parallel performance, realizes a standardized distributed modeling scheme, and then can realize the rapid introduction of benchmark tests, provide control over model scaling and independent calculation graphs, match appropriate distributed parallel schemes according to model characteristics, quickly evaluate the performance of different distributed parallel schemes, and evaluate the effects of different model optimization schemes.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a distributed system modeling method provided according to an embodiment of the present invention;

[0025] Figure 2 is a flowchart of an implementation method for obtaining benchmark test description information of a target large model to be distributed modeled according to an embodiment of the present invention;

[0026] Figure 3 is a flowchart of another implementation method for obtaining benchmark test description information of a target large model to be distributed modeled according to an embodiment of the present invention;

[0027] Figure 4 It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by group;

[0028] Figure 5It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by dimension;

[0029] Figure 6 It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by numerical value;

[0030] Figure 7 It is a flowchart of another implementation method of obtaining benchmark test description information of a target large model to be distributed modeled, provided by an embodiment of the present invention;

[0031] Figure 8 It is a schematic diagram of a computational graph applicable to an embodiment of the present invention and used to illustrate a derivation process of input and output tensor dimension information and distributed segmentation information of all model operators;

[0032] Fig. 9 It is a configuration diagram of a multi-dimensional distributed solution applicable to an embodiment of the present invention;

[0033] Fig.10 It is a schematic diagram of a computation graph obtained after distributed segmentation according to a multi-dimensional distributed solution applicable to an embodiment of the present invention;

[0034] Fig.11 It is a flowchart of an implementation method of determining a target distributed parallel solution when deploying a target large model in a distributed cluster among various alternative distributed parallel solutions provided by an embodiment of the present invention;

[0035] Fig.12 It is a flowchart of another implementation method of determining a target distributed parallel solution when deploying a target large model in a distributed cluster among various alternative distributed parallel solutions provided by an embodiment of the present invention;

[0036] Fig.13 It is a specific application scenario diagram applicable to the embodiment of the present invention;

[0037] Fig.14 is a schematic diagram of a model training storage content applicable to an embodiment of the present invention;

[0038] Fig.15 It is a schematic diagram of obtaining a reasonable distributed parallel solution applicable to an embodiment of the present invention;

[0039] Fig.16 It is a schematic diagram of constructing a single-card computing graph applicable to an embodiment of the present invention;

[0040] Fig.17 It is a flowchart of an implementation method for obtaining a system performance analysis result when a target large model is deployed in a distributed cluster according to a target distributed parallel solution provided in an embodiment of the present invention;

[0041] Fig.18 It is a flowchart of another implementation method of obtaining system performance analysis results when a target large model is deployed in a distributed cluster according to a target distributed parallel solution provided by an embodiment of the present invention;

[0042] Fig.19 It is a schematic diagram of the structure of a cost evaluation platform of a distributed system applicable to an embodiment of the present invention;

[0043] Fig. 20 It is a structural diagram of a basic operator cost model applicable to an embodiment of the present invention;

[0044] Fig.21 It is a flow chart of adding optimization strategy pre-processing and optimization strategy post-processing to the cost evaluation process of a distributed system applicable to an embodiment of the present invention;

[0045] Fig. 22 This is an overall flow chart for calculating the cluster-level cost applicable to the embodiment of the present invention;

[0046] Fig.23 It is a schematic diagram of a single-device calculation graph generation process applicable to an embodiment of the present invention;

[0047] Fig.24 is a structural diagram of a distributed system modeling device provided according to an embodiment of the present invention;

[0048] Fig.25 It is a structural schematic diagram of an electronic device for implementing the distributed system modeling method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] Figure 1 A flow chart of a distributed modeling method provided in an embodiment of the present invention. This embodiment can be applied to distributed modeling of a set target large model, exploring the most suitable target distributed parallel scheme when deploying the target large model in a set distributed cluster, and evaluating the distributed deployment effect of the target large model based on the explored target distributed parallel scheme. The method can be executed by a distributed modeling device, which can be implemented in the form of hardware and / or software and can generally be configured in a computer device with data processing capabilities.

[0052] Correspondingly, such as Figure 1 As shown, the method includes:

[0053] S110, obtaining benchmark description information of a target large model to be subjected to distributed modeling and cluster information of a distributed cluster for deploying the target large model.

[0054] In the embodiment of the present invention, the target large model can be specifically understood as: a model with a large number of parameters or a very long context for distributed system modeling. Among them, due to the large number of parameters, the structure of the model is often very complex. The very long context means that the text data that the model needs to process is very long, requiring the model to be able to process and parse large amounts of text data, which not only involves the storage and call of a large amount of historical information, but also relates to the accurate grasp of the overall coherence and deep meaning of the text. To this end, the model must rely on sufficient computing resources to support its efficient processing of large amounts of data during training and reasoning.

[0055] The target large model may be a deep learning model applied in the fields of image classification, target detection, face recognition, text classification, machine translation or speech recognition, etc. The target large model may be a set pre-trained model.

[0056] Among them, the benchmark description information includes the operator information of each model operator in the target large model, the input and output tensor dimension information, the distributed segmentation information and the candidate insertion position of the communication operator between each model operator.

[0057] The model operator can be understood as a computational operator used to implement the set operation logic, such as a square operator or a convolution operator, etc. The operator information may include: semantic information of the operator (typically, the operation logic of the operator), operator name, operator type, data type of the operator's input and output data, data format of the operator's input and output data, and storage location of the operator's input and output data (for example, L3 or L4), etc.

[0058] The input and output tensor dimension information of the model operator may specifically include the dimension information of the input tensor and the dimension information of the output tensor of the model operator.

[0059] The distributed splitting information of a model operator may include at least one piece of distributed strategy information. Each piece of distributed strategy information is used to describe what type of splitting can be performed on the model operator under what splitting parameters (for example, splitting on the first dimension of the input tensor) to implement what type of distributed strategy (or parallel method).

[0060] Among them, if it is necessary to execute a distributed strategy on a specific model operator in actual application, it may involve different copies of data, or parts of a piece of data are split and executed in different computing devices in a distributed cluster. Therefore, in order to obtain complete data processing results, it is necessary to insert a communication operator for realizing data aggregation function at a specific position between model operators. The types of communication operators may include: All-Reduce communication operator, Reduce-Scatter communication operator, All-Gather communication operator or point-to-point communication operator (P2P), etc.

[0061] In a specific example, if in the original computational graph, model operator A and model operator B are connected end to end, and if it is determined that the output tensor dimension of model operator A is split on the first dimension, while the input tensor of model operator B is not split, then a fully aggregated communication operator needs to be inserted between model operator A and model operator B. Furthermore, the position between model operator A and model operator B can be determined as a candidate insertion position for a communication operator.

[0062] In this embodiment, based on the benchmark description information of the target large model, distributed modeling of the target large model can be implemented efficiently and conveniently. That is, after the hardware architecture of the distributed cluster is uniquely determined, the distributed parallel solution with the best performance when the target large model is deployed in the distributed cluster can be explored based on the benchmark description information.

[0063] In the embodiment of the present invention, the distributed cluster can be specifically understood as: a group of computers (nodes) connected through a network, which work together to complete computing tasks to improve processing power, storage capacity and reliability, while reducing costs and improving computing efficiency. These computers can be physical machines or virtual machines, which share resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), memory, storage and network bandwidth, and can run distributed applications. Among them, GPU can also be called GPU card or card, which is a hardware accelerator for performing parallel computing, and is particularly suitable for large-scale matrix operations in deep learning. When the model is very large or the data to be processed is very large, such a large-scale cluster (such as a cluster of thousands of cards or tens of thousands of cards) is required to provide the required computing power. The cluster information of the distributed cluster can be specifically understood as: detailed data describing all nodes in the distributed system and their mutual relationships. The cluster information can specifically be: cluster hardware equipment specifications, interconnection information between devices, storage capacity, geographical location of nodes, operating system and version, etc. Cluster information is the basis for distributed system modeling and the premise that the distributed system can meet business needs and provide high-quality services.

[0064] In an optional implementation of this embodiment, the distributed cluster may include multiple cabinets, each cabinet includes multiple servers, and each server includes multiple computing cards (also referred to as computing devices). The above servers and computing cards are connected to each other using a set topological connection method (for example, a ring or a network) to establish a communication connection.

[0065] S120. Determine, from a plurality of candidate distributed parallel schemes, a target distributed parallel scheme for deploying the target large model in a distributed cluster according to the benchmark description information and the cluster information.

[0066] Each candidate distributed parallel scheme includes the number of parallel operations under at least one distributed strategy.

[0067] Among them, after the target large model is deployed in the distributed cluster, the model training task or model reasoning task can be executed accordingly. The above-mentioned model training task or model reasoning task can be collectively referred to as computing tasks.

[0068] The target distributed parallel scheme includes the number of parallelism under at least one distributed strategy. The distributed strategy may include: pipeline parallelism (PP), tensor parallelism (TP), sequence parallelism (SP), context parallelism (CP), data parallelism (DP) and expert parallelism (EP).

[0069] Accordingly, the target distributed parallel scheme may specifically include: one or more of the pipeline parallel number, tensor parallel number, sequence parallel number, context parallel number, data parallel number and expert parallel number.

[0070] In this embodiment, the target distributed parallel solution can be understood as a distributed parallel solution that can achieve optimal performance (for example, the least total time consumption, the least time consumption of a single computing device, or the least total computing devices used) after the target large model is deployed in the target distributed cluster.

[0071] S130. According to the target distributed parallel scheme, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme from three dimensions: operator-level cost, device-level cost, and cluster-level cost.

[0072] In this embodiment, after determining the target distributed parallel scheme, the target large model can be clearly deployed to the computing subtasks on each computing device in the distributed cluster, and then the standard single-device computing graph acting on each computing device in the distributed cluster can be clearly determined.

[0073] In this embodiment, the operator-level cost of each operator in each standard single-device calculation graph, the device-level cost corresponding to each standard single-device calculation graph, and the cluster-level cost when the target large model is deployed in the target distributed cluster can be calculated accordingly, and based on the above operator-level cost, device-level cost and cluster-level cost, different types of statistical indicator values ​​are obtained according to multiple statistical dimensions, for example, the operator cost statistics of each target operator (typically, operators that the user is interested in or focuses on) obtained in each computing device, the operator cost statistics of each type of communication operator obtained in each computing device, etc.

[0074] Based on the obtained statistical indicator values, the system performance analysis results when the target large model is deployed in the target distributed cluster according to the target distributed parallel solution can be obtained.

[0075] Through the above settings, the performance of the same target large model under distributed clusters of different hardware architectures can be compared horizontally, and then the architecture of the distributed cluster with the best performance for deploying the target large model can be explored. After the target large model is deployed on the distributed cluster with the best performance, the computing tasks matching the target large model can be executed efficiently and accurately, meeting the high-performance execution requirements of the target large model.

[0076] The technical solution of the embodiment of the present invention obtains benchmark description information of the target large model to be distributed modeled and cluster information of the distributed cluster used to deploy the target large model; determines the target distributed parallel scheme when the target large model is deployed in the distributed cluster from multiple alternative distributed parallel schemes according to the benchmark description information and the cluster information; obtains the technical means of the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme from three dimensions of operator-level cost, device-level cost and cluster-level cost according to the target distributed parallel scheme, decomposes the entire distributed modeling process into independent parts such as the construction of benchmark description information, the exploration of the target distributed parallel scheme and the evaluation of distributed parallel performance, realizes a standardized distributed modeling scheme, and then can realize the rapid introduction of benchmark tests, provide control over model scaling and independent calculation graphs, match appropriate distributed parallel schemes according to model characteristics, quickly evaluate the performance of different distributed parallel schemes, and evaluate the effects of different model optimization schemes.

[0077] Figure 2 This is a flowchart of an implementation method for obtaining benchmark test description information of a target large model to be distributed modeled in an embodiment of the present invention. This implementation method is refined based on the above-mentioned embodiments. In this implementation method, the operation of "obtaining benchmark test description information of a target large model to be distributed modeled" is specifically concretized.

[0078] Correspondingly, such as Figure 2 As shown, the above implementation may specifically include:

[0079] S210. According to the target source code file of the target large model, parse and obtain each model operator matching the target large model, and annotate the operator information of each model operator.

[0080] Generally speaking, developers can write source code files of large models based on the distributed deep learning framework on the market, and deploy the large models in a specific distributed cluster based on the source code files. The purpose of distributed system modeling is to effectively explore the best distributed solution before deploying a specific large model in a specific distributed cluster. If a benchmark suitable for distributed modeling is constructed efficiently and at low cost before subsequent distributed system modeling, the accuracy and convenience of subsequent distributed modeling can be improved.

[0081] Specifically, the target source code file is generally obtained by setting a programming language, for example, it is written based on the pytorch open source framework. Accordingly, the target source code file can be parsed using a parsing method that matches the programming language to obtain each model operator that matches the target large model. Of course, while obtaining the above-mentioned model operators, the connection relationship between the model operators can also be parsed, that is, by parsing the target source code file of the target large model, the original calculation graph that matches the target large model can be obtained. The original calculation graph contains the parsed model operators, and the model operators are interconnected by setting a connection relationship.

[0082] Furthermore, after determining each model operator that matches the target large model, in order to enable efficient subsequent distributed system modeling, in each embodiment of the present invention, detailed operator information is firstly annotated for each model operator in the target large model.

[0083] In this embodiment, the operator information of each model operator in the original calculation graph can be annotated in response to the annotation information manually input by the user, or the operator information of each model operator can be automatically annotated by automatically analyzing the pre-imported operator description document.

[0084] S220. According to the distributed configuration information matching the target large model, mark the distributed segmentation information of at least one target model operator among the multiple model operators.

[0085] In this embodiment, the distributed configuration information may include: activation value segmentation setting information, model parameter segmentation setting information, and collective communication setting information.

[0086] Among them, the activation value can be understood as the product of the input and the weight. In a large language model, the continuously updated input is usually called activations. Furthermore, the input flowing into each model operator can be understood as the activation value. Correspondingly, the activation value split setting information can be understood as the setting for distributed splitting of the input of one or more tensor dimensions of one or more model operators.

[0087] Model parameters are variables that control the behavior of machine learning models, generally including weights and biases. Generally speaking, model parameters are used to operate with the input of model operators to obtain new inputs to provide the next model operator. Accordingly, model parameter splitting setting information can be understood as the setting of distributed splitting of model parameters (typically, weights) of one or more tensor dimensions of one or more model operators.

[0088] The collective communication setting information can be understood as setting one or more distributed strategies for one or more model operators, wherein the distributed strategies may include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism, and expert parallelism.

[0089] The distributed configuration information matching the target large model may be input by a user through a configuration setting interface, or may be obtained by parsing a configuration file, which is not limited in this embodiment.

[0090] Based on the distributed configuration information matching the target large model, the distributed segmentation information of at least one target model operator in the target large model can be directly marked. The target model operator can be a model operator directly configured through the distributed configuration information, for example, a core model operator in the large model or a model operator of interest to the user.

[0091] S230. Based on the labeling results of the operator information and / or distributed split information, the input and output tensor dimension information and distributed split information of all model operators are derived, and based on the distributed split information of each model operator, the candidate insertion position of the communication operator between each model operator is determined.

[0092] As mentioned above, parsing the target source code file and obtaining the distributed configuration information matching the target large model for annotation may not cover all model operators. Therefore, based on the above annotation results, the embodiment of the present invention further adopts a set derivation strategy to derive the input and output tensor dimension information and distributed segmentation information of all model operators in the target large model.

[0093] Among them, if it is necessary to execute a distributed strategy on a specific model operator in actual application, it may involve different copies of data, or part of the content of a piece of data is split and executed in different computing devices in a distributed cluster. Therefore, in order to obtain complete data processing results, it is necessary to insert a communication operator for realizing data aggregation function at a specific position between model operators.

[0094] As mentioned above, by parsing the target source code file of the target large model, the original calculation graph matching the target large model can be obtained. Then, the derived input and output tensor dimension information and distributed segmentation information of each model operator can be marked at the matching model operator position in the original calculation graph. At the same time, the candidate insertion position of the determined communication operator between each model operator can also be marked in the original calculation graph.

[0095] S240. Generate a forward computation graph and a reverse computation graph as benchmark description information of the target large model based on the operator information, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of the communication operator between the model operators corresponding to each model operator.

[0096] In this embodiment, based on the original calculation graph, the operator information of each model operator in the determined original calculation graph, the input and output tensor dimension information and the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators that do not exist in the original calculation graph, a forward calculation graph and a reverse calculation graph matching the target large model can be generated as the benchmark description information of the target large model. Based on the benchmark description information of the target large model, the distributed system modeling of the target large model can be realized efficiently and conveniently, that is, after the hardware architecture of the distributed cluster is uniquely determined, the distributed parallel solution with the best performance when the target large model is deployed in the distributed cluster can be explored based on the benchmark description information.

[0097] The above implementation method, by parsing the target source code file of the target large model to be modeled for the distributed system, obtains each model operator matching the target large model, and annotates the operator information of each model operator; annotates the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model; derives the input and output tensor dimension information and distributed segmentation information of all model operators according to the annotated results of the operator information and / or the distributed segmentation information, and determines the candidate insertion position of the communication operator between each model operator according to the distributed segmentation information of each model operator; according to the operator information, input and output tensor dimension information, distributed segmentation information, and communication operator corresponding to each model operator, At the candidate insertion positions between the operators of each model, a forward calculation graph and a reverse calculation graph are generated as an implementation method of the benchmark description information of the target large model. Based on the source code file of the large model, the operator information, input and output tensor dimension information, distributed segmentation information of each model operator in the large model, and the candidate insertion positions of the communication operators between the operators of each model can be determined efficiently and at low cost, and then the benchmark description information of the target large model is generated. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box calculation graph control capabilities to provide support for the modeling of engineering optimization solutions such as calculation graph optimization, communication efficiency optimization, and low-bit quantization.

[0098] Figure 3 This is a flowchart of another implementation method for obtaining benchmark test description information of a target large model to be distributed modeled in an embodiment of the present invention. This implementation method is refined based on the above-mentioned embodiments. In this implementation method, the operation of "limiting the method for obtaining target source code files and distributed configuration information in the benchmark test description information of the target large model to be distributed modeled" is concretized.

[0099] Correspondingly, such as Figure 3 As shown, the above implementation may specifically include:

[0100] S310. Receive the original source code file corresponding to the target large model through a pytorch compatible interface.

[0101] The pytorch compatible interface is used to achieve full compatibility with the programming interface based on the python language.

[0102] In each embodiment of the present invention, a Benchmark generator front end is constructed to receive a target source code file, parse the target source code file of a target large model to be used for distributed system modeling to obtain each model operator matching the target large model, and annotate the operator information of each model operator, receive distributed configuration information matching the target large model, and annotate the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model.

[0103] At present, the distributed technology of large models is still in a period of rapid development and iteration. Various distributed deep learning frameworks emerge in an endless stream, and the distributed technologies and external interfaces supported by various frameworks vary greatly. The benchmark construction system needs to be compatible with different framework interfaces to quickly complete the modeling of existing models and distributed technologies, and provide more general interface expansion capabilities, so that new models and new technologies can be modeled forward-lookingly before the framework supports them. Given that the current mainstream distributed frameworks are all developed based on pytorch, the Benchmark generator front end provides full compatibility with the pytorch programming interface, and models composed of pytorch modules such as torch.nn.module or torch.nn.funtinal can be imported at low cost.

[0104] The above implementation method realizes the control of abstract granularity. That is, the pytorch compatible interface is used instead of the computational graph exported by the framework for modeling. This is mainly because the computational graph exported by the framework is usually for the purpose of compilation, optimization, and execution. For example, the exported onnx computational graph is for execution in the inference engine. Its structure is relatively complex and not suitable for human understanding and analysis, and it cannot express distributed segmentation. The use of the pytorch compatible interface can take into account both low-cost model import and flexible control of abstract granularity. For example, the modeling granularity can be controlled on core large operators such as flash-attention (Fast and Memory-EfficientExact Attention with IO-Awareness, a new attention algorithm with IO awareness and both fast and memory-efficient), rather than the dozens of small operators that are converted at runtime.

[0105] In addition, the above implementation can support distributed modeling. Among them, the distributed framework is in a period of rapid iteration, and there is no unified standard for interfaces, implementations, and even terminology. Therefore, combined with the compatible support for the relatively stable pytorch interface and the interface abstraction for distributed settings, it can take into account the low-cost import of model implementation and the maximum support for distributed technology modeling.

[0106] S320. According to preset modeling requirements, perform at least one of simplification, scaling and expansion operations on the original source code file to obtain a target source code file of a target large model.

[0107] Through the above settings, low-cost model import and expansion can be achieved. Among them, the framework exports the calculation graph usually depends on the execution process of the model, especially dynamic models such as MoE will change at runtime, and the execution process of large models requires a large-scale distributed system, and the cost of exporting the calculation graph is huge. The torch-compatible interface proposed in the present invention can be executed independently of hardware, and can facilitate the simplification, scaling and expansion of the model, and make it possible to model for forward-looking research.

[0108] S330. According to the target source code file of the target large model to be modeled for the distributed system, parse and obtain each model operator that matches the target large model, and annotate the operator information of each model operator.

[0109] S340. Receive distributed configuration information input by a user through a distributed configuration interface.

[0110] In an embodiment of the present invention, the Benchmark generator front end also provides a distributed configuration interface, which abstracts distributed settings such as model parameter segmentation, input activation value segmentation, and collective communication. Distributed implementations of different frameworks can be imported into benchmark modeling through this interface.

[0111] The distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information.

[0112] S350. According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is labeled with distributed segmentation information.

[0113] In an optional implementation of the embodiment of the present invention, the distributed segmentation information may include: at least one piece of distributed strategy information, each piece of distributed strategy information includes a distributed strategy, segmentation semantics, and segmentation parameters.

[0114] The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism. The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension and segmentation by value; the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or segmentation by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, segmentation by group or segmentation by value.

[0115] In each embodiment of the present invention, the inventor innovatively defines four types of tensor splitting semantics, namely, no splitting, splitting by group, splitting by dimension, and splitting by value. Among them, no splitting (Not Split), denoted by N. The tensor is not split, which means that it is a complete copy on all computing devices. Group Split (Group Split), denoted by G(i), divides the tensor into different groups. The tensor is complete on all computing devices (also called computing cards) in the group to which it belongs, and does not exist on other groups. i indicates that the tensor belongs to the i-th group. Dim Split (Dim Split), denoted by D(i). The tensor is equally divided on all computing devices along the i-th dimension, and each computing device stores a tensor of size 1 / n (n is the total size of data on dimension i). Value Split (Value Split), denoted by V. The size of the tensor on each computing device card is the same as the original tensor size, but the sum of the tensor values ​​on all computing devices is equal to the original tensor.

[0116] Taking the deployment of the simplest 2D tensor on a single computing device as an example, the specific implementation methods of group splitting, dimension splitting, and value splitting are described in detail. Figure 4 A segmentation schematic diagram when the segmentation semantics applicable to an embodiment of the present invention is segmentation by group is shown in FIG. Figure 5 A schematic diagram of segmentation when the segmentation semantics applicable to an embodiment of the present invention is segmentation by dimension is shown in FIG. Figure 6 A segmentation schematic diagram when the segmentation semantics applicable to an embodiment of the present invention is segmentation by numerical value is shown in FIG.

[0117] like Figure 4 As shown in , if the splitting strategy D(0) (also referred to as D0) is applied to a tensor of dimension 4x4, only one row of the tensor is deployed on each device from Device 0 to Device 3. Figure 5 As shown in the figure, if the splitting strategy G(1) (also referred to as G1) is applied to a tensor of dimension 3x3, then on Device 0 to Device 3, the tensor is deployed on only one of the devices ( Figure 5 In the example, it is deployed on Device 1, but not on other devices. Figure 6 As shown in the figure, if the splitting strategy V is applied to a tensor of dimension 3x3, the sum of the tensor values ​​on the dimension 4x4 is equal to the tensor value of the original tensor.

[0118] Based on the above four tensor segmentation semantics, the current six mainstream distributed strategies (also called 6D parallel strategies) can be expressed as one or several applications of the above four segmentation semantics.

[0119] Specifically, pipeline parallelism can apply segmentation G to divide model operators into groups of different pipelines.

[0120] Tensor parallelism, sequence parallelism, or context parallelism can be applied: 1. Split D, divide the activation value or weight tensor of the model operator equally according to a certain dimension within the TP group; 2. Split V, the reduce dimension of the model operator input is split, resulting in the output value containing only part of the complete output.

[0121] Data parallelism can be applied by splitting D(0) and splitting the batch dimension of the model operator input (activation value).

[0122] Expert parallelism can be applied: 1. Split D and split V, deploy the expert model on all computing devices in the expert group, and split the input activation value or weight tensor of the expert model in the dimension. If the reduce dimension is split, the output only contains part of the complete output; 2. Split G, deploy the expert model on different computing devices according to the group.

[0123] Correspondingly, each piece of distributed strategy information can be marked in the form of distributed strategy name_segmentation semantic name_segmentation parameter.

[0124] For example, a distributed strategy information of the form TP_D_1 indicates that the model operator uses tensor parallelism, and the parallel method is to split the first dimension of the weight tensor by dimension; or a distributed strategy information of the form PP_G_1 indicates that the model operator uses pipeline parallelism, and the parallel method is to divide the input and output tensors of the model operator into the first pipeline group.

[0125] In this embodiment, by parsing the input distributed configuration information, the distributed segmentation information can be annotated in each target model operator of the target large model. Specifically, the target model operator is generally a true subset of the target large model, and in special cases, it can be all model operators included in the target large model. The annotation of the distributed segmentation information performed in the target model operator can be the annotation of all 6D distributed strategies, or it can be the annotation of one or a few distributed strategies in 6D, and this embodiment does not limit this.

[0126] S360. Based on the labeling results of the operator information and / or distributed split information, derive the input and output tensor dimension information and distributed split information of all model operators, and determine the candidate insertion position of the communication operator between each model operator based on the distributed split information of each model operator.

[0127] S370. Generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model based on the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0128] In this embodiment, by using the pytorch compatible interface to achieve full compatibility with the python language-based pytorch AI framework programming interface, low-cost model import and flexible control of abstract granularity can be taken into account, hardware execution can be independent, and the model can be easily simplified, scaled and expanded, making it possible to model for forward-looking research. In addition, combined with compatible support for the relatively stable pytorch interface and interface abstraction for distributed settings, low-cost model implementation and maximum support for distributed technology modeling can be taken into account.

[0129] Figure 7 This is a flowchart of another implementation method of obtaining benchmark description information of a target large model to be distributed modeled in an embodiment of the present invention. This implementation method is optimized based on the above embodiments. In this implementation method, the operation of "deriving the input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of operator information and / or distributed segmentation information, and determining the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator" is concretized.

[0130] Correspondingly, such as Figure 7 As shown, the method specifically includes:

[0131] S710. According to the target source code file of the target large model to be modeled for the distributed system, parse and obtain each model operator that matches the target large model, and annotate the operator information of each model operator.

[0132] S720. According to the distributed configuration information matching the target large model, mark the distributed segmentation information of at least one target model operator among the multiple model operators.

[0133] S730. According to the operator semantics in the annotation results of the operator information and the tensor dimension of at least one input tensor defined in the computing task, derive the input and output tensor dimension information of all model operators.

[0134] Specifically, after the computing task of the target large model is determined, the tensor dimensions of one or more input tensors initially input to the target large model can be clarified. Taking the above input tensors as the starting point and combining the operator semantics of each model operator, the input and output tensor dimension information of each model operator in the target large model can be determined one by one.

[0135] S740. Derive distributed segmentation information of all model operators according to the labeling result of the distributed segmentation information of at least one target model operator.

[0136] In an optional implementation of this embodiment, according to the labeling result of the distributed segmentation information of at least one target model operator, deriving the distributed segmentation information of all model operators may include:

[0137] According to the semantic information of each target model operator and the annotation results of the distributed segmentation information of each target model operator, the first deduction rule is used to derive the completion result of the distributed segmentation information of each target model operator; according to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second deduction rule is used to derive the distributed segmentation information of the remaining model operators.

[0138] In a specific example, serial parallelism and tensor parallelism need to be used together. Therefore, after the distributed strategy information of serial parallelism is marked in the target model operator, the distributed strategy information of tensor parallelism of the target model operator can be completed.

[0139] After determining that a certain target model operator has completed the labeling of all distributed strategy information, the distributed segmentation information of other model operators before and after the target model operator can be further labeled according to preset rules. For example, if the previous unlabeled model operator of the target model operator has similar calculation logic to the target model operator, or belongs to the same type of operator, the distributed segmentation information of the target model operator can be directly used as the distributed segmentation information of the previous unlabeled model operator. Alternatively, the distributed segmentation strategies that each model operator can support can be pre-specified, and based on the distributed segmentation information of the target model operator, the distributed segmentation information of the previous unlabeled model operator can be effectively filtered out.

[0140] S750. According to the distributed segmentation information of each model operator, derive the segmentation semantics of the weight tensor of each model operator.

[0141] Generally speaking, some model operators need to calculate the matching input activation values ​​with the corresponding weight parameters. Therefore, if the input activation values ​​of a model operator are distributedly split, the weight parameters of the model operator also need to be distributedly split accordingly. Alternatively, the user can pre-enter the model parameter split setting information in the distributed configuration information to specify the split semantics of the weight tensor of which model operator or operators. Then, the split semantics of the weight tensor of each model operator can be finally derived.

[0142] S760. According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependencies between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one.

[0143] In this embodiment, different segmentation mapping relationships can be established for different types of model operators. In a specific example, Table 1 shows the segmentation mapping relationship between the input tensor, the weight tensor, and the output tensor based on the Matmul model operator.

[0144] Table 1

[0145]

[0146] Correspondingly, the figure shows a schematic diagram of a computational graph for illustrating the derivation process of distributed segmentation information of all model operators applicable to an embodiment of the present invention. Specifically, for each model operator, based on the segmentation description of the input tensor and combined with the computational semantics of the model operator, the segmentation of the output tensor of the model operator in the distributed system can be derived as shown in Table 1.

[0147] For the Matmul_1 model operator, if the input tensor is specified to be unsplit, and the weight tensor W_1 is specified by the user to be split by dim1 (D1), then according to the semantic output tensor B of the Matmul (matrix multiplication) model operator, it can be determined to be split by dim1 (D1); for the lower-level matmul_2 model operator, if the user specifies that the weight W_2 is split by dim0, then the output tensor C of the matmul_2 model operator can be determined to be split by value V; for the layernorm model operator, the user specifies that the input is split by dim0, which does not match the split of the matmul_2 output, then according to the communication operator, the logic V->D is inserted, and the ReduceScatter communication operator needs to be inserted (this communication operator is not in the original calculation graph, which will be described in detail later). According to the predetermined splitting mapping corresponding to the Layernorm model operator (not shown in the embodiment of the present invention), it can be deduced that the output tensor D is split according to dim0; the Gelu model operator has no user-specified splitting, and according to the input tensor D and the predetermined splitting mapping of the gelu model operator (not shown in the embodiment of the present invention), it can be deduced that the output tensor E is split according to dim0; the Add model operator has no user-specified splitting, and according to the input tensor E and the splitting mapping of the Add model operator, it can be deduced that the output tensor F is split according to dim0.

[0148] S770. Obtain multiple model operator dependency pairs according to the dependency relationship between the model operators.

[0149] Furthermore, in the original computation graph, two adjacent model operators form a model operator dependency pair. Figure 8 , Matmul_1 and Matmul_2, as well as Matmul_2 and Layernorm are all model operator dependency pairs.

[0150] S780. According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, obtain the target model operator dependency pair into which the communication operator is to be inserted, and determine the operator type of the communication operator to be inserted into the target model operator dependency pair.

[0151] S790. Determine the position between the previous model operator and the next model operator in each target model operator dependency pair as a candidate insertion position for the communication operator.

[0152] As mentioned above, after tensor parallelization of a computational operator, if the complete output tensor of the computational operator needs to be obtained, it is necessary to perform all-gather collective communication on the split output tensors on all computing devices where the computational operator is deployed, and then, it is necessary to add an all-gather communication operator after the computational operator. In various embodiments of the present invention, the inventor generates the communication operator insertion logic as shown in Table 2 by analyzing the tensor conversion logic under different splitting strategies.

[0153] Table 2

[0154]

[0155] In a specific example, combined with Table 2, if it is determined that in a model operator dependency pair, the parallel segmentation dimension of the output tensor of the previous model operator belongs to D(i), and the parallel segmentation dimension of the input tensor of the next model operator belongs to N, it means that an All-Gather communication operator needs to be inserted between the previous model operator and the next model operator.

[0156] S7100. Generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model based on the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0157] The technical solution of this embodiment can be based on the source code file of the large model, and can efficiently and cost-effectively determine the operator information, input and output tensor dimension information, distributed segmentation information and candidate insertion positions of communication operators between each model operator in the large model, and then generate the benchmark description information of the target large model. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box computational graph control capabilities to provide support for the modeling of engineering optimization solutions such as computational graph optimization, communication efficiency optimization, and low-bit quantization.

[0158] Multi-dimensional distributed solution configuration scenario

[0159] For the sake of understanding, after constructing the benchmark description information of the target large model, the method of implementing the multi-dimensional distributed solution configuration under a specific distributed cluster based on the benchmark description information is described in detail. Fig. 9 A schematic diagram of a multi-dimensional distributed solution applicable to an embodiment of the present invention is shown in FIG. Fig.10 A schematic diagram of a computational graph obtained after distributed segmentation according to a multi-dimensional distributed solution applicable to an embodiment of the present invention is shown in FIG.

[0160] Specifically, in actual task scenarios, different levels of computing device groups are usually used for distributed deployment. The operator distributed labeling, tensor segmentation description and derivation methods proposed in the embodiments of the present invention can be easily extended to multi-dimensional computing device groups.

[0161] For example, in Fig. 9 The left side shows the benchmark description information of the target large model, and the right side shows the hardware architecture of the distributed cluster to be deployed. Combined with the benchmark description information of the target large model, the distributed parallel solution with the best performance is determined to perform tensor parallelism and sequence parallelism between the computing devices in each server, pipeline parallelism between the servers in each cabinet, and data parallelism between multiple cabinets.

[0162] Among them, the benchmark test description information includes four model operators. According to the annotation of their respective distributed segmentation information and the grouping of computing devices, the following conclusions can be obtained.

[0163] 1. MatMul_1: According to the annotations TP_D_1, SP_D_1, and DP_D_0, it is determined that TP and SP are performed between the 8 computing devices in the server, and DP is performed between the two cabinets. The shape of the input activation tensor A is divided into [b / 2, s / 8, h], and the shape of the weight tensor W1 is divided into [h, h / 8]. According to the division of the matmul operator, it is deduced that the shape of its output tensor B is divided into [b / 2, s / 8, h / 8]. According to PP_G_0, the MatMul_1 model operator and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0164] MatMul_2: The shape of the input activation tensor B is [b / s, s / 8, h / 8], which satisfies the operator segmentation annotations TP_D_0, SP_D_2, and DP_D_0, so no communication operator is required between MatMul_1 and MatMul_2. The shape of the weight tensor W2 is segmented into [h / 8, h], and it can be deduced that the shape of the output tensor C is segmented into [b, s / 8, h(V)]. According to PP_G_0, the MatMul_2 model operators and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0165] Layernorm: According to the segmentation annotations SP_D_1 and TP_N, the input activation tensor needs to be segmented into [b, s / 8, h], but the tensor C is segmented into [b, s / 8, h(V)], and the communication operator reduce-scatter needs to be inserted. The output tensor D is segmented into [b, s / 8, h]. According to PP_G_0, the Layernorm model operators and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0166] Gelu: The distributed splits are labeled as SP_D1 and TP_N, and the input activation tensor D needs to be split into [b, s / 8, h], which can be satisfied, and there is no need to insert communication operators for TP / SP. However, according to PP_G_1, the Gelu model operators and input and output tensors are deployed on the second pipeline grouping, that is, on servers 4-7 in each cabinet, which is different from the deployment location of the first three model operators, so it is necessary to insert a P2P communication operator.

[0167] After completing the distributed derivation and device mapping, the distributed segmented computation graph is obtained as follows: Fig.10 As shown, combined Fig.10 The computational graph can efficiently and conveniently realize the deployment of the target large model on a distributed cluster of a specific hardware architecture, and then accurately and effectively evaluate the performance of the deployment result, such as the computing time of a single device or the end-to-end computing time.

[0168] Among them, according to the distributed segmentation and annotation information of key operators, the technical solution of the embodiment of the present invention can automatically generate a segmentation strategy for related tensors, and derive the distributed segmentation of the entire network and the location of inserting communication operators, and then obtain a benchmark description containing distributed segmentation strategies and communication operator information as input for searching for the optimal distributed parallel solution, so as to search for the best TP / SP / CP / DP / PP / EP combination.

[0169] Fig.11 This is a flowchart of an implementation method for determining a target distributed parallel scheme when deploying a target large model in a distributed cluster among various alternative distributed parallel schemes in an embodiment of the present invention. This embodiment is optimized based on the above embodiments, and specifically concretizes the operation of "determining a target distributed parallel scheme when deploying a target large model in a distributed cluster among multiple alternative distributed parallel schemes based on benchmark description information and cluster information."

[0170] Correspondingly, such as Fig.11 As shown, the method includes:

[0171] S1110. According to the benchmark description information, the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, and according to each search set, a plurality of alternative distributed parallel solutions are obtained.

[0172] Each candidate distributed parallel scheme includes the number of parallel operations under at least one distributed strategy.

[0173] In the embodiment of the present invention, the distributed strategy can be specifically understood as: a method of executing computing tasks simultaneously on multiple processors or multiple computer systems to improve processing speed and efficiency. The distributed strategy can specifically include: tensor parallel strategy, pipeline parallel strategy, sequence parallel strategy, expert parallel strategy, context parallel strategy, data parallel strategy and dynamic adjustment strategy.

[0174] Tensor parallelism refers to a parallel method that splits tensors (multidimensional data structures) among multiple processors. For example, in deep learning, the weights of a large neural network model can be distributed to different GPUs or processors (also called computing cards or computing devices). Each processor only stores and computes a portion of the tensor, and then updates it synchronously through communication operations. Pipeline parallelism refers to breaking down a computing task into a series of stages, each of which can be executed in parallel on different processors, similar to an assembly line in industrial production, where each stage completes its task and passes the result to the next stage. For example, in deep learning, different layers of a model can be processed in pipeline parallelism. Sequential parallelism refers to executing multiple parallel tasks in a specific order, where each task is completed before the next task can start. For example, different computing stages are executed sequentially on different processors. Expert parallelism refers to assigning different models or different parts of a model to different processors or machines. Each processor or machine is responsible for processing a part of the model and can be trained or inferred independently. Context parallelism refers to running the same algorithm or task in parallel in different execution contexts or environments, that is, running the same code on different processors, but each processor has its own data set or parameters. Context parallelism allows other processors to perform calculations while waiting for a task to complete, which can improve resource utilization. Data parallelism refers to splitting large data sets into small blocks and then processing these data blocks simultaneously on multiple processors. For example, in deep learning, models are trained in parallel on multiple GPUs, with each GPU processing different data samples but sharing the same model parameters. During training, each GPU synchronizes their model parameters to ensure that the model on all processors remains consistent. Dynamic adjustment strategy refers to adjusting computing resources and task allocation based on real-time feedback through adaptive algorithms.

[0175] In an embodiment of the present invention, the search set can be specifically understood as: a series of optional distributed parallel strategies, such as a tensor parallel search set, a pipeline parallel search set, a sequence parallel search set, an expert parallel search set, a context parallel search set or a data parallel search set. The search set contains all parallel strategies under the distributed strategy, and each parallel strategy has its corresponding allowed number of parallels. The search sets are orthogonalized to obtain multiple parallel combination schemes, namely, alternative distributed parallel schemes. Among them, orthogonal means that distributed parallel strategies are independently selected from each search set, and they are combined to form a larger distributed parallel search space. The distributed parallel search space contains all possible parallel strategy combinations, and these parallel strategy combinations constitute multiple alternative distributed parallel schemes, from which the optimal or approximately optimal parallel execution scheme can be found. Since the alternative distributed parallel schemes are obtained by orthogonalization of the search sets, each alternative distributed parallel scheme contains at least one parallel number set under a distributed strategy.

[0176] It is understandable that the above-mentioned alternative distributed parallel solutions may fully occupy all computing devices in the distributed cluster, or may only occupy part (for example, 50% or 80%) of the computing devices in the distributed cluster.

[0177] S1120: Generate a single-device computation graph of the target large model under each candidate distributed parallel solution according to the benchmark test description information, and calculate the single-device computation time of each single-device computation graph, wherein the single-device computation graph includes a communication operator.

[0178] In an embodiment of the present invention, a single-device computational graph can be specifically understood as: a computational graph constructed and executed on a single computing device. Among them, a computational graph is a graph representation method that describes a network structure and a computing process, and is composed of nodes (representing operations or calculations) and edges (representing data flows or tensors). Among them, a single-device computational graph contains communication operators. A communication operator refers to an operator that is specifically responsible for exchanging data between different computing nodes, ensuring that data processed in parallel on multiple computing devices can be correctly synchronized and aggregated. Single-device computing time refers to the time required to complete a computing task on a single computing node (computing device), which can be used to evaluate system performance, optimize algorithms, and arrange task scheduling.

[0179] S1130. Calculate the end-to-end total time corresponding to each candidate distributed parallel solution according to the single-device computing time of each single-device computing graph.

[0180] In the embodiment of the present invention, the end-to-end total time can be specifically understood as: the total time from the start to the end of the computing process, including the time of all steps and stages. In a distributed system, computing tasks are usually decomposed and executed in parallel on multiple computing nodes, and the computing graph of each node may contain different tasks. By adding up the single-device computing time of all single-device computing graphs, and adding the time for data transmission, aggregation, and post-processing, the end-to-end total time of the entire distributed computing task can be obtained.

[0181] S1140. According to the calculated calculation time of each single device and the end-to-end total time, determine a target distributed parallel scheme when deploying the target large model in a distributed cluster from among the alternative distributed parallel schemes.

[0182] Specifically, through the benchmark information of the target large model to be modeled for distributed system (including the operator information of each model operator in the target large model, the input and output tensor dimension information, the distributed segmentation information and the forward and reverse calculation graphs of the candidate insertion positions of the communication operator between the model operators), the cluster information of the distributed cluster used to deploy the target large model (such as the cluster hardware device specifications and the interconnection information between devices) and the computing tasks matched by the target large model, the distributed strategy search set (such as the tensor parallel strategy search set, the pipeline parallel strategy search set, the sequence parallel strategy search set, the dedicated parallel strategy search set) that meets the hardware performance requirements and is applicable to the target large model is determined. A plurality of candidate distributed parallel schemes are obtained by performing orthogonal operations on each search set, and a single-device calculation graph corresponding to each candidate distributed parallel scheme is constructed. The corresponding single-device calculation time is calculated and summed to obtain the end-to-end total time corresponding to each candidate distributed parallel scheme. According to the calculated single-device calculation time and the end-to-end total time, the distributed parallel scheme that meets the high performance conditions is screened out from each candidate distributed parallel scheme as the target distributed parallel scheme when the target large model is deployed in a distributed cluster, so as to determine the distributed parallel scheme.

[0183] Among them, the above-mentioned high-performance conditions can be preset according to actual needs. For example, the total end-to-end time can be minimized, the performance of a single computing device can be optimal, or the number of total computing devices in a distributed cluster can be minimized, etc. This embodiment does not limit this.

[0184] This implementation method automatically screens and decides on multiple candidate distributed parallel schemes based on the target large model benchmark test description information and cluster device information, avoiding time-consuming iterative testing of each candidate distributed parallel scheme one by one, reducing manpower costs and equipment resource consumption, and improving the efficiency of determining the target distributed parallel scheme.

[0185] In the above embodiments, the method may further include:

[0186] According to the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster, the weight-related storage requirements and activation-related storage requirements matching the target large model are calculated;

[0187] Accordingly, after obtaining multiple candidate distributed parallel solutions according to each search set, it also includes:

[0188] According to the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated respectively; among each alternative distributed parallel scheme, the invalid distributed parallel scheme whose single-device storage requirement is greater than the single-device standard storage capacity in the distributed cluster is filtered out.

[0189] The model information of the target large model can be specifically understood as: detailed data and parameters of a series of target large models related to distributed system modeling. The model information can specifically include model structure (such as the number of model layers, the type of each layer (such as convolutional layer, recurrent layer or fully connected layer, etc.) or activation function, etc.), model size (referring to the number of parameters), context length, computational complexity (such as the number of floating-point operations required for the model to perform a forward or backward propagation), model input data format and dimension or model output format and dimension.

[0190] In an embodiment of the present invention, the weight-related storage requirements can be specifically understood as: storage resources required to store and manage model parameters. Among them, model parameters may include: weights and biases, model parameters are learned during the training process, and are used in the reasoning and training of the model. Among them, for the model training task, the weight size, gradient size, and optimizer size are calculated and added to obtain the weight-related storage requirements. The weight size can be determined by multiplying the number of parameters by their data types to determine the memory space they occupy. For example, a parameter of type float32 occupies 4 bytes (32 bits), while a parameter of type int8 occupies only 1 byte. The gradient is the derivative of the loss function with respect to the weight and is used to update the weight during the training process. The gradient size is usually the same as the weight size because the gradient is also a floating point number and the number is the same as the weight. The optimizer is an algorithm used to update the model weights, and the weights are adjusted by calculating the gradient to improve the performance of the model. The optimizer not only stores the updated value of the weight, but may also store additional state information for each weight, such as additional states such as momentum and the square of the gradient. Among them, the optimizer size is related to the type of optimizer used and the number of additional states stored. For simple optimizers, such as SGD (Stochastic Gradient Descent), its size may be the same as the weight size. For complex optimizers, such as Adam (Adaptive Moment Estimation), its size may be about twice the weight size because of the need to store additional states. For model reasoning tasks (i.e., model inference or model execution), the model does not need to store gradients and optimizers, that is, the weight-related storage requirements are equal to the weight size.

[0191] In this embodiment, the activation-related storage requirement can be specifically understood as: in order to perform effective gradient calculation, weight update or model recovery, it is necessary to save the storage resources required for the data generated by some intermediate layers (activation layers) of the model in the memory. The activation-related storage requirement can be divided into activation storage size and temporary storage size. Among them, in the reverse propagation process of the model, in order to calculate the gradient of each layer, the activation value during forward propagation is required. Therefore, in order to perform reverse propagation, the activation value of each layer during forward propagation must be saved. The activation storage size refers to the sum of the activation value sizes during forward propagation that need to be saved for reverse calculation. The temporary storage size refers to the maximum possible size of all temporary activation values ​​that need to be saved in order to maintain the continuity of the calculation during the execution of the calculation graph. For example, at a certain point in the calculation graph, when multiple operators require a large number of intermediate activation values, the maximum value of memory usage may be reached. Among them, the intermediate activation value refers to each operator in the calculation graph that may generate temporary activation values ​​during execution, which may no longer be needed after the current operator completes the calculation, but may be used as input in subsequent operators in the graph. For model training tasks, the activation storage size is equal to the sum of the forward activation sizes that need to be saved for reverse calculations. The temporary storage size is the maximum activation size of all operators in the computation graph. For model inference tasks, the activation values ​​in the inference process are only used for the current forward calculation and no activation storage is required; during the model inference process, some operations may require intermediate results, but these results are no longer needed in subsequent calculations, so temporary variables can be stored in temporary storage and released after use. In addition, for model inference tasks, the KV cache (key value-cache) storage size can also be included. In some model architectures, such as the Transformer model (a self-attention mechanism model), KV cache can be used to improve efficiency during inference. The KV cache stores the key (Key) and value (Value) information in the self-attention mechanism, which is reused in each step of the generated sequence instead of being recalculated each time. By caching this information, the amount of calculation can be reduced and the inference speed can be accelerated, but it will also take up a certain amount of storage space.

[0192] In an optional implementation of this embodiment, in the model training stage, weight-related storage may specifically include: weight storage, gradient storage and optimizer storage, and activation-related storage may specifically include: activation storage and temporary storage; in the model inference stage, weight-related storage may only include weight storage, and activation-related storage may specifically include: activation storage (the amount of activation storage data is much smaller than that in the model training stage), temporary storage and KV cache storage.

[0193] In the embodiment of the present invention, the single-device storage requirement can be specifically understood as: the storage capacity required by a single computing device when executing a task, including but not limited to weight-related storage requirements and activation-related storage requirements.

[0194] Specifically, when designing a distributed solution, it is crucial to comprehensively consider the deployment scenario information on the distributed cluster and the model information of the target large model to calculate the single-device storage requirements and screen the alternative distributed solutions to achieve performance optimization, reasonable resource allocation, cost-effectiveness maximization, system reliability and fault tolerance. By calculating the weight-related storage requirements and activation-related storage requirements in the two task scenarios respectively, and according to each search set, a variety of alternative distributed parallel solutions are obtained, and then the single-device standard storage capacity in the distributed cluster and the single-device storage requirements corresponding to each alternative distributed parallel solution are evaluated respectively. By screening out invalid alternative distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster, not only the number of single-device calculation graphs corresponding to each alternative solution and the total time-consuming calculation amount are reduced, but also the time-consuming of the overall calculation task is reduced, and the efficiency of determining the distributed parallel solution is improved.

[0195] On the basis of the above embodiments, the single device storage requirements corresponding to each candidate distributed parallel solution are evaluated respectively according to the weight-related storage requirements and the activation-related storage requirements, which may include:

[0196] Obtain the current tensor parallel number, current pipeline parallel number, and current context parallel number in the target alternative distributed parallel scheme currently being processed; calculate the single-device weight-related storage requirement based on the weight-related storage requirement, the current tensor parallel number, and the current pipeline parallel number; calculate the single-device activation-related storage requirement based on the activation-related storage requirement, the current tensor parallel number, and the current context parallel number; calculate the single-device storage requirement corresponding to the target alternative distributed parallel scheme based on the single-device weight-related storage requirement and the single-device activation-related storage requirement.

[0197] Specifically, in the process of respectively evaluating the single-device standard storage capacity in the distributed cluster and the single-device storage requirements corresponding to each alternative distributed parallel scheme, the calculation method of the single-device storage requirement corresponding to the target alternative distributed parallel scheme (referring to the alternative distributed parallel scheme to be evaluated) can be: first, obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target alternative distributed parallel scheme currently being processed, and then calculate the single-device weight-related storage requirement based on the weight-related storage requirement, the current tensor parallel number, and the current pipeline parallel number, that is, the single-device weight-related storage requirement is calculated in the storage module required by the model. The weight-related storage requirement is divided by the product of the current tensor parallelism number and the current pipeline parallelism number, that is, the weight-related storage requirement of a single device = weight-related storage requirement / (current tensor parallelism number × current pipeline parallelism number); according to the activation-related storage requirement, the current tensor parallelism number and the current context parallelism number, the activation-related storage requirement of a single device is calculated, that is, the activation-related storage requirement of a single device is the activation-related storage requirement calculated in the storage module required by the model divided by the product of the current tensor parallelism number and the current context parallelism number, that is, the activation-related storage requirement of a single device = activation-related storage requirement / (current tensor parallelism number × current context parallelism number).

[0198] Among them, the tensor parallel number, pipeline parallel number, and context parallel number can all be obtained from the computing tasks and cluster information that match the target large model. In the process of evaluating the alternative distributed parallel solutions, the current tensor parallel number, the current pipeline parallel number, and the current context parallel number can be directly obtained. Finally, based on the single-device weight-related storage requirements and the single-device activation-related storage requirements, the single-device storage requirements corresponding to the target alternative distributed parallel solution are calculated and used for comparison with the single-device standard storage capacity in the distributed cluster, and invalid alternative distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity are screened out.

[0199] Optionally, a method for determining a distributed parallel solution, the distributed strategies may include: pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism, and expert parallelism.

[0200] Specifically, pipeline parallelism improves resource utilization through pipeline processing; tensor parallelism parallelizes computationally intensive layers by splitting tensors; sequence parallelism is suitable for application scenarios that process long sequence data, such as time series or long texts; context parallelism allows different tasks or model instances to run independently on different devices; data parallelism is suitable for large data set application scenarios, and accelerates training by replicating models on multiple devices; expert parallelism distributes different experts to different devices in hybrid expert models to improve the sparsity and efficiency of the model. By comprehensively utilizing multiple parallel technologies such as pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism, and expert parallelism, each technology can play its unique advantages in specific application scenarios, thereby improving the determination rate of distributed parallel solutions, optimizing resource utilization, enhancing the scalability and flexibility of distributed parallel solutions, and improving the reliability of distributed parallel solutions.

[0201] In an optional implementation of this embodiment, the alternative distributed parallel scheme may specifically include: the number of tensor parallelism, the number of sequence parallelism, the number of context parallelism, the number of data parallelism, and the number of expert parallelism.

[0202] Optionally, according to the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, including at least one of the following:

[0203] According to the number of high-bandwidth interconnected devices (which can also be understood as computing devices that meet high-bandwidth interconnection requirements) supported by the distributed cluster in the cluster information, determine the tensor parallel search set corresponding to the tensor parallelism; according to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, determine the pipeline parallel search set corresponding to the pipeline parallelism; extract the hardware information in the cluster information, and determine the maximum number of context parallelism based on the hardware information; according to the computing tasks matching the target large model, determine the context length information; according to the maximum number of context parallelism and the context length information, determine the context parallel search set corresponding to the context parallelism; according to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallelism; according to the total number of cluster devices in the cluster information, the determined tensor parallel search set, pipeline parallel search set and context parallel search set, calculate the maximum number of data parallelism, and according to the maximum number of data parallelism, determine the data parallel search set corresponding to the data parallelism.

[0204] Specifically, since the search set is orthogonally obtained from various search sets (including tensor parallel search set, pipeline parallel search set, sequence parallel search set, expert parallel search set, context parallel search set and data parallel search set), the search set includes at least one of the above sets.

[0205] Among them, the tensor parallel search set can be defined as a set of powers of 2 in the range of 1 to the number of high-bandwidth interconnection cards (interface cards that provide high-speed data transmission between high-performance computing devices) according to the number of high-bandwidth interconnection devices supported by the distributed cluster. For example, if the number of high-bandwidth interconnection devices is 32, the tensor parallel search set is 1, 2, 4, 8, 16, 32. Alternatively, if the number of high-bandwidth interconnection devices is small, the set of all integers in the range of 1 to the number of high-bandwidth interconnection devices can be defined as the tensor parallel search set. For example, if the number of high-bandwidth interconnection devices is 8, the tensor parallel search set is 1, 2, 3, 4, 5, 6, 7, 8.

[0206] The pipeline parallel search set can be an integer starting from 1, and supports setting some prior restrictions, including but not limited to setting the maximum value of pipeline parallelism or setting the number of pipeline parallelism to be divisible by the number of model layers. For example, when the number of model layers is 96, if the maximum value of pipeline parallelism is set to 8 and the number of pipeline parallelism is set to be divisible by the number of model layers, the pipeline parallel search set can be 1, 2, 4, 6, 8.

[0207] Sequence parallelism needs to be used together with tensor parallelism and the number of sequence parallelism is equal to the number of tensor parallelism, so there is no need to define a sequence parallel search set.

[0208] The context parallel search set can be defined as an integer in the range of 1 to the maximum context parallel number. Specifically, the context parallel search set can be: a set including 1 and the maximum context parallel number, and satisfying the restriction that the context parallel number in the search set is divisible by the context size (that is, context length information). The context length information is the context length of the computing task input into the target large model.

[0209] There are two ways of expert parallelism. One is that all experts are distributed on different devices, and the other is that some of all experts are deployed on the same device. If all experts are distributed on different devices, the number of experts in parallel is equal to the total number of model experts; if some of all experts are deployed on the same device, the number of experts in parallel is equal to 1. Among them, experts can be understood as specialized sub-models, which can be an independent neural network, usually a feedforward network, or a more complex network structure. Each expert can generally participate in solving problems in the field of his expertise through a "gated network".

[0210] The maximum number of data parallelism can be calculated based on the total number of cluster devices. The maximum number of data parallelism is equal to the total number of cluster devices divided by the number of devices in one data parallel group. Among them, the number of devices in one data parallel group is equal to the product of the number of tensor parallelism, the number of pipeline parallelism, and the number of context parallelism. The search set of data parallelism is an integer (integer or power of 2) in the range from 1 to the maximum number of data parallelism, which can include 1 and the maximum number of data parallelism.

[0211] Fig.12 This is a flowchart of another implementation method of determining the target distributed parallel scheme when deploying the target large model in a distributed cluster among various alternative distributed parallel schemes in an embodiment of the present invention. This embodiment is a refinement of the single-device calculation graph for generating the target large model under each alternative distributed parallel scheme in the above embodiment.

[0212] Correspondingly, such as Fig.12 As shown, the method includes:

[0213] S1210. Determine a search set corresponding to each distributed strategy based on the benchmark description information, computing tasks matching the target large model, and cluster information, and obtain multiple candidate distributed parallel solutions based on each search set.

[0214] Among them, the distributed strategies required to determine the search set can be determined according to the distributed segmentation information defined in the benchmark description information. For example, in the distributed segmentation information, only pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism and data parallelism are defined, and then the search sets corresponding to pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism and data parallelism can be determined accordingly.

[0215] In this embodiment, after determining the search set corresponding to each distributed strategy and the relationship between the parallel numbers under different distributed strategies, all alternative distributed parallel schemes can be determined by permutations and combinations, and each alternative distributed parallel scheme includes the parallel number under each distributed strategy.

[0216] S1220, obtaining the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel solution currently being processed.

[0217] S1230. According to the current number of parallel pipelines and a preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline.

[0218] In the embodiment of the present invention, the load distribution strategy can be specifically understood as: a method of allocating tasks to different computing nodes in the system to optimize resource utilization, improve system performance, and ensure efficient completion of tasks. An effective load distribution strategy can balance the workload of each node, avoid overloading some nodes while other nodes are idle, thereby improving overall computing efficiency and throughput. For example, polling and random allocation, which are easy to implement but may not consider the actual load of the node, resulting in uneven load; least connection and least load strategies, which can allocate tasks according to the current load of the node to avoid overload; weighted polling and hash allocation, which ensure load balancing and session consistency by considering the weight of the node or the key attributes of the task.

[0219] Typically, the load distribution strategy may be a load balancing strategy that evenly distributes each model operator (also referred to as a computational operator) in the original computation graph to each pipeline that matches the number of parallel operations of the current pipeline.

[0220] In a specific example, if the original computation graph contains 10 sequentially connected model operators and the current pipeline parallel number is 2, the first 5 model operators in the original computation graph can be divided into a pipeline computation graph on one pipeline, and the last 5 model operators in the original computation graph can be divided into a pipeline computation graph on another pipeline.

[0221] On the basis of the above embodiments, considering that expert parallelism can enable experts who are "good at" the set field to perform the set type of calculation, the number of expert parallelism can be further combined to update the pipeline calculation graph on each pipeline. Alternatively, considering that the actual impact of the number of expert parallelism on the pipeline calculation graph is small, in order to simplify the calculation, the impact of the number of expert parallelism on the pipeline calculation graph can also be ignored.

[0222] S1240. Calculate a single-device computing operator graph based on the pipeline computing graph, the current tensor parallel number, and the current context parallel number.

[0223] In the embodiment of the present invention, the single-device computing operator graph can be specifically understood as: a computing graph constructed and executed on a single computing device, in which the computing operators (ie, model operators) contained therein are mainly used for tensor calculations.

[0224] In this embodiment, considering that sequence parallelism is used together with tensor parallelism, and the number of sequence parallelism is the same as the number of tensor parallelism, the number of sequence parallelism is no longer introduced here. Specifically, tensor (sequence) parallelism and context parallelism are mainly used to realize the splitting of activations or weights in the tensor dimension. This operation reduces the size of the input and output tensors of the model operator. Only local calculation results will be calculated on each single device, thereby reducing the amount of calculation on a single device. However, these local calculation results need to be transmitted and aggregated between multiple devices to obtain the final complete calculation result. Therefore, it is necessary to introduce a communication operator in the obtained single-device computing operator graph.

[0225] In this embodiment, the calculated single-device computing operator graph includes the input and output tensor sizes of each computing operator.

[0226] In this embodiment, the parallel splitting dimension of the input and output tensors of each model operator in the single-device model operator graph can be derived by combining the tensor size of the input and output tensors of each model operator in the original computation graph (which can be obtained through prior knowledge or derived based on its actual input tensor), the operator semantics of each model operator, and the pre-annotated splitting description information.

[0227] In a specific example, if the current tensor (sequence) parallel number is determined to be 8, the current context parallel number is 1, and the model operator currently being processed is the matmul operator for implementing matrix multiplication, and it is pre-specified that the tensor (sequence) is split on the first dimension (starting from the 0th dimension) of the matmul operator. When the original input tensors of the matmul operator are A[b, s, h] and W1[h, h], it can be deduced that the parallel split dimension of the input tensor A of the matmul operator is [b, s / 8, h], and the parallel split dimension of the input tensor W1 is [h, h / 8]. Furthermore, combined with the semantics of the matmul operator, it can be deduced that the parallel split dimension of the output tensor B of the matmul operator is [b, s / 8, h / 8].

[0228] It should be noted that data parallelism is equivalent to assigning a complete computing task to computing devices in different data parallel groups for parallel computing. It does not split the target large model and will not change the single-card computing graph. However, it will introduce communication time between different computing devices. Therefore, when calculating the single-device computing operator graph, the current number of data parallelism is not taken into account.

[0229] S1250. Determine the parallel splitting dimension of the input and output tensor of each model operator in the single-device computing operator graph according to the input and output tensor dimension information of each model operator contained in the benchmark description information and the candidate insertion position of the communication operator between each model operator, and derive the communication operator that needs to be inserted in the single-device computing operator graph and the insertion position of the communication operator.

[0230] Specifically, a single-device computation graph refers to a computational process built on a computing device, which includes operators that perform computational tasks and possible communication operations. In such a computation graph, computational operators and communication operators work together to complete the forward and backward propagation of the model. After obtaining the model information of the target large model to be modeled in the distributed system, and the cluster information of the distributed cluster used to deploy the target large model, determining the search set corresponding to each distributed strategy, and obtaining multiple alternative distributed parallel schemes, obtain the current tensor parallel number, current pipeline parallel number, and current context parallel number in the target alternative distributed parallel scheme currently being processed, and first analyze the original large model computation graph, understand the structure of the model, data flow, and computational requirements, and identify the nodes (such as convolutional layers, fully connected layers, and activation functions, etc.) and edges (data flows) in the graph. Then, according to the current number of pipeline parallelism and the preset load distribution strategy, the original calculation graph is decomposed into multiple pipeline stages, each stage executes a part of the computing tasks, and the uniform distribution of the model layers or computing tasks is achieved. The point-to-point communication operators for sending and receiving are added to convert the original calculation graph into a calculation graph on the pipeline to achieve parallel execution of multiple operations and improve throughput and resource utilization. Then, according to the pipeline calculation graph, the current number of tensor parallelism, and the current number of context parallelism, the original pipeline calculation graph is analyzed to understand the structure and data flow of the model, and the parallel execution strategy of the model is determined according to the number of context parallelism and tensor parallelism. The pipeline calculation graph is integrated into parts suitable for execution on a single device, and optimizations such as operator fusion and constant folding are performed to improve execution efficiency. The optimized calculation graph is mapped to a single device, and a single-device computing operator graph optimized for the device is generated. According to this operator graph, the computing tasks are scheduled and executed on the device. In this embodiment, based on the input and output tensor dimension information of each model operator contained in the benchmark description information and the target alternative distributed parallel scheme currently being processed, the parallel splitting dimension of the input and output tensor of each model operator in the single-device model operator graph can be determined.

[0231] Specifically, according to the parallel partitioning dimension of the input and output tensors of the model operator, the partitioning method of the input and output tensors of each model operator can be determined, for example, whether it is partitioned by row, column or block, understand the role of each operator in tensor parallelism, and how it operates the partitioned tensors, and check whether the input and output tensor layouts between adjacent operators match. If the output tensor layout of the previous operator is inconsistent with the input tensor layout of the next operator, then it is necessary to insert a communication operator to rearrange the tensors. According to the partitioning method of the tensor and the layout compatibility between operators, determine which communication operators are needed to synchronize or reorganize the data, insert the communication operator between operators with incompatible layouts to ensure that the data is correctly synchronized or reorganized before continuing processing, and realize automatic derivation of the inserted tensor parallel communication operator type and the insertion position of the communication operator, such as the full protocol communication operator, the protocol distribution communication operator and the full aggregation communication operator, and insert the communication operator into the single-device computing operator graph to obtain a single-device computing graph.

[0232] As mentioned above, since the candidate insertion positions of the communication operator between each model operator have been defined in the benchmark description information, after mapping the above candidate insertion positions to each single-device calculation graph, by combining the above-mentioned inspection of the parallel splitting dimensions of the output tensor of the previous model operator of each candidate insertion position, and the parallel splitting dimensions of the input tensor of the next model operator, it is possible to determine which of the above candidate insertion positions are the insertion positions where the communication algorithm needs to be inserted, as well as the operator type of the communication operator that needs to be inserted at each of the above insertion positions.

[0233] S1260. Insert the communication operator into the single-device computing operator graph according to the insertion position to obtain a single-device computing graph under the target candidate distributed parallel solution.

[0234] Through the above settings, a single-device computing graph including a communication operator can be obtained. The communication operator may include: a communication operator for realizing pipeline parallelism or a communication operator for realizing tensor parallelism. Furthermore, the communication operator may also include a communication operator for realizing expert parallelism, etc.

[0235] S1270. Calculate the single-device computing time of each single-device computing graph, and calculate the end-to-end total time corresponding to each alternative distributed parallel solution based on the single-device computing time of each single-device computing graph.

[0236] S1280. According to the calculated calculation time of each single device and the end-to-end total time, determine the target distributed parallel scheme when deploying the target large model in the distributed cluster from among the alternative distributed parallel schemes.

[0237] Specifically, the corresponding single-device computing time is calculated and added up to obtain the end-to-end total time corresponding to each alternative distributed parallel solution; based on the calculated single-device computing time and end-to-end total time, a distributed solution that meets actual needs is screened out from each alternative distributed parallel solution, for example, a distributed parallel solution with optimal performance (minimum end-to-end total time or minimum single-device time, etc.) and minimum computing device consumption, is used as the target distributed parallel solution when the target large model is deployed in a distributed cluster, thereby realizing the determination of the distributed parallel solution.

[0238] This implementation method determines multiple alternative distributed parallel schemes based on the benchmark description information of the target large model and the cluster information of the distributed cluster. Through the distributed computing graph and parallel computing technology, the original computing graph is converted into a pipeline computing graph and then a single-device computing graph is constructed for the alternative distributed parallel schemes. Through pipeline parallelism, different parts of the model can be distributed to different devices, so that each device is processing data, reducing the idle time of the device, improving the overall utilization of resources, speeding up the training speed of the model, and better balancing the load of each device. Finally, the target distributed parallel scheme with the best performance is determined based on the total time consumption of each scheme, avoiding time-consuming iterative testing of each distributed parallel scheme one by one, and realizing automatic screening and decision-making of alternative schemes based on cluster device information and model information of the target large model, reducing labor costs and equipment resource consumption, and improving the efficiency of determining distributed parallel schemes.

[0239] Based on the above embodiments, calculating the single-device computing time of each single-device computing graph may include:

[0240] Obtain the target single-device calculation graph currently being processed; calculate the first-category calculation time of each model operator and the second-category calculation time of each communication operator in the target single-device calculation graph according to the cluster information of the distributed cluster; calculate the single-device calculation time of the target single-device calculation graph according to the deployment scenario information of the target large model on the distributed cluster, the first-category calculation time and the second-category calculation time.

[0241] Specifically, the single-device computing time of the target single-device computing graph can be calculated through the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time. Among them, the first type of computing time refers to the time consumption of each model operator in the target single-device computing graph, and the second type of computing time refers to the time consumption of each communication operator in the target single-device computing graph. The time consumption of model operators and communication operators can be collected and analyzed through the performance analysis tools provided by the deep learning framework, such as the MindInsight tool of the MindSpore framework, to obtain detailed time consumption statistics of operators including the time consumption of model operators and communication operators.

[0242] On the basis of the above embodiments, according to the cluster information of the distributed cluster, calculating the first type of computing time consumption of each model operator in the target single-device computing graph may include:

[0243] Obtain the target model operator currently being processed in the target single-device computation graph; calculate the input and output memory access time of the target model operator according to the size of all input tensors and all output tensors of the target model operator, as well as the device memory access bandwidth and memory access efficiency; calculate the computation time of the target model operator according to the number of floating-point operations of the target model operator, the number of computation operations per second of the device, and the device computation efficiency; calculate the first category computation time of the target model operator according to the predetermined operator kernel program startup time, the input and output memory access time of the target model operator, and the computation time of the target model operator.

[0244] Specifically, the target model operator currently being processed is obtained in the target single-device computation graph; the input and output memory access times of the target model operator are calculated according to the sizes of all input tensors and all output tensors of the target model operator, as well as the device memory access bandwidth and memory access efficiency.

[0245] That is, the input and output memory access time of the target model operator = (the size of all input tensors of the operator + the size of all output tensors of the operator) / (device memory access bandwidth × memory access efficiency).

[0246] Calculate the computation time of the target model operator based on the floating-point operations of the target model operator, the number of computations per second of the device, and the device computation efficiency.

[0247] That is, the computation time of the target model operator = the number of floating-point operations of the operator / (the number of computation operations per second of the device) × computation efficiency.

[0248] Among them, the device memory access bandwidth, memory access efficiency, number of computing operations per second of the device and computing efficiency match the hardware specifications of the computing device and can be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0249] Finally, the first type of computing time of the target model operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target model operator, and the computing time of the target model operator.

[0250] That is, the first type of computing time of the target model operator = operator kernel program startup time + max(input and output memory access time of the target model operator, computing time of the target model operator), where the operator kernel program startup time is the inherent time of the corresponding device, and max(.) is the maximum value function.

[0251] On the basis of the above embodiments, according to the cluster information of the distributed cluster, calculating the second type of computing time consumption of each communication operator in the target single-device computing graph may include:

[0252] In the target single-device computation graph, obtain the target communication operator currently being processed, and obtain the parallel type of the target communication operator; if the parallel type is tensor parallelism, calculate the second type of computational time consumption of the target communication operator according to the device interconnection mode, the number of tensor parallelisms, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator; if the parallel type is pipeline parallelism, calculate the second type of computational time consumption of the target communication operator according to the pipeline parallel communication operator communication volume, the pipeline parallel communication bandwidth, and the pipeline parallel communication efficiency.

[0253] In an optional implementation of this embodiment, generally speaking, there are two types of device interconnection between computing devices: full connection or ring connection. According to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth and the operator type of the target communication operator, a corresponding time consumption formula is selected to calculate the second type of computing time of the tensor parallel target communication operator.

[0254] In a specific example, when the device interconnection mode in the distributed cluster is fully connected, the calculation formula for the second type of computing time consumption of the target communication operator with the parallel type of tensor parallelism = tensor parallel communication operator communication volume / (tensor parallel communication bandwidth*tensor parallel communication efficiency).

[0255] Among them, the communication volume of tensor parallel communication operators = the original data volume of the operator * (computing device - 1) / total number of computing devices. The original data volume of the operator can be understood as the original data volume before tensor parallelization according to the tensor parallel number. The original data volume is compatible with the operator type of the target communication operator. The total number of computing devices is consistent with the tensor parallel number. The tensor parallel communication bandwidth and tensor parallel communication efficiency match the hardware specifications of the computing device and can be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0256] In another optional implementation of this embodiment, the calculation formula for the second type of calculation time consumption of the target communication operator with pipeline parallel type=pipeline parallel communication operator communication volume / (pipeline parallel communication bandwidth*pipeline parallel communication efficiency).

[0257] The communication volume of pipeline parallel communication operators can be understood as the data volume of a single card. For example, if the amount of data processed by each computing device is A, then the communication volume of pipeline parallel communication operators is also A. The pipeline parallel communication bandwidth and pipeline parallel communication efficiency can also be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0258] Furthermore, the method may further include: if the parallel type is expert parallelism, the second type of computing time consumption of the target communication operator may be calculated according to the expert parallel communication operator communication volume, the expert parallel communication bandwidth and the expert parallel communication efficiency.

[0259] On the basis of the above embodiments, according to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single device computing time of the target single device computing graph is calculated, which may include:

[0260] If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth and data parallel communication efficiency, and the single-device calculation time of the target single-device calculation graph is calculated based on the first type of calculation time, the second type of calculation time and the gradient reduction time; if the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device calculation time of the target single-device calculation graph is calculated based on the first type of calculation time and the second type of calculation time.

[0261] Specifically, for the deployment scenario information of the target large model on the distributed cluster, the model training scenario, the single device computing time of the target single device computing graph includes not only the first type of computing time and the second type of computing time, but also the gradient reduction time. Among them, the gradient reduction time can be calculated based on the model gradient reduction communication volume, data parallel communication bandwidth and data parallel communication efficiency.

[0262] That is, the calculation formula for gradient reduction time = model gradient reduction communication volume / (data parallel communication bandwidth × data parallel communication efficiency).

[0263] The model gradient reduction communication volume refers to the amount of gradient information of all model operators. The data parallel communication bandwidth and data parallel communication efficiency can also be directly obtained from the cluster information of the distributed cluster or obtained through actual measurement.

[0264] Specific application scenarios

[0265] For ease of understanding, a specific application scenario to which each embodiment of the present invention is applicable is now described. In this application scenario, specific operation steps for efficiently generating a 6D (6 types of distributed strategies) distributed strategy when mapping a specific large model to a specific distributed cluster are specifically described.

[0266] Specifically, Fig.13As shown in the figure, the 6D distributed parallel generation method of the large model is broken down into six functional modules as shown in the figure below, namely, the storage calculation required for the large model, the definition of the distributed parallel search space, the acquisition of a reasonable combination of distributed parallel solutions, the construction of a single-card calculation graph (that is, a single-device calculation graph), the end-to-end total time calculation, and the generation of the best distributed solution.

[0267] 1. Storage and computing required for large models

[0268] in, Fig.14 A schematic diagram of a model training storage content applicable to an embodiment of the present invention. As mentioned above, after a specific large model is deployed in a specific distributed cluster, the model training task or model reasoning task can be executed accordingly. Different task types may not be suitable for the best distributed solution, because the above six functional modules can be executed according to the actual task type. Specifically, for different task types, the calculation content of the storage calculation required by the large model is not consistent. In this specific application scenario, taking the model training task as an example, the storage calculation required by the large model is introduced.

[0269] Specifically, Fig.14 As shown, the content of model training storage may include: weight-related storage (whose size is equivalent to the weight-related storage requirement in the previous text) and activation-related storage (whose size is equivalent to the activation-related storage requirement in the previous text). Among them, weight-related storage includes weight storage, gradient storage, and optimizer storage. Activation-related storage includes activation storage and temporary storage. Based on the above specific storage content, the amount of data in the weight-related storage during model training storage and the amount of calculation in the activation-related storage can be calculated accordingly.

[0270] 2. Distributed Parallel Search Space Definition

[0271] In this application scenario, the entire distributed parallel search space is orthogonally obtained by tensor parallel search set, pipeline parallel search set, top-bottom parallel search set and data parallel search set. The definitions of each search set are as follows:

[0272] (1) Tensor parallel search set: Based on the number of high-bandwidth interconnect devices supported by the cluster, the tensor parallel search set is defined as a set of powers of 2 ranging from 1 to the number of high-bandwidth interconnect card devices. For example, if the number of high-bandwidth interconnect devices is 32, the tensor parallel search set is 1, 2, 4, 8, 16, and 32.

[0273] (2) Pipeline parallel search set: The pipeline parallel search set can be an integer starting from 1. The present invention supports setting some a priori restrictions, such as setting the maximum value of pipeline parallelism, and setting the number of pipeline parallelism to be divisible by the number of model layers. Taking GPT175B as an example, the number of layers of the model is 96. If the maximum value of pipeline parallelism is set to 8 and the number of pipeline parallelism is set to be divisible by the number of model layers, the pipeline parallel search set is 1, 2, 4, 6, 8.

[0274] (3) Sequence parallel search set: Sequence parallelism needs to be used together with tensor parallelism and the number of sequence parallelism is equal to the number of tensor parallelism, so there is no need to define a sequence parallel search set.

[0275] (4) Context parallel search set: The context parallel search set is defined as a set of integers ranging from 1 to the maximum context parallel number, including 1 and the maximum context parallel number, and satisfying the restriction that the context parallel number is divisible by the context size.

[0276] (5) Number of experts in parallel: There are two ways to parallelize experts. One is that all experts are distributed on different devices, and the other is that some of all experts are on the same device. If all experts are distributed on different devices, the number of experts in parallel is equal to the number of model experts; if some of all experts are on the same device, the number of experts in parallel is equal to 1.

[0277] (6) Data parallel search set: The maximum number of data parallels is calculated based on the total number of cluster devices. The maximum number of data parallels is equal to the total number of cluster devices divided by the number of devices in one data parallel group, where the number of devices in one data parallel group = the number of tensor parallels × the number of pipeline parallels × the number of context parallels. The data parallel search set is an integer (integer or power of 2) in the range from 1 to the maximum number of data parallels, and can be 1 and the maximum number of data parallels.

[0278] 3. Obtain a reasonable combination of distributed parallel solutions

[0279] Fig.15 FIG. 1 is a schematic diagram of obtaining a reasonable distributed parallel solution applicable to an embodiment of the present invention. Fig.15 As shown, the steps to obtain a reasonable distributed parallel solution combination are as follows:

[0280] (1) Traverse each distributed parallel solution defined in the distributed search space.

[0281] (2) Still taking the model training task as an example, the single-device storage calculation required by the distributed parallel solution = single-device weight-related storage size + single-device activation-related storage size. The single-device weight-related storage size is the weight-related storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the pipeline parallel number; the single-device activation-related storage size is the activation storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the context parallel number. The single-device temporary storage size is the temporary storage size calculated in the storage module required by the model divided by the product of the tensor parallel number and the context parallel number.

[0282] (3) For each distributed parallel solution, a single device storage condition check is performed. If the single device storage required by the distributed parallel solution is less than or equal to the single device storage specification multiplied by the storage utilization rate, then it is a reasonable distributed parallel solution; otherwise, it is an invalid parallel solution.

[0283] 4. Build a single card calculation graph

[0284] Fig.16 This is a schematic diagram of constructing a single-card calculation graph applicable to an embodiment of the present invention. The original calculation graph is first converted into a pipeline calculation graph, and then the single-card calculation graph is constructed. Fig.16 As shown, the specific method of constructing a single-card calculation graph is described as follows:

[0285] (1) Build a pipeline computation graph.

[0286] Evenly distribute strategies or computing tasks according to the number of model layers, evenly distribute strategies and the number of pipeline parallelism, and add point-to-point communication operators for sending and receiving to convert the original calculation graph into a calculation graph on the pipeline.

[0287] (2) Construct a computational graph for a single device.

[0288] A computational operator graph for a single device is obtained from a pipeline computation graph, the number of context parallelism, and the number of tensor parallelism. The type of inserted tensor parallel communication operators is automatically derived based on the operator tensor parallel segmentation dimension, such as the all-reduce communication operator, the reduce-scatter communication operator, and the all-gather communication operator. The computational graph for a single device is obtained, including computational operators and communication operators.

[0289] 5. End-to-end total time calculation method

[0290] First, calculate the time consumed by computing the graph on a single device, and then use the pipeline parallel scheduling strategy, such as a forward and a reverse scheduling strategy, to reversely calculate the end-to-end total time consumed. The method for computing the graph on a single device is as follows:

[0291] The time consumption of a single device computation graph includes the computational operator time consumption, tensor parallel communication operator time consumption, and pipeline parallel communication time consumption. If it is a model training task, it also includes the gradient reduction communication time consumption.

[0292] The time consumption of the single-device calculation graph is calculated based on the single-device calculation graph and the parallel scheduling method. If the communication operator and the calculation operator are executed serially, the single-device calculation graph time consumption = the sum of the calculation operator time consumption + the sum of the tensor parallel communication operator time consumption + the pipeline parallel communication time consumption + the gradient reduction time consumption. If some tensor parallel communication operators are executed in parallel with some calculation operators, the single-device calculation graph time consumption needs to first take the maximum value of the communication time consumption and the calculation operator time consumption of the partial parallel execution, and then sum it with the other time consumption. The calculation formula for each time consumption is as follows:

[0293] (1) Computational single operator time-consuming calculation formula.

[0294] The input and output memory access time of the target computing operator = (the size of all input tensors of the operator + the size of all output tensors of the operator) / (device memory access bandwidth * memory access efficiency).

[0295] The computation time of the target computing operator = the number of floating-point operations of the operator / (the number of computing operations per second of the device * computing efficiency).

[0296] The first type of computing time of the target computing operator = the operator kernel program startup time + max (the input and output memory access time of the target computing operator, the computing time of the target computing operator), where the operator kernel program startup time is the inherent time of the corresponding device.

[0297] (2) Formula for calculating the time consumption of a single operator in tensor parallel communication.

[0298] There are two types of device interconnection: full connection and ring connection. Select the corresponding time consumption formula to calculate the communication time consumption according to the device interconnection mode, the number of tensor parallel operations, the tensor parallel interconnection bandwidth, and the communication operator type. For example, the time consumption calculation formula for the fully connected communication operator = the communication volume of the tensor parallel communication operator ( / (tensor parallel communication bandwidth*tensor parallel communication efficiency).

[0299] (3) Formula for calculating the time consumption of a single operator in pipeline parallel communication.

[0300] The time consumption calculation formula of a single operator in pipeline parallel communication = pipeline parallel communication operator communication volume / (pipeline parallel communication bandwidth*pipeline parallel communication efficiency).

[0301] (4) Gradient reduction time-consuming calculation formula.

[0302] Gradient reduction time calculation formula = model gradient reduction communication volume / (data parallel communication bandwidth * data parallel communication efficiency)

[0303] 6. Determine the best distributed parallel solution

[0304] The present invention supports multiple performance indicators such as minimum end-to-end total time consumption, best single device performance or minimum total number of devices, and outputs the corresponding optimal distributed parallel solution.

[0305] This implementation method proposes a method for determining a distributed parallel solution. According to the model structure, model size, context length, cluster hardware device specifications, and device interconnection, a distributed parallel solution with the shortest end-to-end total time consumption, the best single device performance, or the smallest total number of devices is determined, so as to avoid algorithm personnel from iterating all feasible distributed combination solutions one by one in a cluster environment of 1,000 or 10,000 cards, and comparing and analyzing the results. Automatic screening and decision-making of backup solutions is achieved based on cluster device information and model information of the target large model, which reduces the consumption of manpower and equipment resources and improves the efficiency of determining distributed parallel solutions.

[0306] Fig.17 This is a flow chart of a cost evaluation method for a distributed system in an embodiment of the present invention, and this embodiment is refined based on the above embodiments. In this embodiment, the operation of "obtaining the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme from the three dimensions of operator-level cost, device-level cost and cluster-level cost according to the target distributed parallel scheme" is concretized.

[0307] Correspondingly, such as Fig.17 As shown, the method includes:

[0308] S1710. Extract the original calculation graph of the target large model from the benchmark test description information of the target large model.

[0309] Among them, the forward calculation graph of the target large model can be extracted from the benchmark description information of the target large model as the original calculation graph.

[0310] S1720. Split the original computation graph according to the target distributed parallel scheme to obtain at least one standard single-device computation graph for execution in a single computing device, wherein the standard single-device computation graph includes a communication operator.

[0311] Specifically, a standard single-device computation graph refers to a computational process built on a computing device, which includes operators that perform computational tasks and possible communication operations. In such a computation graph, computational operators and communication operators work together to complete the forward and backward propagation of the model.

[0312] It is understandable that by deploying the target large model according to the target distributed parallel scheme, it is often necessary to distribute the various calculations required to be performed by the target large model to different computing devices in different servers located in different cabinets for execution. Furthermore, the original calculation graph needs to be split into a standard single-device calculation graph executed by each single computing device according to the target distributed parallel scheme. This further involves the situation where different copies of data, or parts of a piece of data, are split into different computing devices in a distributed cluster for execution. Therefore, in order to obtain complete data processing results, it is necessary to insert a communication operator for realizing the data aggregation function at a specific position between the basic operators (that is, the model operators or the computing operators).

[0313] That is, the final standard single-device computation graph includes both the basic operators in the original computation graph and the communication operators subsequently inserted for realizing data exchange.

[0314] S1730. According to the annotation information of each operator in the standard single-device computing graph and the architecture information of the distributed cluster, a pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the standard single-device computing graph.

[0315] Among them, the annotation information of each operator in the standard single-device calculation graph can be understood as the pre-annotated basic description information of each basic operator (computational operator) in the original calculation graph. The architecture information of the distributed cluster (also called cluster information) can include: cluster hardware device specifications, interconnection information between devices, storage capacity, geographical location of nodes, operating system and version, etc. Cluster information is the basis of distributed system modeling and the prerequisite for distributed systems to meet business needs and provide high-quality services.

[0316] Furthermore, the architecture information may also include the read and write bandwidth and bandwidth utilization of different storage locations, as well as the parallel mode of computing resources of the distributed cluster (for example, 2D and 1D parallelism), etc., which is not limited in this embodiment.

[0317] In this embodiment, operator cost calculation models corresponding to different operators can be pre-built through various modeling methods. Among them, different types of operators can correspond to different operator cost calculation models. For example, the Matmul basic operator used to implement matrix multiplication has a corresponding operator cost calculation model, and the All-Gather communication operator has a corresponding operator cost calculation model.

[0318] In an optional implementation of this embodiment, each operator cost calculation model takes one or more annotation information and one or more architecture information of the corresponding operator as input, and calculates the cost as output, where the output is also called the operator-level cost.

[0319] S1740. Calculate the device-level cost corresponding to each standard single-device computation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain a cluster-level cost.

[0320] In this embodiment, by accumulating and summing the operator-level costs of each operator included in each standard single-device calculation graph, the device-level cost corresponding to each standard single-device calculation graph can be obtained. In addition, by accumulating and summing the device-level costs corresponding to each standard single-device calculation graph, the cluster-level cost corresponding to the distributed cluster can be preliminarily obtained. Further, considering the distributed strategy defined in the target distributed parallel scheme, typically data parallelism, pipeline parallelism or expert parallelism, new cluster-level costs will also be introduced. Then, considering the device-level cost corresponding to each standard single-device calculation graph and the various distributed strategies defined in the target distributed parallel scheme, the cluster-level cost is calculated.

[0321] S1750. Obtain system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel solution based on the operator-level cost, the device-level cost, and the cluster-level cost.

[0322] In this embodiment, after calculating the operator-level cost of each operator in each standard single-device calculation graph, the device-level cost corresponding to each standard single-device calculation graph, and the cluster-level cost when the target large model is deployed in a distributed cluster, different types of statistical indicator values ​​can be obtained according to multiple statistical dimensions, for example, the operator cost statistics of each target operator (typically, operators that the user is interested in or focuses on) obtained in each computing device, the operator cost statistics of each type of communication operator obtained in each computing device, etc.

[0323] Based on the obtained statistical indicator values, the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel solution can be obtained.

[0324] Through the above settings, the performance of the same target large model under distributed clusters of different hardware architectures can be compared horizontally, and then the architecture of the distributed cluster with the best performance for deploying the target large model can be explored. After the target large model is deployed on the distributed cluster with the best performance, the computing tasks matching the target large model can be executed efficiently and accurately, meeting the high-performance execution requirements of the target large model.

[0325] This implementation provides a new method for evaluating the cost of a distributed system. It can efficiently and accurately evaluate the operator-level cost, device-level cost, and cluster-level cost in distributed modeling based on the target distributed parallel scheme used when deploying a specific target large model in a distributed cluster, and thus obtain accurate system performance analysis results, so as to achieve a better balance in evaluation accuracy, evaluation time, and evaluation cost during the system performance analysis process.

[0326] Based on the above embodiments, before the original computation graph is segmented according to the target distributed parallel scheme to obtain at least one standard single-device computation graph for execution in a single computing device, the following may also be included:

[0327] The original calculation graph is pre-processed according to the optimization strategy.

[0328] Based on the above embodiments, before obtaining the system performance analysis result when the target large model is deployed in the distributed cluster according to the target distributed parallel solution according to the operator-level cost, the device-level cost and the cluster-level cost, the following may also be included:

[0329] An optimization strategy post-processing is performed on at least one of the operator-level cost, the device-level cost, and the cluster-level cost.

[0330] Specifically, the optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy.

[0331] The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

[0332] Specifically, there are many algorithms and engineering optimization strategies and methods in the computing process of AI applications (for example, training, reasoning or fine-tuning), such as: operator fusion to improve operator computing efficiency, low-bit quantization to reduce data volume, kv-cache or GQA to improve reasoning efficiency, and Lora or QLora to reduce the amount and difficulty of fine-tuning (also known as fine-tuning). In order to model the impact of various optimization strategies on system costs under given hardware and cluster architectures, the architecture of optimization strategy pre-processing and optimization strategy post-processing is proposed in various embodiments of the present invention.

[0333] The optimization strategy pre-processing mainly adjusts the calculation graph before cost calculation, and the optimization strategy post-processing mainly further optimizes the costs at all levels. To achieve the relative independence of optimization processing and cost calculation, multiple pre- and post-processing can be superimposed on the cost calculation, which simplifies the modeling of multiple optimization strategies and makes it easy to verify and compare the impact of a certain optimization strategy on the system. These optimization strategies have different principles and complexity. Some optimization strategies only require pre-processing (such as operator fusion, kv-cache or GQA) or post-processing (such as various pipeline parallel optimizations), while others require both pre- and post-processing to work together (such as low-bit quantization).

[0334] Fig.18 This is a flowchart of another cost evaluation method for a distributed system in an embodiment of the present invention. This embodiment is optimized based on the above embodiments. In this embodiment, the operations of "slicing the original calculation graph according to the target distributed parallel scheme to obtain at least one standard single-device calculation graph for execution in a single computing device" and "combining the device-level costs according to the distributed strategies to obtain the cluster-level cost" are concretized.

[0335] Correspondingly, such as Fig.18 As shown, the method may include:

[0336] S1810. Extract the original calculation graph of the target large model from the benchmark description information of the target large model.

[0337] S1820. Split the original computation graph according to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme to obtain at least one standard single-device computation graph for execution in a single computing device.

[0338] As mentioned above, four segmentation semantics can be predefined, namely no segmentation, segmentation by group, segmentation by dimension, and segmentation by value;

[0339] Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

[0340] Among them, not split (Not Split), denoted as N. The tensor is not split, which means that it is a complete copy on all computing devices. Group Split (Group Split), denoted as G(i), divides the tensor into different groups. The tensor is complete on all computing devices (also called computing cards) in the group to which it belongs, and does not exist on other groups. i indicates that the tensor belongs to the i-th group. Dim Split (Dim Split), denoted as D(i). The tensor is equally divided on all computing devices along the i-th dimension, and each computing device stores a tensor of size 1 / n (n is the total size of data on dimension i). Value Split (ValueSplit), denoted as V. The size of the tensor on each computing device card is the same as the original tensor size, but the sum of the tensor values ​​on all computing devices is equal to the original tensor.

[0341] Generally speaking, pipeline parallelism or expert parallelism involves dividing the model operators (also called basic operators) contained in the original calculation graph into groups. At this time, it involves dividing the model operators contained in the original calculation graph into different standard single-device calculation graphs. At this time, the original calculation graph can be split and processed according to at least one of the pipeline parallel number and expert parallel number in the target distributed parallel scheme.

[0342] S1830. According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, mark the parallel split dimensions of the input and output tensors of each basic operator in the standard single-device computation graph, and add communication operators to each standard single-device computation graph based on the marking results.

[0343] Furthermore, the tensor parallelism, sequence parallelism and context parallelism involve splitting the input activation values ​​or model weights of the basic operators in the tensor dimension, and thus, the complete input and output tensors and model weights in a standard single-device computation graph can be further divided into multiple standard single-device computation graphs. It can be understood that the above-mentioned splitting method will not change the organization of the model operators contained in each standard single-device computation graph, but will change the number of tensors contained in each standard single-device computation graph.

[0344] That is, if the number of parallel tensors corresponding to a pipeline calculation graph obtained by S1820 division is 8, it is determined that the pipeline calculation graph needs to be executed in parallel on 8 computing devices. Each of the above computing devices corresponds to a standard single-device calculation graph with the same structure as the pipeline calculation graph. The difference is that the input and output tensors of each basic operator in each standard single-device calculation graph are divided into 1 / 8 of the original compared with the pipeline calculation graph.

[0345] S1840. According to the annotation information of each operator in the standard single-device computing graph and the architecture information of the distributed cluster, a pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the standard single-device computing graph.

[0346] In an optional implementation of this embodiment, the operator cost calculation model is constructed using at least one of the following methods:

[0347] Disassemble the large-grained operators to obtain the combination of ATen operators;

[0348] Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators;

[0349] Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results;

[0350] The operator cost calculation model includes: a basic operator cost calculation model and a communication operator cost calculation model.

[0351] S1850. Calculate the device-level cost corresponding to each standard single-device computation graph according to the operator-level cost of each operator.

[0352] S1860. Calculate the cluster communication cost between computing devices deployed with different standard single-device computing graphs based on at least one of the data parallel number, pipeline parallel number, and expert parallel number.

[0353] In this embodiment, considering that data parallelism is to distribute computing tasks to computing devices of different data parallel groups for parallel computing, data parallelism will not split the target large model, nor will it change the standard single-device computing graph, but it will introduce new communication costs. Therefore, the cluster communication cost can be directly accumulated and calculated based on the number of data parallelism. Generally speaking, the larger the number of data parallelism, the higher the value of the cluster communication cost introduced. Therefore, a corresponding computing model can be constructed to calculate the cluster communication cost that matches the number of data parallelism.

[0354] Similarly, the number of pipeline parallelism and the number of expert parallelism can affect or determine the basic operators included in each standard single-device calculation graph, but will not introduce communication operators in the standard single-device calculation graph. The communication cost introduced can also be obtained by directly accumulating the cluster communication cost. Similarly, the larger the number of pipeline parallelism and the number of expert parallelism, the higher the value of the cluster communication cost introduced. Therefore, a corresponding calculation model can be constructed to calculate the cluster communication cost that matches the number of pipeline parallelism or the number of expert parallelism.

[0355] S1870: Calculate a cluster-level cost based on each device-level cost and the cluster communication cost.

[0356] S1880. According to the operator-level cost, the device-level cost, and the cluster-level cost, obtain a system performance analysis result when the target large model is deployed in a distributed cluster according to the target distributed parallel solution.

[0357] In an optional implementation of this embodiment, according to the operator-level cost, the device-level cost, and the cluster-level cost, a system performance analysis result is obtained when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme, including:

[0358] The operator-level cost of each operator, the IO (input and output) cost and computational cost of each operator in the forward calculation phase, the backward BPI (backward propagation input gradient) calculation phase, and the backward BPK (backward propagation kernel gradient) calculation phase, and the judgment result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results;

[0359] The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results;

[0360] The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

[0361] On the basis of the above embodiments, according to the operator-level cost, the device-level cost and the cluster-level cost, obtaining the system performance analysis result when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme may also include:

[0362] Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators;

[0363] The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

[0364] The target operator may be an operator that the modeling user is interested in or a pre-set core operator, etc. The computing power type may include: 2D or 1D, etc.

[0365] This implementation provides a new method for evaluating the cost of a distributed system. It can efficiently and accurately evaluate the operator-level cost, device-level cost, and cluster-level cost in distributed modeling based on the target distributed parallel scheme used when deploying a specific target large model in a distributed cluster, and thus obtain accurate system performance analysis results to meet people's growing demand for convenient and personalized distributed system cost evaluation.

[0366] On the basis of the above embodiments, after obtaining the system performance analysis result when the target large model is deployed in the distributed cluster according to the target distributed parallel solution according to the operator-level cost, the device-level cost and the cluster-level cost, the following may also be included:

[0367] In response to a target data item selected by a user in the system performance analysis result through a human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

[0368] Specific application scenarios

[0369] For easier understanding, the embodiments of the present invention are first described in the form of a specific application scenario. The solutions of the embodiments of the present invention can be executed by a cost evaluation platform. Fig.19 The schematic diagram shows the structure of a cost evaluation platform of a distributed system to which the present invention is applicable.

[0370] Specifically, Fig.19 As shown, the present invention breaks down the cost evaluation platform into four main parts, namely: operator cost modeling, optimization strategy processing (including optimization strategy preprocessing and optimization strategy postprocessing), Roofline Model (model) Cost calculation and system Cost analysis. The above four main parts are described below.

[0371] 1. The invention integrates the following inputs to model the operator cost

[0372] (1) aten operator decomposition: The benchmark construction system used for distributed system modeling supports abstract granularity scaling, and the operator granularity in the resulting distributed computing graph is generally larger than the kernel operator granularity during actual runtime (such as the pytorch aten operator). In each embodiment of the present invention, an operator decomposition tool is provided, which can export the pytorch module and subgraph to Fx Graph, decompose large-grained distributed operators (such as RoPE, flash-attention, etc.) into a combination of aten basic operators, and then form a cost model of large-grained operators based on the known cost of aten operators.

[0373] (2) Algorithm implementation: When building a forward-looking model, new operators without code implementation may be encountered. In response to this situation, the embodiments of the present invention provide a modeling programming interface that can easily build an operator cost model based on the algorithm logic.

[0374] (3) System measured data: Based on the calculation of the basic operator cost according to theoretical values ​​such as the amount of computation and the amount of data transported, the embodiments of the present invention further introduce the operator cost model of the system measured data correction theory. For example, the operator cost model is calibrated using the performance data such as computing power utilization, effective bandwidth, and kernel startup time measured on different platforms.

[0375] (4) Hardware simulation data: For hardware architectures that are still in the R&D stage, the theoretical model can use the performance data of the simulation platform to calibrate the operator cost.

[0376] (5) SoC (System on Chip) modeling results: In the early stage of product development, such as the architecture exploration stage, hardware, software and simulation platforms are not available. The basic operator cost model of the present invention can refer to the results of other SoC level modeling for performance calibration.

[0377] Accordingly, in Fig. 20 A schematic diagram of the structure of a basic operator cost model applicable to an embodiment of the present invention is shown in FIG. Fig. 20 As shown, the basic operator Cost model includes three parts: forward Cost, reverse Bpi Cost and reverse Bpk Cost. In this embodiment, forward and reverse cost modeling is performed for each basic operator, rather than modeling the forward and reverse operators separately, so that the cost calculation does not depend on the reverse graph, and it is more convenient to apply optimization strategies. For example, a configuration option can be used to control the parallel optimization of reverse Bpk cost and reverse communication without changing the reverse calculation graph.

[0378] The forward cost, reverse Bpi cost and reverse Bpk cost are composed of three parts: kernel startup time (that is, kernel startup cost), data transfer cost and calculation cost. Data transfer and calculation can be done in parallel, so: total cost = kernel startup cost + Max (data transfer cost, calculation cost), where Max (.) represents the maximum value operation.

[0379] The data transfer cost is the sum of the read and write costs of the input and output data. The amount of data is determined by the input and output Tensor shape, data type, and segmentation scheme, and the total transfer time is determined by the amount of data, the read and write bandwidth of the data storage location (such as L3 DeviceMemory, L4 Cxl memory), and the bandwidth utilization:

[0380] Data transfer time = L3 data volume / (L3 bandwidth * L3 bandwidth utilization) + L4 data volume / (L4 bandwidth * L4 bandwidth utilization)

[0381] This cost evaluation platform can also support the separate settings of read bandwidth and write bandwidth, as well as their respective utilization rates, to model the read-write imbalanced architecture.

[0382] The computational cost is divided into four types: 2D, 1D, SFU, and RNG, depending on the computing resources used. Simple basic operators have only one type of computational cost, while complex basic operators (such as Flash-Attention) may include multiple or all types of costs. This cost evaluation platform counts the costs of different computing resources separately, which can better compare the performance indicators of different architectures under different computing power ratios. For hardware platforms that support parallel operation of different computing resources (such as 2D / 1D parallel operation), parallel cost conversion can also be conveniently performed. In addition to the inherent computational cost, there may also be additional computational costs, such as the cost of scale and cast calculations added after the quantization solution is applied.

[0383] In the standard single-device computing graph, in addition to the basic operators, it further includes a communication operator, wherein the communication operator modeling is as follows.

[0384] Optionally, the communication volume of commonly used communication operators is shown in Table 3 below: (n represents the number of computing cards (that is, the number of computing devices), and T represents the original Tensor size)

[0385] Table 3

[0386]

[0387] Specifically, communication operator Cost = control delay + data volume / (interconnection bandwidth * bandwidth utilization)

[0388] Control delay and bandwidth utilization are determined by factors such as communication protocol (for example, RDMA, Infinband, or RoCE), network topology (for example, Full-Mesh or Ring), network hardware, and algorithm efficiency, and can be obtained through system measurement or simulation.

[0389] 2. Optimize strategy processing

[0390] Among them, Fig.21 A flow chart showing a method of adding optimization strategy pre-processing and optimization strategy post-processing to a cost evaluation process of a distributed system applicable to an embodiment of the present invention is shown in FIG.

[0391] like Fig.21 As shown in the figure, this cost evaluation platform supports modeling of various deep learning optimization strategies, such as operator fusion, low-bit quantization, MQA / GQA, and recalculation. The processing of optimization strategies is generally divided into two parts: (1) preprocessing the computational graph; (2) correcting and converting the cost.

[0392] Taking low-bit quantization as an example, through the computational graph control capability of the modeling system, the non-quantized computational graph imported by the benchmark can be quantized through a simple configuration file, including: (1) modifying the input and output Tensor types of the operators that need to be quantized; (2) setting quantization parameters and selecting the corresponding quantization cost model; (3) inserting quantization (or dequantization) operators or integrating quantization (or dequantization) calculations with other operators.

[0393] Specifically, for quantization-aware training (QAT), after completing the cost calculation of the entire model, it is also necessary to attach the calculation cost of quantizing the model parameters for each training iteration.

[0394] 3. Roofline Model Cost calculation, also known as system cost calculation

[0395] Among them, Fig. 22 The overall flow chart of calculating the cluster-level cost applicable to the embodiment of the present invention is shown in FIG. Fig.23 It is a schematic diagram of a standard single-device calculation graph generation process applicable to an embodiment of the present invention.

[0396] like Fig. 22 and Fig.23 As shown in the figure, the system cost calculation includes two scheduling processes:

[0397] (1) Cluster scheduling: According to the input distribution scheme, the original computation graph is divided into different pipeline groups, data groups or expert groups according to the pipeline parallelism, data parallelism or expert parallelism logic to obtain a standard single-device computation graph executed on a single computing device.

[0398] (2) Operator scheduling for a single computing device: Based on the logical relationship between operators (including communication operators with tensor parallelism, sequence parallelism, or context parallelism) in the standard single-device computing graph, a logical ID is generated for each operator, and the cost of each operator is calculated in sequence according to the logical ID. Cost optimization can be performed as needed during the process. For example, different cost models can be selected for each operator cost calculation based on configurations such as quantization and fusion.

[0399] Among them, basic operators (i.e., model operators) and communication operators with the same logic id (i.e., no logic dependency) can be optimized in parallel. Finally, according to the cluster scheduling logic, the cost of a single computing device and the communication cost of pipeline parallelism, data parallelism, or expert parallelism are combined to calculate the cost of the entire cluster.

[0400] 4. System Cost Analysis

[0401] Specifically, the embodiments of the present invention can provide the following system cost analysis capabilities:

[0402] (1) Low-cost comparative experiment capability. From the benchmark import method to the distributed search strategy and the modeling of the roofline cost model, the entire cost evaluation platform is rationally abstracted around distributed modeling, and can complete modeling calculations without distributed cluster hardware. The cost evaluation platform provides a highly customized and parallel computing framework. Using the control variable method, it can complete thousands of end-to-end cost calculations in minutes, and compare and analyze the impact of key parameter changes on system performance.

[0403] (2) Detailed data statistics function. The modeling system provides cost data statistics at three levels: operator, single computing device, and cluster:

[0404] a) Operator-level cost: Focus on the data IO cost and computing cost of the forward, reverse BPI, and reverse BPK parts of a single operator to determine whether the operator performance is IO-intensive or computing-intensive. When calculating the operator cost, the cost of different computing power types (2D, 1D, SFU, or RNG, etc.) and the kenerl startup cost are also counted.

[0405] b) Single-device cost: Traverse the standard single-device computation graph on a single computing device after distributed partitioning, combine the single operator cost and communication cost according to the specified parallel and optimization strategy, and obtain the total cost on a computing device. The following cost statistics are performed while calculating the single-device cost: different computing power costs (2D, 1D, SFU, RNG, etc.), the costs of core operators and user-focused operators (typically, dropout, linear, and flash-attention operators, etc.), and single-device communication costs (typically, tensor parallelism or context parallelism, etc.)

[0406] c) Cluster-level cost: Combine the single-device cost according to the distributed parallel strategy, apply the optimization strategy, and get the cluster cost. At the same time, perform cluster performance statistics, including: different computing power costs (2D, 1D, SFU, and RNG, etc.), core operator and user-focused operator costs, and cluster-level communication costs (pipeline parallelism, expert parallelism, and data parallelism, etc.)

[0407] 3. Efficient data analysis and display functions.

[0408] a) The cost evaluation platform provides tools to summarize and analyze batch end-to-end costs, including: changes in core parameters (such as architecture, number of batches, number of sequences, model scale, and number of TP / CP / PP), system performance indicators (such as total time, throughput, TPS, effective computing power, and effective bandwidth), the proportion of various computing power costs (for example, 2D / 1D / SFU / RNG), the proportion of various communication costs (for example, TP / CP / PP / DP / EP), the cost proportion of IO-intensive operators and compute-intensive operators, and the cost proportion of operators that users are concerned about (for example, the cost proportion of operators quantized by FP8).

[0409] b) The cost evaluation platform provides an online data display tool that can filter data and draw various charts through a graphical interface to assist users in data analysis.

[0410] Fig.24 FIG. 2 shows a structural diagram of a distributed system modeling device provided by an embodiment of the present invention. Fig.24 As shown, the device comprises:

[0411] A basic information acquisition module 2410 is used to acquire benchmark description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model;

[0412] Among them, the benchmark description information includes the operator information of each model operator in the target large model, the input and output tensor dimension information, the distributed segmentation information and the candidate insertion position of the communication operator between each model operator.

[0413] The distributed scheme determination module 2420 is used to determine the target distributed parallel scheme when deploying the target large model in a distributed cluster from multiple alternative distributed parallel schemes based on the benchmark description information and cluster information; wherein each alternative distributed parallel scheme includes the number of parallels under at least one distributed strategy.

[0414] The system performance analysis result acquisition module 2430 is used to obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme from three dimensions: operator-level cost, device-level cost and cluster-level cost.

[0415] The technical solution of the embodiment of the present invention obtains benchmark description information of the target large model to be distributed modeled and cluster information of the distributed cluster used to deploy the target large model; determines the target distributed parallel scheme when the target large model is deployed in the distributed cluster from multiple alternative distributed parallel schemes according to the benchmark description information and the cluster information; obtains the technical means of the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme from three dimensions of operator-level cost, device-level cost and cluster-level cost according to the target distributed parallel scheme, decomposes the entire distributed modeling process into independent parts such as the construction of benchmark description information, the exploration of the target distributed parallel scheme and the evaluation of distributed parallel performance, realizes a standardized distributed modeling scheme, and then can realize the rapid introduction of benchmark tests, provide control over model scaling and independent calculation graphs, match appropriate distributed parallel schemes according to model characteristics, quickly evaluate the performance of different distributed parallel schemes, and evaluate the effects of different model optimization schemes.

[0416] Based on the above embodiments, the basic information acquisition module 2410 may include:

[0417] An operator information labeling unit is used to parse the target source code file of the target large model to obtain each model operator matching the target large model, and label the operator information of each model operator;

[0418] A distributed segmentation information labeling unit, used to label at least one target model operator among the multiple model operators with distributed segmentation information according to the distributed configuration information matching the target large model;

[0419] A candidate insertion position determination unit, used to derive input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of operator information and / or distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator;

[0420] The benchmark description information generation unit is used to generate a forward calculation graph and a reverse calculation graph as the benchmark description information of the target large model according to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0421] On the basis of the above embodiments, the method may further include: a target source code file acquisition unit, which is used to:

[0422] Before parsing the source code file of the target large model to obtain each model operator matching the target large model, the original source code file corresponding to the target large model is received through a pytorch compatible interface, wherein the pytorch compatible interface is used to achieve full compatibility with the programming interface of the pytorch AI framework based on the python language;

[0423] According to preset modeling requirements, at least one of simplification, scaling and expansion operations is performed on the original source code file to obtain a target source code file of a target large model.

[0424] Based on the above embodiments, a distributed configuration information acquisition unit may be further included, which is used to:

[0425] Before marking distributed segmentation information for at least one target model operator among the multiple model operators according to the distributed configuration information matching the target large model, receiving distributed configuration information input by a user through a distributed configuration interface;

[0426] The distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information.

[0427] On the basis of the above embodiments, the distributed segmentation information may include: at least one piece of distributed strategy information, each piece of distributed strategy information including a distributed strategy, segmentation semantics and segmentation parameters;

[0428] The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism;

[0429] The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension, and segmentation by value;

[0430] Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

[0431] Based on the above embodiments, the candidate insertion position determination unit may be specifically used for:

[0432] According to the operator semantics in the annotation results of the operator information and the tensor dimension of at least one input tensor defined in the computing task, the input and output tensor dimension information of all model operators is derived;

[0433] According to the labeling result of the distributed segmentation information of at least one target model operator, the distributed segmentation information of all model operators is derived.

[0434] Based on the above embodiments, the candidate insertion position determination unit may be further specifically configured to:

[0435] According to the semantic information of each target model operator and the annotation result of the distributed segmentation information of each target model operator, the first derivation rule is used to derive the completion result of the distributed segmentation information of each target model operator;

[0436] According to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second derivation rule is adopted to derive the distributed segmentation information of the remaining model operators.

[0437] Based on the above embodiments, the candidate insertion position determination unit may be further specifically configured to:

[0438] Based on the distributed segmentation information of each model operator, the segmentation semantics of the weight tensor of each model operator is derived;

[0439] According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependency relationship between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one;

[0440] According to the dependency relationship between the model operators, multiple model operator dependency pairs are obtained;

[0441] According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, the target model operator dependency pair to be inserted into the communication operator is obtained, and the operator type of the communication operator to be inserted into the target model operator dependency pair is determined;

[0442] The position between the previous model operator and the next model operator in each target model operator dependency pair is determined as a candidate insertion position for the communication operator.

[0443] Based on the above embodiments, the distributed solution determination module 2420 may specifically include:

[0444] The candidate distributed parallel solution acquisition unit is used to determine the search set corresponding to each distributed strategy according to the benchmark test description information, the computing tasks and cluster information matching the target large model, and obtain multiple candidate distributed parallel solutions according to each search set;

[0445] A single-device computing time calculation unit is used to generate a single-device computing graph of the target large model under each alternative distributed parallel scheme according to the benchmark test description information, and calculate the single-device computing time of each single-device computing graph, wherein the single-device computing graph includes a communication operator;

[0446] An end-to-end total time consumption calculation unit, used to calculate the end-to-end total time consumption corresponding to each candidate distributed parallel solution according to the single-device calculation consumption of each single-device calculation graph;

[0447] The target distributed parallel scheme determination unit is used to determine the target distributed parallel scheme when deploying the target large model in the distributed cluster from among the alternative distributed parallel schemes according to the calculated calculation time of each single device and the end-to-end total time.

[0448] Based on the above embodiments, a storage requirement calculation unit may be further included, which is used to:

[0449] According to the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster, the weight-related storage requirements and activation-related storage requirements matching the target large model are calculated;

[0450] Accordingly, an invalid distributed parallel solution filtering unit may also be included, which is used to:

[0451] After obtaining a plurality of candidate distributed parallel schemes according to each search set, the single device storage requirements corresponding to each candidate distributed parallel scheme are evaluated respectively according to the weight-related storage requirements and the activation-related storage requirements;

[0452] Among the candidate distributed parallel solutions, invalid distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster are filtered out.

[0453] Based on the above embodiments, the invalid distributed parallel solution filtering unit can be further used to:

[0454] Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed;

[0455] The weight-related storage requirements of a single device are calculated based on the weight-related storage requirements, the current tensor parallel number, and the current pipeline parallel number.

[0456] The activation-related storage requirements of a single device are calculated based on the activation-related storage requirements, the current tensor parallel number, and the current context parallel number.

[0457] According to the storage requirements related to the weight of a single device and the storage requirements related to the activation of a single device, the storage requirements of a single device corresponding to the target alternative distributed parallel solution are calculated.

[0458] Based on the above embodiments, the alternative distributed parallel solution acquisition unit can be specifically used to implement at least one of the following:

[0459] Determine a tensor parallel search set corresponding to the tensor parallel according to the number of high-bandwidth interconnect devices supported by the distributed cluster in the cluster information;

[0460] According to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, a pipeline parallel search set corresponding to the pipeline parallelism is determined;

[0461] Extracting hardware information from cluster information, and determining the maximum number of context parallelisms based on the hardware information; determining context length information based on computing tasks matching the target large model; determining a context parallel search set corresponding to the context parallelism based on the maximum number of context parallelisms and the context length information;

[0462] According to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallel;

[0463] Calculate the maximum number of data parallelism according to the total number of cluster devices in the cluster information, the determined tensor parallel search set, the pipeline parallel search set, and the context parallel search set, and determine the data parallel search set corresponding to the data parallelism according to the maximum number of data parallelism;

[0464] The distributed strategy required to determine the search space is determined by the distributed segmentation information included in the benchmark description information.

[0465] Based on the above embodiments, the single device computing time consumption calculation unit can be specifically used for:

[0466] Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed;

[0467] According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline;

[0468] According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated;

[0469] According to the input and output tensor dimension information of each model operator and the candidate insertion position of the communication operator between each model operator contained in the benchmark description information, the parallel segmentation dimension of the input and output tensor of each model operator in the single-device computing operator graph is determined, and the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived;

[0470] According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

[0471] On the basis of the above embodiments, the single device computing time consumption calculation unit can be further specifically used for:

[0472] Get the target single device calculation graph currently being processed;

[0473] According to the cluster information of the distributed cluster, the first type of computing time of each model operator and the second type of computing time of each communication operator in the target single device computing graph are calculated;

[0474] According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single-device computing time of the target single-device computing graph is calculated.

[0475] On the basis of the above embodiments, the single device computing time consumption calculation unit can be further specifically used for:

[0476] Get the currently processed target model operator in the target single-device computation graph;

[0477] Calculate the input and output memory access time of the target model operator based on the total input tensor size and total output tensor size of the target model operator, as well as the device memory access bandwidth and memory access efficiency;

[0478] Calculate the computation time of the target model operator based on the floating-point operations of the target model operator, the number of computation operations per second of the device, and the device computation efficiency;

[0479] The first type of computing time of the target model operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target model operator, and the computing time of the target model operator.

[0480] On the basis of the above embodiments, the single device computing time consumption calculation unit can be further specifically used for:

[0481] Obtain the target communication operator currently being processed in the target single-device computation graph, and obtain the parallel type of the target communication operator;

[0482] If the parallel type is tensor parallel, the second type of computing time of the target communication operator is calculated according to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator;

[0483] If the parallel type is pipeline parallelism, the second type of computing time consumption of the target communication operator is calculated according to the communication volume of the pipeline parallel communication operator, the pipeline parallel communication bandwidth and the pipeline parallel communication efficiency.

[0484] On the basis of the above embodiments, the single device computing time consumption calculation unit can be further specifically used for:

[0485] If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth, and data parallel communication efficiency, and the single-device computing time of the target single-device computing graph is calculated based on the first-category computing time, the second-category computing time, and the gradient reduction time.

[0486] If the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device computing time of the target single-device computing graph is calculated based on the first type of computing time and the second type of computing time.

[0487] Based on the above embodiments, the system performance analysis result acquisition module 2430 may specifically include:

[0488] An original computational graph extraction unit, used to extract the original computational graph of the target large model from the benchmark test description information of the target large model;

[0489] A standard single-device computing graph segmentation unit, used to segment the original computing graph according to the target distributed parallel scheme to obtain at least one standard single-device computing graph;

[0490] An operator-level cost determination unit is used to call a pre-built operator cost calculation model according to the basic description information of each operator in the standard single-device computing graph and the architecture information of the distributed cluster, and determine the operator-level cost of each operator in the single-device computing graph;

[0491] A cluster-level cost determination unit is used to calculate the device-level cost corresponding to each standard single-device calculation graph according to the operator-level cost of each operator, and combine each device-level cost according to each distributed strategy to obtain a cluster-level cost;

[0492] The system performance analysis unit is used to obtain the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme based on the operator level cost, the device level cost and the cluster level cost.

[0493] On the basis of the above embodiments, it may also include a pre-processing unit, which is used to perform optimization strategy pre-processing on the original calculation graph before dividing the original calculation graph according to the target distributed parallel scheme to obtain at least one standard single-device calculation graph; and / or

[0494] It may also include a post-processing unit for performing optimization strategy post-processing on at least one of the operator-level cost, the device-level cost and the cluster-level cost before obtaining the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme.

[0495] On the basis of the above embodiments, the optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy;

[0496] The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

[0497] Based on the above embodiments, a cost model building unit may be further included, which is used to:

[0498] The operator cost calculation model is constructed using at least one of the following methods:

[0499] Disassemble the large-grained operators to obtain the combination of ATen operators;

[0500] Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators;

[0501] Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results;

[0502] The operator cost calculation model includes: a model operator cost calculation model and a communication operator cost calculation model.

[0503] Based on the above embodiments, the standard single-device computing graph segmentation unit can be specifically used for:

[0504] According to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme, the original calculation graph is split and processed to obtain at least one standard single-device calculation graph;

[0505] According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, the parallel splitting dimensions of the input and output tensors of each model operator are marked in the standard single-device computation graph, and communication operators are added to each standard single-device computation graph based on the marking results.

[0506] Based on the above embodiments, the cluster-level cost determination unit may be specifically used for:

[0507] According to at least one of the data parallel number, the pipeline parallel number, and the expert parallel number, a cluster communication cost between computing devices deployed with different standard single-device computing graphs is calculated;

[0508] The cluster-level cost is calculated based on the device-level cost and the cluster communication cost.

[0509] Based on the above embodiments, the system performance analysis unit can be specifically used for:

[0510] The operator-level cost of each operator, the IO cost and computation cost of each operator in the forward computation phase, the reverse BPI computation phase, and the reverse BPK computation phase, and the determination result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results;

[0511] The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results;

[0512] The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

[0513] Based on the above embodiments, the system performance analysis unit may be further specifically used for:

[0514] Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators;

[0515] The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

[0516] Based on the above embodiments, a visual display unit may be further included, which is used for:

[0517] After obtaining the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme based on the operator-level cost, device-level cost and cluster-level cost, in response to the target data item selected by the user in the system performance analysis results through the human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

[0518] The distributed system modeling device provided in the embodiment of the present invention can execute the distributed system modeling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0519] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0520] Fig.25 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0521] like Fig.25As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0522] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0523] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, for example, executing the method for distributed system modeling as described in any embodiment of the present invention.

[0524] That is, obtaining the benchmark description information of the target large model to be distributed modeled and the cluster information of the distributed cluster used to deploy the target large model;

[0525] The benchmark description information includes operator information of each model operator in the target large model, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of communication operators between each model operator;

[0526] According to the benchmark description information and the cluster information, a target distributed parallel scheme is determined from a plurality of candidate distributed parallel schemes when the target large model is deployed in the distributed cluster; wherein each candidate distributed parallel scheme includes a parallel number under at least one distributed strategy;

[0527] According to the target distributed parallel scheme, the system performance analysis results are obtained from three dimensions: operator-level cost, device-level cost, and cluster-level cost when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme.

[0528] In some embodiments, the method for modeling a distributed system as described in any embodiment of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for modeling a distributed system as described in any embodiment of the present invention described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for modeling a distributed system as described in any embodiment of the present invention in any other appropriate manner (for example, by means of firmware).

[0529] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0530] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0531] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0532] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0533] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0534] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0535] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0536] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed system modeling method, characterized in that: include: Obtaining benchmark description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model; The benchmark description information includes operator information of each model operator in the target large model, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of communication operators between each model operator; According to the benchmark description information and the cluster information, a target distributed parallel scheme is determined from a plurality of candidate distributed parallel schemes when the target large model is deployed in the distributed cluster; wherein each candidate distributed parallel scheme includes a parallel number under at least one distributed strategy; According to the target distributed parallel scheme, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme from three dimensions: operator-level cost, device-level cost, and cluster-level cost; Among them, according to the target distributed parallel scheme, the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme are obtained from three dimensions: operator-level cost, device-level cost, and cluster-level cost, including: Extracting the original computational graph of the target large model from the benchmark description information of the target large model; Split the original computation graph according to the target distributed parallel scheme to obtain at least one standard single-device computation graph; Based on the basic description information of each operator in the standard single-device computing graph and the architecture information of the distributed cluster, the pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-device computing graph; According to the operator-level cost of each operator, the device-level cost corresponding to each standard single-device computation graph is calculated, and the device-level costs are combined according to each distributed strategy to obtain the cluster-level cost; Based on the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme.

2. The method according to claim 1, characterized in that Obtain the benchmark test description information of the target large model to be distributed modeled, including: According to the target source code file of the target large model, parse and obtain each model operator matching the target large model, and annotate the operator information of each model operator; According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is marked with distributed segmentation information; According to the labeling results of the operator information and / or distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, and according to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined; According to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators, the forward calculation graph and the reverse calculation graph are generated as the benchmark description information of the target large model.

3. The method according to claim 2, characterized in that Before parsing the source code file of the target large model to obtain the model operators matching the target large model, the following steps are also included: Receiving the original source code file corresponding to the target large model through a pytorch compatible interface, wherein the pytorch compatible interface is used to achieve full compatibility with the programming interface of the pytorch AI framework based on the python language; According to preset modeling requirements, at least one of simplification, scaling and expansion operations is performed on the original source code file to obtain a target source code file of a target large model.

4. The method according to claim 2, characterized in that: Before marking distributed segmentation information for at least one target model operator among the multiple model operators according to the distributed configuration information matching the target large model, the method further includes: Receive distributed configuration information input by the user through the distributed configuration interface; The distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information.

5. The method according to claim 2, characterized in that: The distributed segmentation information includes: at least one piece of distributed strategy information, each piece of distributed strategy information includes a distributed strategy, segmentation semantics, and segmentation parameters; The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism; The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension, and segmentation by value; Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

6. The method according to any one of claims 2 to 5, characterized in that: Based on the annotation results of operator information and / or distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, including: According to the operator semantics in the annotation results of the operator information and the tensor dimension of at least one input tensor defined in the computing task, the input and output tensor dimension information of all model operators is derived; According to the labeling result of the distributed segmentation information of at least one target model operator, the distributed segmentation information of all model operators is derived.

7. The method according to claim 6, characterized in that According to the labeling result of the distributed segmentation information of at least one target model operator, the distributed segmentation information of all model operators is derived, including: According to the semantic information of each target model operator and the annotation result of the distributed segmentation information of each target model operator, the first derivation rule is used to derive the completion result of the distributed segmentation information of each target model operator; According to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second derivation rule is adopted to derive the distributed segmentation information of the remaining model operators.

8. The method according to any one of claims 2 to 5, characterized in that: According to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined, including: Based on the distributed segmentation information of each model operator, the segmentation semantics of the weight tensor of each model operator is derived; According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependency relationship between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one; According to the dependency relationship between the model operators, multiple model operator dependency pairs are obtained; According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, the target model operator dependency pair to be inserted into the communication operator is obtained, and the operator type of the communication operator to be inserted into the target model operator dependency pair is determined; The position between the previous model operator and the next model operator in each target model operator dependency pair is determined as a candidate insertion position for the communication operator.

9. The method according to claim 1, characterized in that: According to the benchmark description information and cluster information, a target distributed parallel scheme is determined from multiple candidate distributed parallel schemes when deploying the target large model in a distributed cluster, including: According to the benchmark description information, the computing tasks and cluster information matching the target large model, the search sets corresponding to each distributed strategy are determined, and according to each search set, multiple candidate distributed parallel solutions are obtained; According to the benchmark description information, generate a single-device calculation graph of the target large model under each alternative distributed parallel scheme, and calculate the single-device calculation time of each single-device calculation graph, where the single-device calculation graph contains communication operators; According to the single-device computing time of each single-device computing graph, calculate the end-to-end total time corresponding to each alternative distributed parallel solution; According to the calculated computing time of each single device and the end-to-end total computing time, a target distributed parallel scheme is determined from various alternative distributed parallel schemes when deploying the target large model in a distributed cluster.

10. The method according to claim 9, characterized in that Also includes: According to the model information of the target large model and the deployment scenario information of the target large model on the distributed cluster, the weight-related storage requirements and activation-related storage requirements matching the target large model are calculated; Accordingly, after obtaining multiple candidate distributed parallel solutions according to each search set, it also includes: According to the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated respectively; Among the candidate distributed parallel solutions, invalid distributed parallel solutions whose single-device storage requirements are greater than the single-device standard storage capacity in the distributed cluster are filtered out.

11. The method according to claim 10, characterized in that Based on the weight-related storage requirements and activation-related storage requirements, the single-device storage requirements corresponding to each alternative distributed parallel scheme are evaluated separately, including: Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed; The weight-related storage requirements of a single device are calculated based on the weight-related storage requirements, the current tensor parallel number, and the current pipeline parallel number. The activation-related storage requirements of a single device are calculated based on the activation-related storage requirements, the current tensor parallel number, and the current context parallel number. According to the storage requirements related to the weight of a single device and the storage requirements related to the activation of a single device, the storage requirements of a single device corresponding to the target alternative distributed parallel solution are calculated.

12. The method according to claim 9, characterized in that According to the benchmark description information, the computing tasks and cluster information matching the target large model, a search set corresponding to each distributed strategy is determined, including at least one of the following: Determine a tensor parallel search set corresponding to the tensor parallel according to the number of high-bandwidth interconnect devices supported by the distributed cluster in the cluster information; According to the preset maximum value of pipeline parallelism and the number of model layers of the target large model, a pipeline parallel search set corresponding to the pipeline parallelism is determined; Extracting hardware information from the cluster information, and determining the maximum number of context parallelisms according to the hardware information; Determine context length information based on a computational task that matches a target large model; Determine a context parallel search set corresponding to the context parallelism according to the maximum context parallelism quantity and context length information; According to the distribution method of each expert model on different devices, determine the expert parallel search set corresponding to the expert parallel; Calculate the maximum number of data parallelism according to the total number of cluster devices in the cluster information, the determined tensor parallel search set, the pipeline parallel search set, and the context parallel search set, and determine the data parallel search set corresponding to the data parallelism according to the maximum number of data parallelism; The distributed strategy required to determine the search space is determined by the distributed segmentation information included in the benchmark description information.

13. The method according to claim 9, characterized in that Based on the benchmark description information, generate a single-device computation graph of the target large model under each alternative distributed parallel scheme, including: Obtain the current tensor parallel number, the current pipeline parallel number, and the current context parallel number in the target candidate distributed parallel scheme currently being processed; According to the current number of parallel pipelines and the preset load distribution strategy, the original calculation graph corresponding to the target large model is converted into a pipeline calculation graph on each pipeline; According to the pipeline calculation graph, the current tensor parallel number and the current context parallel number, a single-device calculation operator graph is calculated; According to the input and output tensor dimension information of each model operator and the candidate insertion position of the communication operator between each model operator contained in the benchmark description information, the parallel segmentation dimension of the input and output tensor of each model operator in the single-device computing operator graph is determined, and the communication operator to be inserted in the single-device computing operator graph and the insertion position of the communication operator are derived; According to the insertion position, the communication operator is inserted into the single-device computing operator graph to obtain a single-device computing graph under the target alternative distributed parallel solution.

14. The method according to claim 9, characterized in that Calculate the single-device computing time of each single-device computing graph, including: Get the target single device calculation graph currently being processed; According to the cluster information of the distributed cluster, the first type of computing time of each model operator and the second type of computing time of each communication operator in the target single device computing graph are calculated; According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single-device computing time of the target single-device computing graph is calculated.

15. The method according to claim 14, characterized in that According to the cluster information of the distributed cluster, the first type of computing time of each model operator in the target single device computing graph is calculated, including: Get the currently processed target model operator in the target single-device computation graph; Calculate the input and output memory access time of the target model operator based on the total input tensor size and total output tensor size of the target model operator, as well as the device memory access bandwidth and memory access efficiency; Calculate the computation time of the target model operator based on the floating-point operations of the target model operator, the number of computation operations per second of the device, and the device computation efficiency; The first type of computing time of the target model operator is calculated according to the predetermined operator core program startup time, the input and output memory access time of the target model operator, and the computing time of the target model operator.

16. The method according to claim 14, characterized in that According to the cluster information of the distributed cluster, the second type of computing time of each communication operator in the target single device computing graph is calculated, including: Obtain the target communication operator currently being processed in the target single-device computation graph, and obtain the parallel type of the target communication operator; If the parallel type is tensor parallel, the second type of computing time of the target communication operator is calculated according to the device interconnection mode, the number of tensor parallels, the tensor parallel interconnection bandwidth, and the operator type of the target communication operator; If the parallel type is pipeline parallelism, the second type of computing time consumption of the target communication operator is calculated according to the communication volume of the pipeline parallel communication operator, the pipeline parallel communication bandwidth and the pipeline parallel communication efficiency.

17. The method according to claim 14, characterized in that According to the deployment scenario information of the target large model on the distributed cluster, the first type of computing time and the second type of computing time, the single device computing time of the target single device computing graph is calculated, including: If the deployment scenario information of the target large model on the distributed cluster is a model training scenario, the gradient reduction time is calculated based on the model gradient reduction communication volume, data parallel communication bandwidth, and data parallel communication efficiency, and the single-device computing time of the target single-device computing graph is calculated based on the first-category computing time, the second-category computing time, and the gradient reduction time. If the deployment scenario information of the target large model on the distributed cluster is a model inference scenario, the single-device computing time of the target single-device computing graph is calculated based on the first type of computing time and the second type of computing time.

18. The method according to claim 1, characterized in that Before the original computation graph is segmented according to the target distributed parallel scheme to obtain at least one standard single-device computation graph, the method further includes: Performing optimization strategy pre-processing on the original computation graph; and / or Before obtaining the system performance analysis result when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme according to the operator-level cost, the device-level cost and the cluster-level cost, it also includes: An optimization strategy post-processing is performed on at least one of the operator-level cost, the device-level cost, and the cluster-level cost.

19. The method according to claim 18, characterized in that The optimization strategy pre-processing is used to optimize the original calculation graph by adopting at least one optimization strategy, and the optimization strategy post-processing is used to optimize at least one type of cost by adopting at least one optimization strategy; The optimization strategies include strategies that only need to be introduced in the optimization strategy pre-processing, strategies that only need to be introduced in the optimization strategy post-processing, and strategies that need to be introduced in both the optimization strategy pre-processing and the optimization strategy post-processing.

20. The method according to any one of claims 18-19, characterized in that: The operator cost calculation model is constructed using at least one of the following methods: Disassemble the large-grained operators to obtain the combination of ATen operators; Provides a pre-built modeling programming interface to enable operator cost modeling for code-free operators; Correct the operator cost calculation model through system measured data, hardware simulation data or system-level chip modeling results; The operator cost calculation model includes: a model operator cost calculation model and a communication operator cost calculation model.

21. The method according to claim 1, characterized in that The original computation graph is split according to the target distributed parallel scheme to obtain at least one standard single-device computation graph, including: According to at least one of the pipeline parallel number and the expert parallel number in the target distributed parallel scheme, the original calculation graph is split and processed to obtain at least one standard single-device calculation graph; According to at least one of the tensor parallel number, sequence parallel number, and context parallel number in the target distributed parallel scheme, the parallel splitting dimensions of the input and output tensors of each model operator are marked in the standard single-device computation graph, and communication operators are added to each standard single-device computation graph based on the marking results.

22. The method according to claim 21, characterized in that Combine the device-level costs according to the distribution strategies to get the cluster-level costs, including: According to at least one of the data parallel number, the pipeline parallel number, and the expert parallel number, a cluster communication cost between computing devices deployed with different standard single-device computing graphs is calculated; The cluster-level cost is calculated based on the device-level cost and the cluster communication cost.

23. The method according to claim 1, characterized in that Based on the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel solution, including: The operator-level cost of each operator, the IO cost and computational cost of each operator in the forward computation phase, the reverse BPI computation phase, and the reverse BPK computation phase, and the determination result of whether each operator is an IO-intensive operator or a computation-intensive operator are used as the operator dimension performance analysis results; The device-level costs, the operator cost statistics of different computing power types obtained in each computing device, the operator cost statistics of each target operator obtained in each computing device, and the operator cost statistics of each type of communication operator obtained in each computing device are used as the device dimension performance analysis results; The system-level cost, the statistical values ​​of operator costs of different computing power types obtained from all computing devices, the statistical values ​​of operator costs of each target operator obtained from all computing devices, and the statistical values ​​of communication costs under various types of distributed strategies obtained from all computing devices are taken as the system dimension performance analysis results.

24. The method according to claim 23, characterized in that According to the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel solution, including: Based on the operator-level cost, device-level cost, and cluster-level cost, calculate at least one system performance indicator, the cost ratio of different computing power types, the cost ratio of different types of communication operators, the cost ratio of IO-intensive operators and computing-intensive operators, and the cost ratio of the target operator in all operators; The system performance indicators include total time consumption, throughput, number of transactions processed per second, effective computing power and effective bandwidth.

25. The method according to claim 24, characterized in that: After obtaining the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme based on the operator-level cost, the device-level cost and the cluster-level cost, it also includes: In response to a target data item selected by a user in the system performance analysis result through a human-computer interaction interface, a visualization chart matching the target data item is drawn and displayed.

26. A distributed system modeling device, characterized in that: include: A basic information acquisition module, used to acquire benchmark test description information of a target large model to be distributed modeled and cluster information of a distributed cluster for deploying the target large model; The benchmark description information includes operator information of each model operator in the target large model, input and output tensor dimension information, distributed segmentation information, and candidate insertion positions of communication operators between each model operator; A distributed scheme determination module is used to determine a target distributed parallel scheme when deploying a target large model in a distributed cluster from a plurality of candidate distributed parallel schemes according to the benchmark description information and the cluster information; wherein each candidate distributed parallel scheme includes a parallel number under at least one distributed strategy; The system performance analysis result acquisition module is used to obtain the system performance analysis results when the target large model is deployed in the distributed cluster according to the target distributed parallel scheme from three dimensions: operator-level cost, device-level cost and cluster-level cost; Among them, the system performance analysis result acquisition module is specifically used for: Extracting the original computational graph of the target large model from the benchmark description information of the target large model; Split the original computation graph according to the target distributed parallel scheme to obtain at least one standard single-device computation graph; Based on the basic description information of each operator in the standard single-device computing graph and the architecture information of the distributed cluster, the pre-built operator cost calculation model is called to determine the operator-level cost of each operator in the single-device computing graph; According to the operator-level cost of each operator, the device-level cost corresponding to each standard single-device computation graph is calculated, and the device-level costs are combined according to each distributed strategy to obtain the cluster-level cost; Based on the operator-level cost, device-level cost, and cluster-level cost, obtain the system performance analysis results when the target large model is deployed in a distributed cluster according to the target distributed parallel scheme.

27. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the distributed system modeling method according to any one of claims 1 to 25.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the distributed system modeling method according to any one of claims 1 to 25 when executed.

29. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the distributed system modeling method according to any one of claims 1 to 25.

Images