Benchmark test generation method, device and equipment for distributed system modeling

Abstract

The present invention discloses a benchmark test generation method, device, equipment and medium for distributed system modeling. The method includes: obtaining each model operator of the target large model according to the target source code file of the target large model, and marking the operator information of each model operator; marking the distributed segmentation information of the target model operator according to the distributed configuration information; deriving the input and output tensor dimension information and distributed segmentation information of each model operator according to the marking results of the operator information and / or the distributed segmentation information, and determining the candidate insertion position of the communication operator according to the distributed segmentation information of each model operator; generating the benchmark test description information of the target large model according to the operator information, input and output tensor dimension information, distributed segmentation information and candidate insertion position of the communication operator corresponding to each model operator. The technical solution of the embodiment of the present invention can construct a benchmark test that effectively supports distributed modeling.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a benchmark test generation method, device, equipment and medium for distributed system modeling. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC) and large language models (LLM), deep learning models are undergoing rapid iteration and updating, which is accompanied by the continuous expansion of model scale and the demand for larger clusters.

[0003] In order to cope with the distributed deployment of large models, various innovative technologies continue to emerge. In the process of AI (Artificial Intelligence) chip and system design, it is crucial to efficiently and conveniently build benchmarks for large language model loads. Summary of the invention

[0004] The embodiments of the present invention provide a benchmark test generation method, apparatus, device and medium for distributed system modeling, which flexibly and at low cost construct a benchmark test that effectively supports distributed modeling.

[0005] According to an aspect of an embodiment of the present invention, a benchmark test generation method for distributed system modeling is provided, comprising:

[0006] According to the target source code file of the target large model to be modeled for the distributed system, the model operators matching the target large model are parsed and the operator information of each model operator is annotated;

[0007] According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is marked with distributed segmentation information;

[0008] According to the labeling results of the operator information and / or distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, and according to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined;

[0009] According to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators, the forward calculation graph and the reverse calculation graph are generated as the benchmark description information of the target large model.

[0010] According to another aspect of an embodiment of the present invention, there is also provided a benchmark test generation device for distributed system modeling, comprising:

[0011] The operator information annotation module is used to parse the target source code file of the target large model to be modeled for the distributed system, obtain each model operator that matches the target large model, and annotate the operator information of each model operator;

[0012] A segmentation information labeling module, used to label distributed segmentation information for at least one target model operator among the multiple model operators according to distributed configuration information matching the target large model;

[0013] A candidate insertion position determination module is used to derive the input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of the operator information and / or distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator;

[0014] The benchmark description information generation module is used to generate a forward calculation graph and a reverse calculation graph as the benchmark description information of the target large model according to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the method described in any embodiment of the present invention.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the benchmark test generation method for distributed system modeling described in any embodiment of the present invention when executed.

[0020] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the benchmark test generation method for distributed system modeling as described in any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention is to parse the target source code file of the target large model to be modeled in a distributed system to obtain each model operator matching the target large model, and mark the operator information of each model operator; mark the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model; derive the input and output tensor dimension information and distributed segmentation information of all model operators according to the marking results of the operator information and / or the distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator according to the distributed segmentation information of each model operator; according to the operator information, input and output tensor dimension information, distributed segmentation information, and communication operator corresponding to each model operator, the candidate insertion position of the communication operator between each model operator is determined; The operator's candidate insertion position between the operators of each model is used to generate a forward calculation graph and a reverse calculation graph as an implementation method for the benchmark description information of the target large model. Based on the source code file of the large model, the operator information, input and output tensor dimension information, distributed segmentation information of each model operator in the large model, and the candidate insertion position of the communication operator between the operators of each model can be determined efficiently and at low cost, and then the benchmark description information of the target large model is generated. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box computational graph control capabilities to provide support for the modeling of engineering optimization solutions such as computational graph optimization, communication efficiency optimization, and low-bit quantization.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a benchmark test generation method for distributed system modeling provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flowchart of a benchmark test generation method for distributed system modeling provided according to Embodiment 2 of the present invention;

[0026] Figure 3It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by group;

[0027] Figure 4 It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by dimension;

[0028] Figure 5 It is a segmentation schematic diagram applicable to an embodiment of the present invention when the segmentation semantics is segmentation by numerical value;

[0029] Figure 6 is a flowchart of a benchmark test generation method for distributed system modeling provided by Embodiment 3 of the present invention;

[0030] Figure 7 It is a schematic diagram of a computational graph applicable to an embodiment of the present invention and used to illustrate a derivation process of input and output tensor dimension information and distributed segmentation information of all model operators;

[0031] Figure 8 It is a configuration diagram of a multi-dimensional distributed solution applicable to an embodiment of the present invention;

[0032] Fig. 9 It is a schematic diagram of a computation graph obtained after distributed segmentation according to a multi-dimensional distributed solution applicable to an embodiment of the present invention;

[0033] Fig.10 is a schematic diagram of the structure of a benchmark test generation device for distributed system modeling provided according to a fourth embodiment of the present invention;

[0034] Fig.11 It is a structural schematic diagram of an electronic device for implementing a benchmark test generation method for distributed system modeling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] Figure 1 This is a flowchart of a benchmark test generation method for distributed system modeling provided in the first embodiment of the present invention. This embodiment can be applied to the case of constructing a benchmark test that effectively supports distributed modeling. The method can be executed by a benchmark test generation device for distributed system modeling. The device can be implemented in the form of hardware and / or software and can generally be configured in a computing device with data processing functions. Figure 1 As shown, the method includes:

[0039] S110, according to the target source code file of the target large model to be modeled for the distributed system, parse to obtain each model operator matching the target large model, and annotate the operator information of each model operator.

[0040] The target large model may be a deep learning model applied in the fields of image classification, target detection, face recognition, text classification, machine translation or speech recognition, etc. The target large model may be a set pre-trained model.

[0041] Generally speaking, developers can write source code files of large models based on the distributed deep learning framework on the market, and deploy the large models in a specific distributed cluster based on the source code files. The purpose of distributed system modeling is to effectively explore the best distributed solution before deploying a specific large model in a specific distributed cluster. If a benchmark suitable for distributed modeling is constructed efficiently and at low cost before subsequent distributed system modeling, the accuracy and convenience of subsequent distributed modeling can be improved.

[0042] Specifically, the target source code file is generally obtained by setting a programming language, for example, it is written based on the pytorch open source framework. Accordingly, the target source code file can be parsed using a parsing method that matches the programming language to obtain each model operator that matches the target large model. Among them, the model operator can be understood as a computational operator used to implement the set operation logic, such as a square operator or a convolution operator. Of course, while obtaining the above-mentioned model operators, the connection relationship between the model operators can also be parsed, that is, by parsing the target source code file of the target large model, the original calculation graph that matches the target large model can be obtained, and the original calculation graph contains the parsed model operators, and the model operators are interconnected by setting the connection relationship.

[0043] Furthermore, after determining each model operator that matches the target large model, in order to enable efficient subsequent distributed system modeling, in each embodiment of the present invention, detailed operator information is firstly annotated for each model operator in the target large model.

[0044] The operator information that needs to be annotated may include: semantic information of the operator (typically, the operation logic of the operator), operator name, operator type, data type of the operator's input and output data, data format of the operator's input and output data, and storage location of the operator's input and output data (for example, L3 or L4), etc. Furthermore, the operator information may also include the input and output tensor dimension information of the model operator.

[0045] In this embodiment, the operator information of each model operator in the original calculation graph can be annotated in response to the annotation information manually input by the user, or the operator information of each model operator can be automatically annotated by automatically analyzing the pre-imported operator description document.

[0046] S120. According to the distributed configuration information matching the target large model, mark the distributed segmentation information of at least one target model operator among the multiple model operators.

[0047] In this embodiment, the distributed configuration information may include: activation value segmentation setting information, model parameter segmentation setting information, and collective communication setting information.

[0048] Among them, the activation value can be understood as the product of the input and the weight. In a large language model, the continuously updated input is usually called activations. Furthermore, the input flowing into each model operator can be understood as the activation value. Correspondingly, the activation value split setting information can be understood as the setting for distributed splitting of the input of one or more tensor dimensions of one or more model operators.

[0049] Model parameters are variables that control the behavior of machine learning models, generally including weights and biases. Generally speaking, model parameters are used to operate with the input of model operators to obtain new inputs to provide the next model operator. Accordingly, model parameter splitting setting information can be understood as the setting of distributed splitting of model parameters (typically, weights) of one or more tensor dimensions of one or more model operators.

[0050] Collecting communication settings information can be understood as setting one or more distributed strategies for one or more model operators. The distributed strategies can include: Pipeline Parallelism (PP), Tensor Parallelism (TP), Sequence Parallelism (SP), Context Parallelism (CP), Data Parallelism (DP) and Expert Parallelism (EP).

[0051] The distributed configuration information matching the target large model may be input by a user through a configuration setting interface, or may be obtained by parsing a configuration file, which is not limited in this embodiment.

[0052] Based on the distributed configuration information matching the target large model, the distributed segmentation information of at least one target model operator in the target large model can be directly marked. The target model operator can be a model operator directly configured through the distributed configuration information, for example, a core model operator in the large model or a model operator of interest to the user.

[0053] In this embodiment, the distributed splitting information may include at least one piece of distributed strategy information, and each piece of distributed strategy information is used to describe what type of splitting can be performed under what splitting parameters on the target model operator (for example, splitting is performed on the first dimension of the input tensor) to implement what type of distributed strategy (or parallel method).

[0054] S130. Based on the labeling results of the operator information and / or distributed split information, the input and output tensor dimension information and distributed split information of all model operators are derived, and based on the distributed split information of each model operator, the candidate insertion position of the communication operator between each model operator is determined.

[0055] As mentioned above, parsing the target source code file and obtaining the distributed configuration information matching the target large model for annotation may not cover all model operators. Therefore, based on the above annotation results, the embodiment of the present invention further adopts a set derivation strategy to derive the input and output tensor dimension information and distributed segmentation information of all model operators in the target large model.

[0056] Among them, if it is necessary to execute a distributed strategy on a specific model operator in actual application, it may involve different copies of data, or parts of a piece of data are split and executed in different computing devices in a distributed cluster. Therefore, in order to obtain complete data processing results, it is necessary to insert a communication operator for realizing data aggregation function at a specific position between model operators. The types of communication operators may include: All-Reduce communication operator, Reduce-Scatter communication operator, All-Gather communication operator or point-to-point communication operator (P2P), etc.

[0057] In a specific example, if in the original computational graph, model operator A and model operator B are connected end to end, and if it is determined that the output tensor dimension of model operator A is split on the first dimension, while the input tensor of model operator B is not split, then a fully aggregated communication operator needs to be inserted between model operator A and model operator B. Furthermore, the position between model operator A and model operator B can be determined as a candidate insertion position for a communication operator.

[0058] As mentioned above, by parsing the target source code file of the target large model, the original calculation graph matching the target large model can be obtained. Then, the derived input and output tensor dimension information and distributed segmentation information of each model operator can be marked at the matching model operator position in the original calculation graph. At the same time, the candidate insertion position of the determined communication operator between each model operator can also be marked in the original calculation graph.

[0059] S140. Generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model according to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0060] In this embodiment, based on the original calculation graph, the operator information of each model operator in the determined original calculation graph, the input and output tensor dimension information and the distributed segmentation information, and the candidate insertion positions between the model operators of the communication operator that does not exist in the original calculation graph, a forward calculation graph and a reverse calculation graph matching the target large model can be generated as the benchmark description information of the target large model. Based on the benchmark description information of the target large model, the distributed modeling of the target large model can be realized efficiently and conveniently, that is, after the hardware architecture of the distributed cluster is uniquely determined, the distributed parallel solution with the best performance when the target large model is deployed in the distributed cluster can be explored based on the benchmark description information.

[0061] The technical solution of the embodiment of the present invention is to parse the target source code file of the target large model to be modeled in a distributed system to obtain each model operator matching the target large model, and mark the operator information of each model operator; mark the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model; derive the input and output tensor dimension information and distributed segmentation information of all model operators according to the marking results of the operator information and / or the distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator according to the distributed segmentation information of each model operator; according to the operator information, input and output tensor dimension information, distributed segmentation information, and communication operator corresponding to each model operator, the candidate insertion position of the communication operator between each model operator is determined; The operator's candidate insertion position between the operators of each model is used to generate a forward calculation graph and a reverse calculation graph as an implementation method for the benchmark description information of the target large model. Based on the source code file of the large model, the operator information, input and output tensor dimension information, distributed segmentation information of each model operator in the large model, and the candidate insertion position of the communication operator between the operators of each model can be determined efficiently and at low cost, and then the benchmark description information of the target large model is generated. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box computational graph control capabilities to provide support for the modeling of engineering optimization solutions such as computational graph optimization, communication efficiency optimization, and low-bit quantization.

[0062] On the basis of the above embodiments, after generating the forward calculation graph and the reverse calculation graph as the benchmark test description information of the target large model, the following may also be included:

[0063] Obtaining cluster description information of a distributed cluster of a target large model to be deployed; wherein the cluster description information includes: the number of cabinets included in the distributed cluster, the number of servers included in each cabinet, and the number of computing devices included in each server;

[0064] According to the cluster description information and the benchmark test description information, searching for an optimal distributed solution when mapping the target large model to the distributed cluster;

[0065] According to the optimal distributed solution, a distributed partitioning calculation graph matching the target large model is generated, and based on the distributed partitioning calculation graph, a performance evaluation is performed on the running effect of the target large model on the distributed cluster.

[0066] Specifically, based on the cluster description information and the benchmark test description information, multiple optional distributed solutions can be quickly determined, for example, which distributed strategy set to adopt, and the specific number of parallel operations in each distributed strategy. Furthermore, according to a preset evaluation method (for example, actual measurement or evaluation of the time consumption of each single device and the end-to-end time consumption of the distributed cluster), the optimal distributed solution for mapping the target large model to the distributed cluster can be obtained from each optional distributed solution. Furthermore, based on the optimal distributed solution, the performance of the running effect of the target large model on the distributed cluster can be evaluated by actual measurement.

[0067] Through the above settings, the computing performance of the target large model when using the optimal distributed solution on distributed clusters with different hardware specifications can be evaluated. Then, the most suitable distributed cluster specification and the optimal distributed solution under the distributed cluster can be selected for each target large model, and the target large model can be deployed in the distributed cluster of the distributed cluster specification according to the optimal distributed solution to achieve efficient training or inference calculations.

[0068] Embodiment 2

[0069] Figure 2 A flowchart of a benchmark test generation method for distributed system modeling provided in the second embodiment of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, the method for obtaining the target source code file and distributed configuration information is specifically limited.

[0070] Correspondingly, such as Figure 2 As shown, the method may include:

[0071] S210. Receive the original source code file corresponding to the target large model through a pytorch compatible interface.

[0072] The pytorch compatible interface is used to achieve full compatibility with the programming interface based on the python language.

[0073] In each embodiment of the present invention, a Benchmark generator front end is constructed to receive a target source code file, parse the target source code file of a target large model to be used for distributed system modeling to obtain each model operator matching the target large model, and annotate the operator information of each model operator, receive distributed configuration information matching the target large model, and annotate the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model.

[0074] At present, the distributed technology of large models is still in a period of rapid development and iteration. Various distributed deep learning frameworks emerge in an endless stream, and the distributed technologies and external interfaces supported by various frameworks vary greatly. The benchmark construction system needs to be compatible with different framework interfaces to quickly complete the modeling of existing models and distributed technologies, and provide more general interface expansion capabilities, so that new models and new technologies can be modeled forward-lookingly before the framework supports them. Given that the current mainstream distributed frameworks are all developed based on pytorch, the Benchmark generator front end provides full compatibility with the pytorch programming interface, and models composed of pytorch modules such as torch.nn.module or torch.nn.funtinal can be imported at low cost.

[0075] The above implementation method realizes the control of abstract granularity. That is, the pytorch compatible interface is used instead of the computational graph exported by the framework for modeling. This is mainly because the computational graph exported by the framework is usually for the purpose of compilation, optimization, and execution. For example, the exported onnx computational graph is for execution in the inference engine. Its structure is relatively complex and not suitable for human understanding and analysis, and it cannot express distributed segmentation. The use of the pytorch compatible interface can take into account both low-cost model import and flexible control of abstract granularity. For example, the modeling granularity can be controlled on core large operators such as flash-attention (Fast and Memory-EfficientExact Attention with IO-Awareness, a new attention algorithm with IO awareness and both fast and memory-efficient), rather than the dozens of small operators that are converted at runtime.

[0076] In addition, the above implementation can support distributed modeling. Among them, the distributed framework is in a period of rapid iteration, and there is no unified standard for interfaces, implementations, and even terminology. Therefore, combined with the compatible support for the relatively stable pytorch interface and the interface abstraction for distributed settings, it can take into account the low-cost import of model implementation and the maximum support for distributed technology modeling.

[0077] S220. According to preset modeling requirements, perform at least one of simplification, scaling and expansion operations on the original source code file to obtain a target source code file of a target large model.

[0078] Through the above settings, low-cost model import and expansion can be achieved. Among them, the framework exports the calculation graph usually depends on the execution process of the model, especially dynamic models such as MoE will change at runtime, and the execution process of large models requires a large-scale distributed system, and the cost of exporting the calculation graph is huge. The torch-compatible interface proposed in the present invention can be executed independently of hardware, and can facilitate the simplification, scaling and expansion of the model, and make it possible to model for forward-looking research.

[0079] S230. According to the target source code file of the target large model to be modeled for the distributed system, parse and obtain each model operator that matches the target large model, and annotate the operator information of each model operator.

[0080] S240. Receive distributed configuration information input by a user through a distributed configuration interface.

[0081] In an embodiment of the present invention, the Benchmark generator front end also provides a distributed configuration interface, which abstracts distributed settings such as model parameter segmentation, input activation value segmentation, and collective communication. Distributed implementations of different frameworks can be imported into benchmark modeling through this interface.

[0082] The distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information.

[0083] S250. According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is labeled with distributed segmentation information.

[0084] In an optional implementation of the embodiment of the present invention, the distributed segmentation information may include: at least one piece of distributed strategy information, each piece of distributed strategy information includes a distributed strategy, segmentation semantics, and segmentation parameters.

[0085] The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism. The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension and segmentation by value; the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or segmentation by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, segmentation by group or segmentation by value.

[0086] In each embodiment of the present invention, the inventor innovatively defines four types of tensor splitting semantics, namely, no splitting, splitting by group, splitting by dimension, and splitting by value. Among them, no splitting (Not Split), denoted by N. The tensor is not split, which means that it is a complete copy on all computing devices. Group Split (Group Split), denoted by G(i), divides the tensor into different groups. The tensor is complete on all computing devices (also called computing cards) in the group to which it belongs, and does not exist on other groups. i indicates that the tensor belongs to the i-th group. Dim Split (Dim Split), denoted by D(i). The tensor is equally divided on all computing devices along the i-th dimension, and each computing device stores a tensor of size 1 / n (n is the total size of data on dimension i). Value Split (Value Split), denoted by V. The size of the tensor on each computing device card is the same as the original tensor size, but the sum of the tensor values ​​on all computing devices is equal to the original tensor.

[0087] Taking the deployment of the simplest 2D tensor on a single computing device as an example, the specific implementation methods of group splitting, dimension splitting, and value splitting are described in detail. Figure 3 A segmentation schematic diagram when the segmentation semantics applicable to an embodiment of the present invention is segmentation by group is shown in FIG. Figure 4 A schematic diagram of segmentation when the segmentation semantics applicable to an embodiment of the present invention is segmentation by dimension is shown in FIG. Figure 5 A segmentation schematic diagram when the segmentation semantics applicable to an embodiment of the present invention is segmentation by numerical value is shown in FIG.

[0088] like Figure 3 As shown in , if the splitting strategy D(0) (also referred to as D0) is applied to a tensor of dimension 4x4, only one row of the tensor is deployed on each device from Device 0 to Device 3. Figure 4 As shown in the figure, if the splitting strategy G(1) (also referred to as G1) is applied to a tensor of dimension 3x3, then on Device 0 to Device 3, the tensor is deployed on only one of the devices ( Figure 4 In the example, it is deployed on Device 1, but not on other devices. Figure 5 As shown in the figure, if the splitting strategy V is applied to a tensor of dimension 3x3, the sum of the tensor values ​​on the dimension 4x4 is equal to the tensor value of the original tensor.

[0089] Based on the above four tensor segmentation semantics, the current six mainstream distributed strategies (also called 6D parallel strategies) can be expressed as one or several applications of the above four segmentation semantics.

[0090] Specifically, pipeline parallelism can apply segmentation G to divide model operators into groups of different pipelines.

[0091] Tensor parallelism, sequence parallelism, or context parallelism can be applied: 1. Split D, divide the activation value or weight tensor of the model operator equally according to a certain dimension within the TP group; 2. Split V, the reduce dimension of the model operator input is split, resulting in the output value containing only part of the complete output.

[0092] Data parallelism can be applied by splitting D(0) to split the batch dimension of the model operator input (activation value).

[0093] Expert parallelism can be applied: 1. Split D and split V, deploy the expert model on all computing devices in the expert group, and split the input activation value or weight tensor of the expert model in the dimension. If the reduce dimension is split, the output only contains part of the complete output; 2. Split G, deploy the expert model on different computing devices according to the group.

[0094] Correspondingly, each piece of distributed strategy information can be marked in the form of distributed strategy name_segmentation semantic name_segmentation parameter.

[0095] For example, a distributed strategy information of the form TP_D_1 indicates that the model operator uses tensor parallelism, and the parallel method is to split the first dimension of the weight tensor by dimension; or a distributed strategy information of the form PP_G_1 indicates that the model operator uses pipeline parallelism, and the parallel method is to divide the input and output tensors of the model operator into the first pipeline group.

[0096] In this embodiment, by parsing the input distributed configuration information, the distributed segmentation information can be annotated in each target model operator of the target large model. Specifically, the target model operator is generally a proper subset of the target large model, and in special cases, it can be all model operators included in the target large model. The annotation of the distributed segmentation information performed in the target model operator can be the annotation of all 6D distributed strategies, or it can be the annotation of one or a few distributed strategies in 6D, and this embodiment does not limit this.

[0097] S260. Based on the labeling results of the operator information and / or distributed split information, the input and output tensor dimension information and distributed split information of all model operators are derived, and based on the distributed split information of each model operator, the candidate insertion position of the communication operator between each model operator is determined.

[0098] S270. Generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model based on the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0099] The technical solution of the embodiment of the present invention uses the pytorch compatible interface to achieve full compatibility with the pytorch AI framework programming interface based on the python language, which can take into account both low-cost import of models and flexible control of abstract granularity, can be executed independently of hardware, and can facilitate simplification, scaling and expansion of models, and make modeling for forward-looking research possible. In addition, combined with compatible support for the relatively stable pytorch interface and interface abstraction for distributed settings, it can take into account both low-cost import of model implementation and maximum support for distributed technology modeling.

[0100] Embodiment 3

[0101] Figure 6 A flowchart of a benchmark test generation method for distributed system modeling provided in Example 3 of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, the operation of "deriving the input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of operator information and / or distributed segmentation information, and determining the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator" is concretized.

[0102] Correspondingly, such as Figure 6 As shown, the method specifically includes:

[0103] S610. According to the target source code file of the target large model to be modeled for the distributed system, parse and obtain each model operator that matches the target large model, and annotate the operator information of each model operator.

[0104] S620. According to the distributed configuration information matching the target large model, mark the distributed segmentation information of at least one target model operator among the multiple model operators.

[0105] S630. According to the operator semantics in the annotation results of the operator information and the tensor dimension of at least one input tensor defined in the computing task, derive the input and output tensor dimension information of all model operators.

[0106] Specifically, after the computing task of the target large model is determined, the tensor dimensions of one or more input tensors initially input to the target large model can be clarified. Taking the above input tensors as the starting point and combining the operator semantics of each model operator, the input and output tensor dimension information of each model operator in the target large model can be determined one by one.

[0107] S640. Derive distributed segmentation information of all model operators according to the labeling result of the distributed segmentation information of at least one target model operator.

[0108] In an optional implementation of this embodiment, according to the labeling result of the distributed segmentation information of at least one target model operator, deriving the distributed segmentation information of all model operators may include:

[0109] According to the semantic information of each target model operator and the annotation results of the distributed segmentation information of each target model operator, the first deduction rule is used to derive the completion result of the distributed segmentation information of each target model operator; according to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second deduction rule is used to derive the distributed segmentation information of the remaining model operators.

[0110] In a specific example, serial parallelism and tensor parallelism need to be used together. Therefore, after the distributed strategy information of serial parallelism is marked in the target model operator, the distributed strategy information of tensor parallelism of the target model operator can be completed.

[0111] After determining that a certain target model operator has completed the labeling of all distributed strategy information, the distributed segmentation information of other model operators before and after the target model operator can be further labeled according to preset rules. For example, if the previous unlabeled model operator of the target model operator has similar calculation logic to the target model operator, or belongs to the same type of operator, the distributed segmentation information of the target model operator can be directly used as the distributed segmentation information of the previous unlabeled model operator. Alternatively, the distributed segmentation strategies that each model operator can support can be pre-specified, and based on the distributed segmentation information of the target model operator, the distributed segmentation information of the previous unlabeled model operator can be effectively filtered out.

[0112] S650. According to the distributed segmentation information of each model operator, derive the segmentation semantics of the weight tensor of each model operator.

[0113] Generally speaking, some model operators need to calculate the matching input activation values ​​with the corresponding weight parameters. Therefore, if the input activation values ​​of a model operator are distributedly split, the weight parameters of the model operator also need to be distributedly split accordingly. Alternatively, the user can pre-enter the model parameter split setting information in the distributed configuration information to specify the split semantics of the weight tensor of which model operator or operators. Then, the split semantics of the weight tensor of each model operator can be finally derived.

[0114] S660. According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependency relationship between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one.

[0115] In this embodiment, different segmentation mapping relationships can be established for different types of model operators. In a specific example, Table 1 shows the segmentation mapping relationship between the input tensor, the weight tensor, and the output tensor based on the Matmul model operator.

[0116] Table 1

[0117]

[0118] Correspondingly, the figure shows a schematic diagram of a computational graph for illustrating the derivation process of distributed segmentation information of all model operators applicable to an embodiment of the present invention. Specifically, for each model operator, based on the segmentation description of the input tensor and combined with the computational semantics of the model operator, the segmentation of the output tensor of the model operator in the distributed system can be derived as shown in Table 1.

[0119] For the Matmul_1 model operator, if the input tensor is specified to be unsplit, and the weight tensor W_1 is specified by the user to be split by dim1 (D1), then according to the semantic output tensor B of the Matmul (matrix multiplication) model operator, it can be determined to be split by dim1 (D1); for the lower-level matmul_2 model operator, if the user specifies that the weight W_2 is split by dim0, then the output tensor C of the matmul_2 model operator can be determined to be split by value V; for the layernorm model operator, the user specifies that the input is split by dim0, which does not match the split of the matmul_2 output, then according to the communication operator, the logic V->D is inserted, and the ReduceScatter communication operator needs to be inserted (this communication operator is not in the original calculation graph, which will be described in detail later). According to the predetermined splitting mapping corresponding to the Layernorm model operator (not shown in the embodiment of the present invention), it can be deduced that the output tensor D is split according to dim0; the Gelu model operator has no user-specified splitting, and according to the input tensor D and the predetermined splitting mapping of the gelu model operator (not shown in the embodiment of the present invention), it can be deduced that the output tensor E is split according to dim0; the Add model operator has no user-specified splitting, and according to the input tensor E and the splitting mapping of the Add model operator, it can be deduced that the output tensor F is split according to dim0.

[0120] S670. Obtain multiple model operator dependency pairs according to the dependency relationship between the model operators.

[0121] Furthermore, in the original computation graph, two adjacent model operators form a model operator dependency pair. Figure 7 , Matmul_1 and Matmul_2, as well as Matmul_2 and Layernorm are all model operator dependency pairs.

[0122] S680. According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, obtain the target model operator dependency pair into which the communication operator is to be inserted, and determine the operator type of the communication operator to be inserted into the target model operator dependency pair.

[0123] S690: Determine the position between the previous model operator and the next model operator in each target model operator dependency pair as a candidate insertion position for the communication operator.

[0124] As mentioned above, after performing tensor parallelization on a computational operator, if you need to obtain the complete output tensor of the computational operator, you need to perform all-gather collective communication on the split output tensors on all computing devices where the computational operator is deployed. Furthermore, you need to add an all-gather communication operator after the computational operator.

[0125] In various embodiments of the present invention, the inventors generate the communication operator insertion logic shown in Table 2 by analyzing the tensor conversion logic under different segmentation strategies.

[0126] In a specific example, in combination with Table 2, if it is determined that in a model operator dependency pair, the parallel split dimension of the output tensor of the previous model operator belongs to D(i), and the parallel split dimension of the input tensor of the next model operator belongs to N, then by querying the third item in Table 2, it is indicated that a communication operator of type All-Gather needs to be inserted between the previous model operator and the next model operator.

[0127] Table 2

[0128]

[0129] S6100. Generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model based on the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0130] The technical solution of the embodiment of the present invention can be based on the source code file of the large model, and can efficiently and cost-effectively determine the operator information, input and output tensor dimension information, distributed segmentation information and candidate insertion positions of communication operators between each model operator in the large model, and then generate benchmark description information of the target large model. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box computational graph control capabilities to provide support for the modeling of engineering optimization solutions such as computational graph optimization, communication efficiency optimization, and low-bit quantization.

[0131] Multi-dimensional distributed solution configuration scenario

[0132] For the sake of understanding, after constructing the benchmark description information of the target large model, the method of implementing the multi-dimensional distributed solution configuration under a specific distributed cluster based on the benchmark description information is described in detail. Figure 8 A schematic diagram of a multi-dimensional distributed solution applicable to an embodiment of the present invention is shown in FIG. Fig. 9 A schematic diagram of a computational graph obtained after distributed segmentation according to a multi-dimensional distributed solution applicable to an embodiment of the present invention is shown in FIG.

[0133] Specifically, in actual task scenarios, different levels of computing device groups are usually used for distributed deployment. The operator distributed labeling, tensor segmentation description and derivation methods proposed in the embodiments of the present invention can be easily extended to multi-dimensional computing device groups.

[0134] For example, in Figure 8 The left side shows the benchmark description information of the target large model, and the right side shows the hardware architecture of the distributed cluster to be deployed. Combined with the benchmark description information of the target large model, the distributed parallel solution with the best performance is determined to perform tensor parallelism and sequence parallelism between the computing devices in each server, pipeline parallelism between the servers in each cabinet, and data parallelism between multiple cabinets.

[0135] Among them, the benchmark test description information includes four model operators. According to the annotation of their respective distributed segmentation information and the grouping of computing devices, the following conclusions can be obtained.

[0136] 1. MatMul_1: According to the annotations TP_D_1, SP_D_1, and DP_D_0, it is determined that TP and SP are performed between the 8 computing devices in the server, and DP is performed between the two cabinets. The shape of the input activation tensor A is divided into [b / 2, s / 8, h], and the shape of the weight tensor W1 is divided into [h, h / 8]. According to the division of the matmul operator, it is deduced that the shape of its output tensor B is divided into [b / 2, s / 8, h / 8]. According to PP_G_0, the MatMul_1 model operator and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0137] MatMul_2: The shape of the input activation tensor B is [b / s, s / 8, h / 8], which satisfies the operator segmentation annotations TP_D_0, SP_D_2, and DP_D_0, so no communication operator is required between MatMul_1 and MatMul_2. The shape of the weight tensor W2 is segmented into [h / 8, h], and it can be deduced that the shape of the output tensor C is segmented into [b, s / 8, h(V)]. According to PP_G_0, the MatMul_2 model operators and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0138] Layernorm: According to the segmentation annotations SP_D_1 and TP_N, the input activation tensor needs to be segmented into [b, s / 8, h], but the tensor C is segmented into [b, s / 8, h(V)], and the communication operator reduce-scatter needs to be inserted. The output tensor D is segmented into [b, s / 8, h]. According to PP_G_0, the Layernorm model operators and input and output tensors are deployed on the first pipeline group, that is, servers 0-3 in each cabinet.

[0139] Gelu: The distributed splits are labeled as SP_D1 and TP_N, and the input activation tensor D needs to be split into [b, s / 8, h], which can be satisfied, and there is no need to insert communication operators for TP / SP. However, according to PP_G_1, the Gelu model operators and input and output tensors are deployed on the second pipeline grouping, that is, on servers 4-7 in each cabinet, which is different from the deployment location of the first three model operators, so it is necessary to insert a P2P communication operator.

[0140] After completing the distributed derivation and device mapping, the distributed segmented computation graph is obtained as follows: Fig. 9 As shown, combined Fig. 9 The computational graph can efficiently and conveniently realize the deployment of the target large model on a distributed cluster of a specific hardware architecture, and then accurately and effectively evaluate the performance of the deployment result, such as the computing time of a single device or the end-to-end computing time.

[0141] Among them, according to the distributed segmentation and annotation information of key operators, the technical solution of the embodiment of the present invention can automatically generate a segmentation strategy for related tensors, and derive the distributed segmentation of the entire network and the location of inserting communication operators, and then obtain a benchmark description containing distributed segmentation strategies and communication operator information as input for searching for the optimal distributed parallel solution, so as to search for the best TP / SP / CP / DP / PP / EP combination.

[0142] Embodiment 4

[0143] Fig.10 This is a schematic diagram of the structure of a benchmark test generation device for distributed system modeling provided by the fourth embodiment of the present invention. Fig.10 As shown, the device includes: an operator information labeling module 1010, a segmentation information labeling module 1020, a candidate insertion position determination module 1030 and a benchmark test description information generation module 1040, wherein:

[0144] The operator information labeling module 1010 is used to parse the target source code file of the target large model to be modeled for the distributed system, obtain each model operator that matches the target large model, and label the operator information of each model operator;

[0145] The segmentation information labeling module 1020 is used to label the distributed segmentation information of at least one target model operator among the multiple model operators according to the distributed configuration information matching the target large model;

[0146] A candidate insertion position determination module 1030 is used to derive input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of operator information and / or distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator;

[0147] The benchmark description information generation module 1040 is used to generate a forward calculation graph and a reverse calculation graph as benchmark description information of the target large model according to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

[0148] The technical solution of the embodiment of the present invention is to parse the target source code file of the target large model to be modeled in a distributed system to obtain each model operator matching the target large model, and mark the operator information of each model operator; mark the distributed segmentation information of at least one target model operator among multiple model operators according to the distributed configuration information matching the target large model; derive the input and output tensor dimension information and distributed segmentation information of all model operators according to the marking results of the operator information and / or the distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator according to the distributed segmentation information of each model operator; according to the operator information, input and output tensor dimension information, distributed segmentation information, and communication operator corresponding to each model operator, the candidate insertion position of the communication operator between each model operator is determined; The operator's candidate insertion position between the operators of each model is used to generate a forward calculation graph and a reverse calculation graph as an implementation method for the benchmark description information of the target large model. Based on the source code file of the large model, the operator information, input and output tensor dimension information, distributed segmentation information of each model operator in the large model, and the candidate insertion position of the communication operator between the operators of each model can be determined efficiently and at low cost, and then the benchmark description information of the target large model is generated. Based on the above benchmark description information, it has the ability to quickly model the distributed segmentation of the large model, ensuring the needs of distributed deployment solution search, optimization and evaluation. At the same time, it can also provide white-box computational graph control capabilities to provide support for the modeling of engineering optimization solutions such as computational graph optimization, communication efficiency optimization, and low-bit quantization.

[0149] On the basis of the above embodiments, a target source code file acquisition module may also be included, which is used to:

[0150] Before parsing the source code file of the target large model to be modeled for the distributed system to obtain each model operator matching the target large model, the original source code file corresponding to the target large model is received through a pytorch compatible interface, wherein the pytorch compatible interface is used to achieve full compatibility with the programming interface of the pytorch AI framework based on the python language;

[0151] According to preset modeling requirements, at least one of simplification, scaling and expansion operations is performed on the original source code file to obtain a target source code file of a target large model.

[0152] Based on the above embodiments, a distributed configuration information receiving module may be further included, which is used to:

[0153] Before marking distributed segmentation information for at least one target model operator among the multiple model operators according to the distributed configuration information matching the target large model, receiving distributed configuration information input by a user through a distributed configuration interface;

[0154] The distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information.

[0155] On the basis of the above embodiments, the distributed segmentation information may include: at least one piece of distributed strategy information, each piece of distributed strategy information including a distributed strategy, segmentation semantics and segmentation parameters;

[0156] The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism;

[0157] The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension, and segmentation by value;

[0158] Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

[0159] Based on the above embodiments, the candidate insertion position determination module 1030 may specifically include:

[0160] An input and output tensor dimension information derivation unit, used to derive input and output tensor dimension information of all model operators according to operator semantics in the annotation result of operator information and tensor dimension of at least one input tensor defined in the computing task;

[0161] The distributed segmentation information derivation unit is used to derive the distributed segmentation information of all model operators according to the labeling result of the distributed segmentation information of at least one target model operator.

[0162] Based on the above embodiments, the distributed segmentation information derivation unit can be specifically used for:

[0163] According to the semantic information of each target model operator and the annotation result of the distributed segmentation information of each target model operator, the first derivation rule is used to derive the completion result of the distributed segmentation information of each target model operator;

[0164] According to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second derivation rule is adopted to derive the distributed segmentation information of the remaining model operators.

[0165] Based on the above embodiments, the candidate insertion position determination module 1030 may be further used to:

[0166] Based on the distributed segmentation information of each model operator, the segmentation semantics of the weight tensor of each model operator is derived;

[0167] According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependency relationship between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one;

[0168] According to the dependency relationship between the model operators, multiple model operator dependency pairs are obtained;

[0169] According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, the target model operator dependency pair to be inserted into the communication operator is obtained, and the operator type of the communication operator to be inserted into the target model operator dependency pair is determined;

[0170] The position between the previous model operator and the next model operator in each target model operator dependency pair is determined as a candidate insertion position for the communication operator.

[0171] Based on the above embodiments, a performance evaluation module may also be included, which is used to:

[0172] After generating the forward calculation graph and the reverse calculation graph as the benchmark description information of the target large model, obtaining the cluster description information of the distributed cluster of the target large model to be deployed; wherein the cluster description information includes: the number of cabinets included in the distributed cluster, the number of servers included in each cabinet, and the number of computing devices included in each server;

[0173] According to the cluster description information and the benchmark test description information, searching for an optimal distributed solution when mapping the target large model to the distributed cluster;

[0174] According to the optimal distributed solution, a distributed partitioning calculation graph matching the target large model is generated, and based on the distributed partitioning calculation graph, a performance evaluation is performed on the running effect of the target large model on the distributed cluster.

[0175] The benchmark test generation device for distributed system modeling provided by the embodiment of the present invention can execute the benchmark test generation method for distributed system modeling provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0177] Embodiment 5

[0178] Fig.11 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0179] like Fig.11 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0180] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as executing the benchmark generation method for distributed system modeling as described in various embodiments of the present invention.

[0182] In some embodiments, the benchmark test generation method for distributed system modeling as described in the various embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the benchmark test generation method for distributed system modeling as described in the various embodiments of the present invention described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the benchmark test generation method for distributed system modeling as described in the various embodiments of the present invention by any other appropriate means (for example, by means of firmware). That is:

[0183] According to the target source code file of the target large model to be modeled for the distributed system, the model operators matching the target large model are parsed and the operator information of each model operator is annotated;

[0184] According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is marked with distributed segmentation information;

[0185] According to the labeling results of the operator information and / or distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, and according to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined;

[0186] According to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators, the forward calculation graph and the reverse calculation graph are generated as the benchmark description information of the target large model.

[0187] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0189] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0192] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0193] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0194] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A benchmark generation method for distributed system modeling, characterized in that: include: According to the target source code file of the target large model to be modeled for the distributed system, the model operators matching the target large model are parsed and the operator information of each model operator is annotated; According to the distributed configuration information matching the target large model, at least one target model operator among the multiple model operators is marked with distributed segmentation information; wherein the distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information; According to the labeling results of operator information and distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, and according to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined; According to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators, the forward calculation graph and the reverse calculation graph are generated as the benchmark description information of the target large model.

2. The method according to claim 1, characterized in that Before parsing the source code file of the target large model to be modeled for the distributed system to obtain the model operators matching the target large model, the method further includes: Receiving the original source code file corresponding to the target large model through a pytorch compatible interface, wherein the pytorch compatible interface is used to achieve full compatibility with the programming interface of the pytorch AI framework based on the python language; According to preset modeling requirements, at least one of simplification, scaling and expansion operations is performed on the original source code file to obtain a target source code file of a target large model.

3. The method according to claim 1, characterized in that Before marking distributed segmentation information for at least one target model operator among the multiple model operators according to the distributed configuration information matching the target large model, the method further includes: Receive distributed configuration information input by the user through the distributed configuration interface.

4. The method according to claim 1, characterized in that: The distributed segmentation information includes: at least one piece of distributed strategy information, each piece of distributed strategy information includes a distributed strategy, segmentation semantics, and segmentation parameters; The distributed strategies include pipeline parallelism, tensor parallelism, sequence parallelism, context parallelism, data parallelism and expert parallelism; The segmentation semantics include no segmentation, segmentation by group, segmentation by dimension, and segmentation by value; Among them, the segmentation semantics corresponding to pipeline parallelism include segmentation by group; the segmentation semantics corresponding to tensor parallelism, sequence parallelism and context parallelism include segmentation by dimension or by value; the segmentation semantics corresponding to data parallelism include segmentation by dimension; the segmentation semantics corresponding to expert parallelism include segmentation by dimension, by group or by value.

5. The method according to any one of claims 1 to 4, characterized in that: Based on the annotation results of operator information and distributed segmentation information, the input and output tensor dimension information and distributed segmentation information of all model operators are derived, including: According to the operator semantics in the annotation results of the operator information and the tensor dimension of at least one input tensor defined in the computing task, the input and output tensor dimension information of all model operators is derived; According to the labeling result of the distributed segmentation information of at least one target model operator, the distributed segmentation information of all model operators is derived.

6. The method according to claim 5, characterized in that According to the labeling result of the distributed segmentation information of at least one target model operator, the distributed segmentation information of all model operators is derived, including: According to the semantic information of each target model operator and the annotation result of the distributed segmentation information of each target model operator, the first derivation rule is used to derive the completion result of the distributed segmentation information of each target model operator; According to the completed distributed segmentation information of each target model operator and the dependency relationship between the model operators, the second derivation rule is adopted to derive the distributed segmentation information of the remaining model operators.

7. The method according to any one of claims 1 to 4, characterized in that: According to the distributed segmentation information of each model operator, the candidate insertion position of the communication operator between each model operator is determined, including: Based on the distributed segmentation information of each model operator, the segmentation semantics of the weight tensor of each model operator is derived; According to the operator semantics of each model operator, the segmentation semantics of the weight tensor of each model operator, and the dependency relationship between the model operators, the segmentation semantics of the input tensor and the segmentation semantics of the output tensor of each model operator are derived one by one; According to the dependency relationship between the model operators, multiple model operator dependency pairs are obtained; According to the segmentation semantics of the output tensor of the previous model operator and the segmentation semantics of the input tensor of the next model operator in each model operator dependency pair, the target model operator dependency pair to be inserted into the communication operator is obtained, and the operator type of the communication operator to be inserted into the target model operator dependency pair is determined; The position between the previous model operator and the next model operator in each target model operator dependency pair is determined as a candidate insertion position for the communication operator.

8. The method according to claim 4, characterized in that After generating the forward calculation graph and the reverse calculation graph as the benchmark description information of the target large model, it also includes: Obtaining cluster description information of a distributed cluster of a target large model to be deployed; wherein the cluster description information includes: the number of cabinets included in the distributed cluster, the number of servers included in each cabinet, and the number of computing devices included in each server; According to the cluster description information and the benchmark test description information, searching for an optimal distributed solution when mapping the target large model to the distributed cluster; According to the optimal distributed solution, a distributed partitioning calculation graph matching the target large model is generated, and based on the distributed partitioning calculation graph, a performance evaluation is performed on the running effect of the target large model on the distributed cluster.

9. A benchmark test generation device for distributed system modeling, characterized in that: include: The operator information annotation module is used to parse the target source code file of the target large model to be modeled for the distributed system, obtain each model operator that matches the target large model, and annotate the operator information of each model operator; A segmentation information labeling module, used to label at least one target model operator among multiple model operators with distributed segmentation information according to distributed configuration information matching the target large model; wherein the distributed configuration information includes: activation value segmentation setting information, model parameter segmentation setting information and collective communication setting information; A candidate insertion position determination module is used to derive the input and output tensor dimension information and distributed segmentation information of all model operators based on the labeling results of the operator information and the distributed segmentation information, and determine the candidate insertion position of the communication operator between each model operator based on the distributed segmentation information of each model operator; The benchmark description information generation module is used to generate a forward calculation graph and a reverse calculation graph as the benchmark description information of the target large model according to the operator information corresponding to each model operator, the input and output tensor dimension information, the distributed segmentation information, and the candidate insertion positions of the communication operator between the model operators.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the benchmark test generation method for distributed system modeling according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the benchmark test generation method for distributed system modeling according to any one of claims 1 to 8 when executed.

Images