Optimization method, device and equipment for distributed marking of computational graph and medium

Abstract

The invention provides a calculation graph distributed mark optimization method and device, equipment and a medium, and relates to the technical field of computers, in particular to artificial intelligence, deep learning and distributed training technologies. According to the specific implementation scheme, the method comprises the steps of obtaining a calculation graph of a distributed operation model and an optimization normal form of a target optimization strategy; identifying a corresponding to-be-optimized calculation sub-graph in the calculation graph according to the optimization normal form; adjusting the to-be-optimized calculation sub-graph according to the optimization normal form; and adjusting the tensor distributed mark in the computational graph according to the original tensor distributed mark in the computational graph and the operation logic of the operator. According to the method, the optimization normal form is automatically identified and converted into the calculation sub-graph of various optimization strategies fused into the distributed networking, so that the usability of automatic parallel is ensured, the difficulty and the use cost of using the optimization strategies are reduced, the adaptability and the flexibility of automatic parallel are enhanced, and the efficiency and the performance of distributed training are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence, deep learning and distributed training technology. Background Art

[0002] As the depth of deep neural networks continues to increase, the number of model parameters is also increasing, showing an exponential growth trend, from millions a few years ago to hundreds of billions now. Therefore, large-scale model operations now mostly rely on distributed systems. Distributed operation of models is a technology that decomposes computing tasks and distributes them to multiple computing nodes for parallel execution.

[0003] In order to make the model run in a distributed system composed of multiple computing nodes, it is necessary to build a computational graph of the model. The computational graph includes many operators that conform to the model operation logic. Then, it is necessary to add distributed tags to the input tensors and output tensors of the operators in the computational graph to meet the parallel operation requirements of each operator. The distributed operation of the model can be achieved by single-card networking or multi-card networking. A single card refers to an independent graphics card (GPU). Each physical host can run one or more graphics cards, and each graphics card can also be called a computing node. When networking with multiple cards, data communication between the graphics cards needs to be realized.

[0004] For the distributed operation of the model realized by single-card networking, the distributed marking of the calculation graph is relatively simple and can be added manually. However, when multiple-card networking is used to realize distributed operation, the distributed marking is more complicated, and even some optimization operators cannot be described by distributed marking. Therefore, when adding distributed marking manually, the technical threshold for personnel is high and the workload is large. Summary of the invention

[0005] The present disclosure provides a method, apparatus, device, medium and program product for generating distributed tags of a computational graph, so as to optimize the way of adding distributed tags to the computational graph of a distributed model operation.

[0006] According to one aspect of the present disclosure, a method for optimizing distributed marking of a computational graph is provided, comprising:

[0007] Obtain the computational graph of the distributed running model and the optimization paradigm of the target optimization strategy;

[0008] In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm;

[0009] Adjust the computation subgraph to be optimized according to the optimization paradigm;

[0010] Adjust the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator;

[0011] Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

[0012] According to another aspect of the present disclosure, there is also provided an optimization device for distributed labeling of a computational graph, comprising:

[0013] The graph acquisition module is used to obtain the computational graph of the distributed operation model and the optimization paradigm of the target optimization strategy;

[0014] A graph identification module to be optimized, used to identify the corresponding computation subgraph to be optimized in the computation graph according to the optimization paradigm;

[0015] A graph adjustment module, used for adjusting the computational subgraph to be optimized according to the optimization paradigm;

[0016] A tag adjustment module, used to adjust the tensor distribution tags in the calculation graph according to the original tensor distribution tags in the calculation graph and the operation logic of the operator;

[0017] Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

[0018] According to another aspect of the present disclosure, there is also provided an electronic device, including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the optimization method for distributed marking of computational graphs as described in any one of the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute the optimization method for distributed marking of computational graphs according to any one of the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the optimization method for distributed marking of computational graphs as described in any one of the embodiments of the present disclosure.

[0024] The technical solution of the disclosed embodiment can automatically complete the calculation graph adjustment of the optimization strategy and the derivation change of the distributed marking, reducing the difficulty of manual marking.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 It is a schematic diagram of an optimization method for distributed marking of a computational graph provided according to an embodiment of the present disclosure;

[0028] Figure 2 It is a schematic diagram of another optimization method for distributed marking of computational graphs provided according to an embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram of another method for optimizing distributed marking of a computational graph provided according to an embodiment of the present disclosure;

[0030] Figure 4 It is a schematic diagram of another method for optimizing distributed marking of a computational graph provided according to an embodiment of the present disclosure;

[0031] Figure 5 It is a schematic diagram of automatic parallel distributed marking optimization applicable to the embodiments of the present disclosure;

[0032] Figure 6 It is a schematic diagram of distributed marking optimization implemented by a setting tool software applicable to the embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram of a parallel column splitting embedding vector optimization applicable to the embodiment of the present disclosure;

[0034] Figure 8 is a schematic diagram of a parallel column segmentation probability distribution cross entropy optimization applicable to the embodiments of the present disclosure;

[0035] Fig. 9 It is a structural diagram of an optimization device for distributed marking of a computational graph provided according to an embodiment of the present disclosure;

[0036] Fig.10 It is a block diagram of an electronic device used to implement the optimization method of distributed marking of computational graphs in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] First, the related technologies of the embodiments of the present disclosure are introduced:

[0039] A computational graph is a graph structure that describes the computational process of a deep neural network. A computational subgraph is a part of a complete computational graph that contains part of the computational process of the model. A distributed operation model is a model that splits the model operation into multiple tasks and uses multiple computing nodes to run in parallel. The operation of a model can be either model training or the process of reasoning with a trained model. The computational graph of a distributed operation model can be specifically understood as: representing the computational process of a model in a distributed system in the form of a graph, describing how data flows between different computing nodes, and how various computing tasks work together to complete the training or reasoning process of the entire model. Among them, the operators in the computational graph represent any tasks such as computing or communication, and there is a flow of tensor inputs and outputs between operators.

[0040] Tensors are generalizations of vectors and matrices, and can be divided into input tensors and output tensors according to the processing steps. Input tensors can be input processing to be processed, or they can be data required for operator operation, such as weight parameter matrices. The processing logic of the operator can determine the relationship between the input tensor and the output tensor.

[0041] The placements of a tensor are used to indicate the placement of the distributed tensor in the dimension of the corresponding computing node. There are three optional placements: Replicate, Shard, and Partial. The following are detailed descriptions of the three placements:

[0042] Replicate means that the tensor is kept in full state on all computing nodes.

[0043] Shard (axis) means splitting the tensor along the axis dimension and placing it on different computing nodes. Usually it can include 0-dimensional (splitting by column) and 1-dimensional (splitting by row).

[0044] Partial means that each computing node only has partial values ​​and needs to be restored to the full data through specified reduction operations.

[0045] Among them, row splitting is splitting along the row dimension of the tensor. For example, for a 1k×1k dimensional two-dimensional tensor, the tensor is divided into 4 blocks by row, each block contains 250 rows, but the number of columns of all rows is 1k. In a distributed environment, each row block can be assigned to different computing nodes for processing.

[0046] Column splitting is splitting along the column dimension of the tensor. For example, for a 1k×1k dimensional two-dimensional tensor, the tensor is divided into 4 blocks by column, each block contains 250 columns, but the number of rows of all columns is 1k. In distributed computing, each column block can be assigned to different computing nodes.

[0047] Local segmentation is to divide the tensor into blocks of partial dimensions, each block contains a part of rows and a part of columns, which is suitable for complex multi-dimensional data processing. For example, for a 1k×1k dimensional two-dimensional tensor, the tensor is divided into 25 blocks according to the data of 200×200 dimensions. Preferably, the tensor dimension obtained by local segmentation can still be 1k×1k, only the data corresponding to the 200×200 dimension is retained, and the remaining positions are filled with 0, so that the final result can be obtained by superposition operation after distributed calculation of each local segmented tensor.

[0048] The distribution of tensors directly affects the operation mode of distributed computing and needs to be determined according to the specific computing requirements of the operator. In a single-card network, when the model generates a calculation graph, distributed tags can be added manually, but given the huge workload of marking each input and output tensor, it is usually only marked on key operators, such as large-scale matrix multiplication, convolution operations, or complex, time-consuming and high-communication cost operations. For other parts, the system automatically derives and adjusts the tensor distributed tags in the calculation graph based on the added tensor distributed tags in the calculation graph and the operation logic of the calculation graph.

[0049] Automatic parallelization technology refers to the technology that can automatically determine the parallel processing mode according to the operation logic of the operator in the calculation graph, specifically, it can determine the tensor distributed tags of each operator. A small number of distributed tags can be added manually, and the framework automatically completes the remaining tensor distributed tags and inserts necessary communication operations to achieve distributed training.

[0050] Figure 1 This is a schematic diagram of a method for optimizing distributed marking of a computational graph according to an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the case of automatically applying various optimization strategies based on distributed networking. The method can be executed by an optimization device for distributed marking of a computational graph, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 , the method specifically includes the following:

[0051] S101, obtaining a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy;

[0052] S102, in the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm;

[0053] S103, adjusting the computation subgraph to be optimized according to the optimization paradigm;

[0054] S104, adjusting the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator;

[0055] Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

[0056] In the above solution, the target optimization strategy refers to the strategy that is not used in the original calculation graph but is expected to be optimized. For example, when the calculation graph corresponding to the model of a single card network is deployed to a distributed system of a multi-card network, some parallel strategies that can only be supported by multi-card networks can be used as target optimization strategies and adjusted to the original calculation graph.

[0057] The optimization paradigm of the target optimization strategy can be specifically understood as: requirements and conditions corresponding to a specific optimization strategy, mainly including optimization conditions and optimization operations. Operators that meet the optimization conditions in the calculation graph can be optimized, and the optimization operation is used to specify how to adjust the operators to be optimized to implement the target optimization strategy in the calculation graph. Among them, the target optimization strategy may be an operation strategy with the same operating purpose as the original parallel strategy, but adopts different operators, such as an optimization strategy adopted to reduce the amount of data communication between computing nodes, or to improve the efficiency of parallel processing. The embodiments of the present disclosure do not limit this. For the calculation graph supported by a single-card network, it is generally impossible to consider some more efficient parallel processing strategies that can be implemented on a multi-card network. These parallel processing strategies can be target optimization strategies.

[0058] The computational subgraph to be optimized can be specifically understood as a set of operators in the computational graph that meet the optimization conditions in the optimization paradigm. The optimization conditions can specifically include: operator type, and tensor properties of operator input tensors, such as distributed tags.

[0059] In different networking environments, such as single-card networking and multi-card networking, the data reading and processing logic will be different due to the influence of media and transmission delay. Each model runs independently according to its calculation graph logic, involving multiple operator operations, such as convolution processing, embedded vector conversion, reduction operation, and communication operation, which together constitute the calculation graph of the model. Even if the basic model is the same, the calculation graphs of single-card networking and multi-card networking will be different due to different networking methods. Therefore, in order to optimize performance, it is necessary to adjust the calculation subgraph in the single-card networking according to the specific optimization strategy to ensure that the calculation graph can adapt to different network environments and computing requirements. The calculation graph before adjustment is a model calculation graph supported by single-card networking. The hardware structure of single-card networking is simple, so the operation of users marking tensors in the calculation graph is relatively simple, with low technical threshold and small workload; the automatic parallel derivation technology can automatically run the operator tasks in the calculation graph in parallel in the distributed system of multi-card networking. Furthermore, the technical solution of the disclosed embodiment can also adjust the model calculation graph and the tensor distributed marking therein based on the optimization strategy on the basis of automatic parallelism, and the adjusted calculation graph is a model calculation graph supported by multi-card networking. As a result, the operation mode of the adjusted model calculation graph can adopt the unique parallel strategy supported by multi-card networking, so that the operation mode can better play the hardware advantages of multi-card networking and effectively utilize the hardware device resources of multi-card networking.

[0060] Generally speaking, adjusting the computational subgraph to be optimized according to the optimization paradigm may specifically include: performing insertion and deletion operations of operators in the computational subgraph to be optimized according to the optimization operations in the optimization paradigm to adjust the computational subgraph to be optimized. Optimization operations generally include operators that need to be inserted or deleted, as well as the order of insertion and deletion operations. For example: the computational subgraph to be optimized includes operator op1 (operator, operator 1), which is the optimized object, and the operators that can be inserted include the new operator op1_new, then op1 can be replaced with op1_new. Among them, operators op and op1_new only represent the computational operation links, and do not specify or limit a specific operation. In order to meet the needs of multi-card networking, it may also be necessary to add communication operators, such as communication protocol operators (AllReduce), communication broadcast operators (Broadcast) or communication collection operators (AllGather), etc., to achieve efficient data transmission and synchronization across devices; it is also possible to reduce the amount of data calculation or communication through operator fusion, reduce communication overhead, and improve computing efficiency.

[0061] After the calculation graph is adjusted, this change requires the tensor distribution labels of other operators to be adaptively adjusted to ensure that the tensor distribution labels of the entire calculation graph are correct. Therefore, automatic parallel technology can be used to automatically complete the derivation and adjustment of tensor distribution labels, realize the automatic application of various optimization strategies on the multi-card group network and derive the distribution of tensors.

[0062] The technical solution of the disclosed embodiment utilizes the optimization paradigm to automatically identify and convert operators into a variety of optimization strategies that are integrated into distributed networking, and can automatically derive distributed labels of tensors in parallel, thereby ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0063] In an optional implementation of this embodiment, the optimization method for distributed marking of the computational graph may also set a simple switch mechanism.

[0064] The switch mechanism can be specifically understood as a control parameter, which is used to indicate whether to execute the optimization method of distributed marking of the computational graph, so as to realize the use of distributed marking optimization based on the existing distributed marking through a simple switch, and quickly apply those optimization strategies based on distributed networking. The switch mechanism can be implemented through options in the configuration file, environment variables, command line parameters, constants in the code, check boxes in the user interface, or settings in the database. Through these flexible implementation methods, the switch mechanism provides an intuitive control means for administrators. After turning on this function, the administrator can select the target optimization strategy, so that the solution of the embodiment of the present disclosure can be used to automatically complete the adjustment of the computational graph and the labeling of the distributed marking of the tensor based on the target optimization strategy, ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0065] In a specific example, a part of the computational graph can be analyzed, modified or replaced by a setting tool software with automatic parallel deduction function to achieve a specific optimization goal. First, a specific optimization strategy subgraph matching pattern needs to be implemented in the setting tool software, and when the corresponding subgraph is identified, the corresponding operator is replaced and the distributed mark of the corresponding tensor is adjusted. In the final stage of the setting tool software processing flow, it is also necessary to call the splitting and derivation rules to re-split and derive the entire input computational subgraph to ensure that the distributed marking state of the computational graph remains correct. Among them, the splitting and derivation rules refer to the derivation of the input and output tensor splitting states actually required according to the input tensor state and operator implementation. The use of automated processing not only improves efficiency but also reduces human errors.

[0066] Figure 2It is a schematic diagram of another optimization method for distributed marking of computational graphs provided according to an embodiment of the present disclosure. This embodiment concretizes the operation of "obtaining the computational graph of the distributed operation model" in the above embodiment as follows: "obtaining the computational graph of the model and some tensor distributed markings pre-added in the computational graph; according to the some tensor distributed markings, using automatic parallel computing, deriving the remaining tensor distributed markings in the computational graph, and adding them to the computational graph".

[0067] Accordingly, reference Figure 2 , the method may specifically include:

[0068] S201, obtaining a computation graph of a model and some tensor distributed markers pre-added in the computation graph.

[0069] The pre-added distributed marking of some tensors can be manually added by the user. Specifically, it can be understood as follows: in a distributed computing environment, since the model operation needs to be distributed on multiple computing nodes, in order to optimize performance and resource utilization or to make the parallel strategy meet the user's preset requirements, the distribution method of the tensor (such as row splitting, column splitting, and local splitting, etc.) is clearly specified at some specific locations or key locations of the calculation graph to provide more refined control than the fully automatic splitting strategy. Typically, the distributed marking can be a small number of tensor splitting method markings for some specific locations of the calculation graph supported by the single-card networking.

[0070] S202. According to the partial tensor distributed labels, use automatic parallel computing to derive the remaining tensor distributed labels in the calculation graph and add them to the calculation graph.

[0071] After adding distributed labels for some tensors, the manually specified distributed labels are used as the starting point. According to the distributed labels of known tensors, the networking architecture of computing nodes, and the logical structure of the computational graph, automatic parallelization technology is applied to automatically derive parallel strategies, thereby achieving the derivation and update of the distributed labels of the unlabeled tensors of the operator, so that the distributed labels of the remaining unlabeled tensors can be completed in the computational graph.

[0072] Generally speaking, since the functions and relationships of operators are certain, the segmentation methods of other tensors in the entire computational graph can be deduced through a small number of segmentation marks. If the segmentation methods of some tensors cannot be automatically deduced, it means that the manually added segmentation marks are insufficient, and more segmentation marks need to be added to guide the automatic derivation process.

[0073] S203, obtaining an optimization paradigm of a target optimization strategy;

[0074] S204: In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm.

[0075] S205. Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0076] S206. Adjust the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator.

[0077] In multi-card networking, since more data and operations are involved, manually adding distributed tags is complex and labor-intensive. The implementation of its distributed operation strategy needs to comprehensively consider the segmentation of a single operator, as well as the data flow and dependencies between operators. In addition, the manually added tensor distributed tags in multi-card networking may conflict with other tags because they do not take into account the needs of global optimization and resource allocation, especially in terms of operator distribution and data synchronization.

[0078] Generally speaking, in automatic parallel mode, most optimization strategies can be automatically applied to multi-card networking by identifying the distributed tags marked in single-card networking. However, some special optimization strategies based on multi-card networking contain distributed-specific optimization operators and are associated with multi-card communication operations. Since these operators and communication operations are integrated together, these strategies cannot be used in single-card networking; at the same time, since these optimization strategies use distributed-specific optimization operators, these operators are semantically similar to ordinary operators but are completely different in implementation. Users cannot describe these optimization strategies in single-card networking by using distributed tags to convert the original ordinary operators into distributed-specific optimization operators, that is, they cannot describe these optimization strategies in single-card networking by manually adding distributed tags.

[0079] Optionally, the calculation graph before automatic parallel calculation is a model calculation graph supported by a single-card network, and the calculation graph after automatic parallel calculation is a model calculation graph supported by a multi-card network. The disclosed embodiment can manually add distributed tags based on the distributed tags of the single-card network calculation graph, perform automatic parallel derivation to improve the tensor distributed tags of the entire calculation graph; then use the target optimization strategy to optimize the calculation graph, and update the tensor distributed tags, so that the entire calculation graph is consistent with the application of the corresponding optimization strategy. Based on the existing distributed tags and automatic parallel derivation technology, it is possible to quickly apply those optimization strategies based on multi-card networks.

[0080] The technical solution of the disclosed embodiment, by pre-adding some tensor distributed tags for key operators or special operators in the calculation graph of single-card networking and deriving the addition of remaining tensor distributed tags, and then performing the adjustment of the optimization strategy, realizes the use of automatic parallel derivation tools to handle the addition and optimization of distributed tags in multi-card networking. Such an optimization method can maximize the advantages of automatic parallelization, make better decisions based on the entire calculation graph and network topology, while maintaining human control over key parts, realize simple and efficient distributed training, ensure the ease of use of automatic parallelization, reduce the difficulty and cost of using optimization strategies, enhance the adaptability and flexibility of automatic parallelization, and improve the efficiency and performance of distributed training.

[0081] In an optional implementation of this embodiment, the target optimization strategy corresponds to a distributed optimization strategy supported by multi-card networking.

[0082] Typically, as mentioned above, in view of the complex structure of multi-card networking, the manual addition of distributed markings is labor-intensive and prone to errors. The optimization method for distributed marking of the computational graph proposed in the embodiment of the present disclosure is for the optimization of multi-card networking. Accordingly, the target optimization strategy corresponds to the distributed optimization strategy supported by the multi-card networking, so as to reduce the difficulty and cost of using the multi-card networking optimization strategy and improve the efficiency and performance of distributed training of the multi-card networking. As a result, the distributed optimization strategy of the multi-card networking may not be used in the original computational graph where the distributed marking is manually added, so that the complexity of manual addition is low.

[0083] The optimization paradigm may include the addition of communication operators. In the process of optimizing the multi-card networking calculation graph by applying distributed operation strategies based on the distributed marking of the single-card networking calculation graph, in order to maintain the consistency of the distributed training results, the model parameters need to be frequently synchronized between different nodes. This synchronization operation needs to be implemented through specific communication operators. Since data and model parameters do not need to be synchronized between multiple computing nodes in a single-card network, communication operators are usually not required. Therefore, the calculation subgraph corresponding to the multi-card network includes communication operators between each graphics card / computing node to ensure data synchronization and parameter consistency across computing nodes and to satisfy the computing and communication dependencies between computing nodes.

[0084] Typically, in a multi-card networking environment, in addition to communication operators, the optimized operator graph may also include some unique operators, such as data sharding operators, task scheduling operators, memory management operators, and load balancing operators.

[0085] In an optional implementation of this embodiment, in the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm may include:

[0086] The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

[0087] By traversing each part in the computational graph according to the optimization conditions, the matched data streams have the same composition and contain the computational subgraphs to be optimized of the corresponding operators. The automated traversal method ensures comprehensive identification of optimization opportunities, reduces the need for manual intervention, reduces the complexity and error rate of the optimization work, ensures the ease of use of automatic parallelism, reduces the difficulty and cost of using optimization strategies, and improves the efficiency and performance of distributed training.

[0088] In an optional implementation of this embodiment, the optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine that the matching computation subgraph to be optimized includes:

[0089] Using the operator conditions in the optimization conditions, traversing and matching in the computation graph to determine a matching candidate computation subgraph;

[0090] If, according to the tensor distribution mark of the candidate computation subgraph, it is identified that the distribution mode meets the segmentation requirement in the optimization condition, the candidate computation subgraph is determined as the computation subgraph to be optimized.

[0091] The candidate computational subgraph can be specifically understood as: by traversing each part in the computational graph through the optimization conditions, the matched data flow composition is the same and contains the computational subgraph of the corresponding operator. On the basis of determining the candidate computational subgraph, it is also necessary to determine the matching relationship between the tensor distribution mode in the candidate computational subgraph and the segmentation requirements in the optimization conditions, such as row segmentation, column segmentation or local segmentation. Only when there is a match, the candidate computational subgraph is determined as the computational subgraph to be optimized, which can ensure the consistency and accuracy in distributed computing, improve the performance of distributed training, and avoid inconsistent segmentation and erroneous computing results caused by incorrect identification of the computational subgraph to be optimized.

[0092] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0093] According to the optimization operation in the optimization paradigm, inserting and deleting operations of operators are performed in the computation subgraph to be optimized to adjust the computation subgraph to be optimized.

[0094] The optimization paradigm records the logical structure corresponding to the target optimization strategy, including the type and logical relationship of each operator. After identifying the computational subgraph to be optimized, it is necessary to insert or delete at least one operator in the computational subgraph to be optimized according to the optimization operation in the optimization paradigm to implement the target optimization strategy into the computational graph.

[0095] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator includes:

[0096] Starting from the adjusted operator in the computational graph, forward traversal and / or backward traversal is performed, and according to the original tensor distributed markings of the computational graph and the operation logic of the operator, the tensor distributed markings in the computational graph are adjusted until the tensor distributed markings determined according to the operation logic are consistent with the original tensor distributed markings.

[0097] After adjusting the computational subgraph to be optimized, starting from the adjusted operator in the graph, traverse forward from the input flow of the operator and traverse backward from the output flow of the operator to determine whether the tensor distributed labels of other operators in the front and back flows need to change. During the traversal process, the distributed labels of the corresponding tensors are determined and updated according to the operation logic of the operator. Since the logical structure of the adjusted computational subgraph to be optimized remains unchanged except for the adjusted operator, when it is found that the original tensor distributed label of an operator in the computational graph is consistent with the tensor distributed label determined according to the operation logic, there is no need to continue the traversal update, which can effectively reduce unnecessary calculations, optimize the traversal process, enhance the adaptability and flexibility of automatic parallelization, and improve the efficiency and performance of distributed training.

[0098] In a preferred implementation of this embodiment, the target optimization strategy is an embedding vector (Embedding) layer vocabulary segmentation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator and a communication protocol operator of the word segmentation pattern.

[0099] Embedding vectors are a common feature representation in the model. Input data needs to be converted into embedded vector outputs. Embedding vector conversion operators are usually used to achieve this function. The embedding vector layer vocabulary is a vector vocabulary used to convert input data into embedded vectors. The vector vocabulary contains all words and their corresponding unique indexes. Each word is assigned an index to retrieve the corresponding word vector from the embedding matrix. Assume that the vocabulary is N*M dimensional, N rows represent the number of words, and M columns represent the dimension of the embedding vector of each word. Of course, the vocabulary dimension and the embedding vector dimension can also be exchanged to M*N dimension. The word segmentation mode also refers to segmentation by row, that is, all words are divided into multiple parts. The vector segmentation mode is generally segmented by column, that is, all the embedding vectors of each word are divided into multiple parts. The communication protocol operator is an operation used to exchange and aggregate data between multiple nodes in distributed computing and parallel processing.

[0100] Embedding vector layer vocabulary optimization is an important distributed computing scenario. It solves the problems of high dimensionality and inability to express word relationships in traditional encoding by mapping vocabulary to vectors in high-dimensional space. The optimization method of this scenario can improve the adaptability of automatic parallel distributed training.

[0101] In another preferred implementation of this embodiment, the target optimization strategy is a parallel classification probability and cross entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross entropy operator and a communication protocol operator.

[0102] Cross entropy loss is a standard choice for handling multi-classification problems. It is often used with neural networks that use the softmax activation function. The softmax function converts the raw values ​​output by the neural network into a probability distribution (classification probability), and the cross entropy loss measures the difference between this probability distribution and the true label distribution. Parallel classification probability and cross entropy loss calculation refers to the above calculation process. Parallel means that the classification probability and cross entropy loss calculation of each operator in the calculation graph are calculated in parallel. The parallel classification cross entropy operator can be specifically understood as: an operation that converts the classification probability and cross entropy loss calculation method of an operator into other methods. It corresponds to the parallel classification probability and cross entropy loss calculation strategy. It is based on the existing distributed marking in single-card networking and is the key to automatically apply the parallel classification probability and cross entropy loss calculation strategy in multi-card networking.

[0103] In distributed scenarios, due to the need to process large-scale data sets and complex models, cross entropy loss calculation and optimization of parallel classification probability is an important distributed computing scenario. The optimization method of this scenario not only helps to improve the efficiency and accuracy of model training, but also can effectively synchronize and aggregate data among multiple nodes, enhance the adaptability of automatic parallelism, and improve the efficiency and performance of distributed training.

[0104] Figure 3 It is a schematic diagram of another optimization method for distributed marking of computational graphs provided according to an embodiment of the present disclosure. This embodiment concretizes the operation of "in the computational graph, identifying the corresponding computational subgraph to be optimized according to the optimization paradigm" in the above preferred embodiment as follows: "in the computational graph, identifying an embedded vector conversion operator, and determining the embedded vector conversion operator as a computational subgraph to be optimized according to the distributed marking of the input tensor of the embedded vector conversion operator as a vector segmentation mode".

[0105] Accordingly, reference Figure 3 , the method may specifically include:

[0106] S301. Obtain a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy.

[0107] S302: In the computation graph, when an embedded vector conversion operator is identified and a vector segmentation mode is determined according to the distributed labeling of the input tensor of the embedded vector conversion operator, the embedded vector conversion operator is determined to be a computation subgraph to be optimized.

[0108] In the embedding vector layer vocabulary segmentation scenario, the input tensor is a vocabulary, and the dimension of the vocabulary can be the number of words × the number of vectors corresponding to each word. Correspondingly, the vector segmentation mode is column segmentation. Correspondingly, when the dimension of the vocabulary can be the number of vectors corresponding to each word × the number of words, the vector segmentation mode is row segmentation.

[0109] Determine the location of all embedding vector operation operators and check the segmentation method of their input parameters. Generally speaking, the dimension of the vocabulary can be the number of words × the number of vectors corresponding to each word. Therefore, the column-segmented (vector-segmented) vocabulary can be optimized and its segmentation mode can be optimized to the row-segmented (word-segmented) mode.

[0110] When the embedding vector conversion operator is identified and the corresponding tensor segmentation method is the vector segmentation mode, the operator can be determined as the computational subgraph to be optimized. In a single-card network, vector segmentation maintains the consistency of the word embedding model, so that the vector representation of each word remains unchanged before and after segmentation, which helps the model understand and process vocabulary. In a multi-card network, since multiple computing nodes are involved, the use of word segmentation allows each node to be responsible for only part of the vocabulary, which can reduce the memory pressure of a single node and can process data in parallel. In addition, word segmentation allows each node to process its own data shards independently, which can reduce the need for data transmission between nodes. Each node only needs to exchange prediction results or model updates with other nodes, rather than the entire vocabulary.

[0111] S303: Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0112] S304: Adjust the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator.

[0113] The technical solution of the disclosed embodiment can realize distributed labeling based on single-card networking through an optimization method for distributed labeling of computational graphs, automatically apply vocabulary segmentation strategies to multi-card networking, and optimize corresponding distributed labeling, thereby ensuring the ease of use of automatic parallelism in the vocabulary segmentation scenario at the embedded vector layer, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0114] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0115] Inserting a new embedding vector conversion operator in a word segmentation mode after the original embedding vector conversion operator in a vector segmentation mode, and redirecting the input of the original embedding vector conversion operator to the input of the new embedding vector conversion operator;

[0116] Inserting a communication protocol operator after the new embedding vector conversion operator, and linking the output of the new embedding vector conversion operator to the input of the communication protocol operator;

[0117] Linking the operator input in the computation graph that receives the output of the original embedding vector conversion operator to the output of the communication protocol operator;

[0118] Remove the original embedding vector conversion operator.

[0119] The original embedding vector conversion operator is replaced with a new embedding vector conversion operator (c_embedding) in word segmentation mode and aggregated with the communication protocol operator. The input of the original embedding vector conversion operator is redirected to the input of the new embedding vector conversion operator to receive the input tensor and convert it into a row segmentation form. The output of the new embedding vector conversion operator is linked to the input of the communication protocol operator. The operator input that receives the output of the original embedding vector conversion operator is changed to receive the output of the communication protocol operator. The output results of each new embedding vector conversion operator are collected through the communication protocol operator and output to the outside, realizing the mode in which the original embedding vector conversion operator in vector segmentation mode is replaced with the new embedding vector conversion operator in word segmentation mode and the communication protocol operator. The vector segmentation of single-card networking is automatically replaced with the word segmentation strategy of multi-card networking, ensuring the efficiency and performance of automatic parallel distributed training.

[0120] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator includes:

[0121] According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the vector segmentation mark in the calculation graph as the original tensor distributed mark is adjusted to a word segmentation mark, and the output tensor mark of the new embedding vector conversion operator is adjusted to a partial segmentation mark;

[0122] According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

[0123] According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the original tensor distributed mark is adjusted to the word segmentation mark (row segmentation), the output tensor mark is adjusted to the partial segmentation mark, and the segmentation mark of the output tensor of the communication protocol operator is set to the copy state, that is, each computing node will store a complete copy of the tensor output, and the final output tensor result is obtained by simply superimposing the output tensors of each partial segmentation method, ensuring the automatic parallel generation of the embedding vector results in the form of word segmentation, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0124] Figure 4 It is a schematic diagram of another optimization method for distributed marking of a computational graph provided according to an embodiment of the present disclosure. This embodiment concretizes the operation of "in the computational graph, identifying the corresponding computational subgraph to be optimized according to the optimization paradigm" in the above-mentioned other preferred embodiment as follows: "in the computational graph, identifying a computational subgraph including a communication acquisition operator and a classification cross entropy operator, and determining that the column splitting mode is determined according to the distributed marking of the input tensor of the communication acquisition operator, then determining that the computational subgraph including the communication acquisition operator and the classification cross entropy operator is the computational subgraph to be optimized".

[0125] Accordingly, reference Figure 4 , the method may specifically include:

[0126] S401. Obtain a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy.

[0127] S402. In the computation graph, a computation subgraph including a communication acquisition operator and a classification cross entropy operator is identified, and when the column splitting mode is determined according to the distributed marking of the input tensor of the communication acquisition operator, the computation subgraph including the communication acquisition operator and the classification cross entropy operator is determined as the computation subgraph to be optimized.

[0128] The communication collection operator (all_gather) can be specifically understood as: communicating with other nodes and collecting the calculation results of other nodes. The classification cross entropy operator can be specifically understood as: the operation of calculating the classification probability and cross entropy of the data.

[0129] In a single-card network, since data is usually processed in batches and each batch contains multiple samples, column splitting (splitting along the category dimension) can parallelly calculate the classification probability and cross entropy loss of each category, thereby improving the processing efficiency of a single card. In a multi-card network, row splitting (splitting along the sample dimension) means that each computing node processes a portion of the samples in the entire batch, and can independently calculate the loss of the samples it is responsible for, thereby reducing the data processing dimension, aggregating gradients through back propagation, reducing inter-node communication, and improving the efficiency of distributed training. Therefore, in the classification cross entropy optimization scenario, the column splitting method can be optimized, that is, the computational subgraph that includes the communication acquisition operator and the classification cross entropy operator, and the input tensor of the communication acquisition operator is distributed and marked as a column splitting mode, is selected as the computational subgraph to be optimized.

[0130] S403: Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0131] S404: Adjust the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator.

[0132] The technical solution of the disclosed embodiment automatically optimizes the classification cross entropy calculation strategy in the form of column segmentation in a single-card network to the classification cross entropy calculation strategy in the form of row segmentation in a multi-card network, thereby reducing the data processing dimension, reducing the communication between nodes, ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0133] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0134] Inserting a new parallel categorical cross entropy operator in a row splitting mode after the original categorical cross entropy operator in a column splitting mode, and redirecting the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator;

[0135] Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator;

[0136] Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator;

[0137] The communication collection operator and the original classification cross entropy operator are removed.

[0138] The original classification cross entropy operator is a classification cross entropy calculation operation in the form of column splitting suitable for single-card networking. The communication collection operator is an operation for collecting the output results of the original classification cross entropy operator suitable for single-card networking. In multi-card networking, it needs to be replaced with a new parallel classification cross entropy operator in the row splitting mode and the corresponding communication protocol operator.

[0139] Insert a new parallel classification cross entropy operator and a communication protocol operator in the row split mode, redirect the input of the original classification cross entropy operator to the input of the new parallel classification cross entropy operator to receive data and calculate the classification probability and cross entropy, link the output of the new parallel classification cross entropy operator to the input of the communication protocol operator, link the operator input that receives the output of the original classification cross entropy operator to the output of the communication protocol operator, and realize the transmission of the calculation result from the new parallel classification cross entropy operator in the row split mode via the communication protocol operator, remove the communication acquisition operator and the original classification cross entropy operator, and realize the mode of replacing the original classification cross entropy operator and communication acquisition operator in the column split mode with the new parallel classification cross entropy operator and communication protocol operator in the row split mode, automatically replace the column splitting of single-card networking with the row splitting strategy of multi-card networking, and ensure the efficiency and performance of automatic parallel distributed training.

[0140] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator includes:

[0141] According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark in the computation graph, which is the original tensor distributed mark, is adjusted to a row split mark;

[0142] According to the operation logic of the communication protocol operator, the output tensor of the communication protocol operator is added with a replication state.

[0143] According to the operating logic of the new parallel classification cross entropy operator in the split mode, the original tensor distributed mark is adjusted to the row split mark, and the split mark of the output tensor of the communication protocol operator is set to the copy state, that is, each computing node will store a complete copy of the tensor output. By storing each output tensor, the output tensor results of each time are obtained, ensuring the automatic parallel generation of the classification cross entropy calculation results in the form of row splits, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0144] In deep learning, the categorical cross entropy operator (softmax_with_cross_entropy) is a commonly used loss function. Its calculation includes two parts: softmax and cross entropy. In the scenario of model parallelism, it is usually necessary to have a complete model output on each computing node for loss calculation. This is usually achieved through an all-gather operation, but this will bring a lot of communication overhead. In order to optimize this process, the parallel categorical cross entropy operator (c_softmax_with_cross_entropy) optimization reduces the amount of communication in the following ways: 1. Local calculation: perform local softmax and cross entropy calculations on each computing node first. 2. Reduce communication: Convert all-gather communication to all-reduce communication. Specifically, first calculate the partial sum on each computing node, and then aggregate these results through an all-reduce operation. In this way, the communication volume is reduced from b*s*v (batch size * sequence length * vocabulary size) to b*s (batch size * sequence length).

[0145] For ease of understanding, the specific application scenarios applicable to each embodiment of the present disclosure are now described. In the context of the continuous increase in the model, it is necessary to give full play to the computing power resources of the hardware to achieve efficient distributed training, which usually requires the application of various distributed optimization strategies. In automatic parallel mode, most optimization strategies can be automatically applied by identifying the tensor distributed tags marked by the user in the single-card networking. However, there are still some optimization strategies based on multi-card networking, which contain optimization operators unique to multi-card networking and are associated with multi-card communication operations, which makes them impossible to use in single-card networking; at the same time, these strategies cannot be described by distributed tags, that is, users cannot describe these optimization strategies by distributed tags in single-card networking, which increases the difficulty and cost of using optimization strategies, reduces the adaptability and flexibility of automatic parallelism, and reduces the efficiency and performance of distributed training. To solve the above problems, the embodiment of the present disclosure proposes an automatic parallel distributed tag optimization method.

[0146] Distributed tagging is to insert distributed tag information into the model's dynamic graph single card network. In automatic parallel mode, users only need to perform distributed tagging on some tensors in the model, and automatic parallelization will automatically expand these distributed tag information to the entire computational graph through segmentation inference rules.

[0147] Distributed tag optimization is a technology that automatically applies different optimization strategies under distributed conditions by replacing and splitting the computational subgraphs of the existing distributed marked single-card networking computational graphs. Specifically, distributed tag optimization can apply various optimization strategies based on distributed networking, especially multi-card networking. Distributed tag optimization will perform subgraph matching on the marked computational graphs to identify the computational subgraphs that need to be optimized; then, based on the optimization strategy that needs to be completed, the computational subgraphs will be replaced, operators will be added or subtracted, and other operations will be performed; finally, the entire computational graph will be split and derived again. In this way, the optimization strategy based on distributed networking can be effectively implemented on single-card networking.

[0148] Figure 5 This is a schematic diagram of automatic parallel distributed marking optimization applicable to the embodiment of the present disclosure, refer to Figure 5 ,The main process of distributed marking optimization includes subgraph matching, operator replacement, and re-segmentation derivation.

[0149] 1. Get the model calculation graph

[0150] In automatic parallelization, the forward network with labeled information input by the user is first used to generate forward, reverse and optimizer calculation graphs with distributed information. This calculation graph is the basis for subsequent optimization steps. Figure 5 As shown in (1), after the user marks the split state of a two-dimensional tensor Y (such as replicate and shard), the framework will automatically generate the split state of the tensor R according to the specific split inference rules.

[0151] 2. Subgraph Matching

[0152] Analyze and identify the features of the entire computational graph, and use the existing paradigm to match the subgraph that needs to be optimized. Figure 5 As shown in (2), by performing subgraph matching on the entire computational graph, the operators OP1 and OP2 in the box and the corresponding input tensors X and Y conform to the paradigm. There may be multiple subgraphs with suitable matches in a computational graph, and all of these subgraphs need to be replaced and optimized.

[0153] 3. Sub-image replacement

[0154] Based on the matched subgraph, replace the specific operators in it to make it conform to the state in the optimization strategy. Figure 5 As shown in (3), for all matched subgraphs, the operators therein are replaced according to the logic of the optimization strategy, and the operator OP1 in the figure is replaced with the new operator OP1. For more complex situations, the operator can be deleted or inserted multiple times until the subgraph meets the optimization strategy.

[0155] 4. Modify the segmentation status

[0156] After replacing a subgraph, the affected tensors need to be re-sliced ​​to ensure the correctness of the computational graph. Figure 5 As shown in (4), after modifying the operator in the subgraph, the tensor segmentation state in the subgraph also needs to be modified. In the figure, the segmentation state of the tensor Y is modified to meet the input requirements of the new operator OP1.

[0157] 5. Re-segmentation derivation

[0158] Based on the replaced subgraph, the entire computation graph is re-split and deduced, including forward and backward deduction, until the state of a certain operator is consistent with the original state. This ensures that the state of the entire computation graph reaches the optimization goal, and all necessary communication operations and data distribution are correctly configured. Figure 5 As shown in (5), after the entire subgraph is replaced, the segmentation state of the entire computational graph needs to be re-derived. In the figure, the segmentation state of the output tensor R of the operator OP2 is modified by re-segmentation derivation.

[0159] After completing the above distributed marking optimization, the execution module will start distributed training of the model according to the calculation graph after distributed marking optimization.

[0160] Figure 6 This is a schematic diagram of distributed marking optimization implemented by a setting tool software applicable to the embodiment of the present disclosure, refer to Figure 6 , distributed marking optimization can be achieved by setting tool software (Pass). For example, a technology used for compiler design and optimization can be specifically understood as a stage in the compilation process, in which the compiler performs specific analysis or transformation on the program code. In the deep learning framework, it refers to the gradual optimization of the computational graph, which is responsible for analyzing, modifying or replacing part of the computational graph to achieve specific optimization goals. Reference Figure 6 , in the setting tool software, it is necessary to implement the subgraph matching paradigm of the specific optimization strategy. After matching the subgraph, it is necessary to complete the replacement of the operator and the modification of the tensor segmentation state in the setting tool software. At the end of the execution of the setting tool software, it is necessary to call the segmentation deduction rule to re-segment and deduce the entire input calculation subgraph to ensure the correctness of the segmentation state of the entire calculation graph. It is worth noting that the setting tool software needs to replace and modify before loading the parameters to avoid loading the parameter tensor with the old segmentation state first, and then updating it to the new segmentation state through communication operations.

[0161] In a preferred disclosed embodiment, the automatic parallel distributed labeling optimization method proposed in the disclosed embodiment is suitable for the application scenario of parallel column segmentation embedding vector optimization.

[0162] In large-scale distributed training, the vocabulary of the embedding layer may be very large. For example, if there are 1,000 words, each represented by a 300-dimensional vector, the vocabulary is a 1,000*300 matrix. The actual vocabulary dimension is larger. If all embedding calculations are completed on a single card, it will affect both video memory and computing efficiency. By splitting the vocabulary and computing in parallel on multiple GPUs, computing efficiency can be improved. The parallel c_embedding (column splitting) optimization reduces video memory and computing overhead by performing row splitting (word splitting) and all_reduce (communication protocol reduction operator) operations on the vocabulary.

[0163] Figure 7 is a schematic diagram of a parallel column split embedding vector optimization applicable to the embodiment of the present disclosure, refer to Figure 7 , builtin.parameter refers to built-in parameters, that is, parameters in the model, such as weights or biases, which are learned during the training process. data refers to the data input to the model. embedding refers to the conversion of input data (usually word indexes) into vector representations in a high-dimensional space. transpose refers to the transposition operation, that is, exchanging the dimensions of the tensor, such as swapping the rows and columns of a matrix. all_gather refers to the communication collection operator, which is used to collect data from all nodes to each node in a distributed system. full_int_array refers to a full integer array, that is, an array filled with integers, which is used to restore the dimensions of the original data after the transposition operation. multiply refers to the multiplication operation, which is used to perform element-level multiplication. T[x,x] refers to the split state of the tensor Tensor in different dimensions. r refers to replicate(), which is a full copy in this dimension. s(x) refers to shard(x), which is split in the x dimension of the machine in this dimension. p(x) refers to partial(x), which is only partial data in the x dimension of the tensor.

[0164] refer to Figure 7 , the optimization method may specifically include:

[0165] 1. Get the model calculation graph

[0166] Get the computational graph with complete state after segmentation and derivation.

[0167] 2. Subgraph Matching

[0168] According to the logic of the optimization strategy, the entire computational graph is matched with subgraphs to identify whether the model is a model parallel structure, obtain parallel dimension information, determine the location of all embedding operations, and check the segmentation method of its input parameters. Only the column-segmented vocabulary is optimized.

[0169] 3. Sub-image replacement

[0170] The core is to replace the embedding operation with c_embedding and all_reduce operations.

[0171] Insert the c_embedding operator after the embedding operator to redirect the original embedding input to c_embedding. Continue to insert the all_reduce operator after the c_embedding operator to link the output of c_embedding to the input of all_reduce. Update all operations in the computation graph that receive the output of embedding to receive the output of all_reduce. Remove all embedding operations in the original computation graph.

[0172] 4. Modify the segmentation status

[0173] Although embedding and c_embedding use the same vocabulary weight, their sharding states are different. Change the vocabulary weight sharding state of c_embedding to row sharding (shard), and change the output from column sharding to partial sharding (partial). Mark the output of all_reduce as replicated (replicate).

[0174] 5. Re-segmentation derivation

[0175] For operators outside the subgraph, since the input and output segmentation states of the input subgraph are modified, they also need to be modified synchronously. According to the topological relationship of the computational graph, traverse the computational graph forward or backward from the subgraph, and call the segmentation derivation rule of each operator.

[0176] Calculate the segmentation state of its input or output, and determine whether the segmentation state is equal to the current state. If not, replace the current segmentation state with the newly derived result.

[0177] In another preferred disclosed embodiment, the automatic parallel distributed marking optimization method proposed in the disclosed embodiment is suitable for the application scenario of parallel column segmentation probability distribution cross entropy optimization.

[0178] In deep learning, softmax_with_cross_entropy is a commonly used loss function. Its calculation includes two parts: softmax and cross entropy. In the scenario of model parallelism, it is usually necessary to have the complete model output on each GPU for loss calculation. This is usually achieved through the all-gather (communication collection operator) operation, but this will bring a lot of communication overhead. To optimize this process, the parallel c_softmax_with_cross_entropy optimization reduces the amount of communication by:

[0179] 1. Local Computation

[0180] First, local softmax and cross entropy calculations are performed on each GPU.

[0181] 2. Reduce communication

[0182] Convert all-gather communication to all-reduce communication. Specifically, first calculate the partial sum on each GPU, and then aggregate these results through the all-reduce operation. In this way, the communication volume is reduced from b×s×v (batch size×sequence length×vocabulary size) to b×s (batch size×sequence length).

[0183] Figure 8 is a schematic diagram of a parallel column segmentation probability distribution cross entropy optimization applicable to the embodiment of the present disclosure, refer to Figure 8 , matmul refers to matrix multiplication, which is used to calculate the product of two matrices. full refers to the filling operation, which is used to create a tensor filled with specific values. c_softmax_with_cross_entropy: The optimized function of softmax with crossentropy in row split form. subtract refers to subtraction used to calculate the difference between two tensors. cross_entropy_with_softmax refers to the cross entropy and softmax function calculation operation, which is used for loss calculation of multi-classification problems. First, the softmax function is applied to convert the output into a probability distribution, and then the cross entropy loss between these probabilities and the true label is calculated. T[x,x] refers to the split state of the tensor Tensor in different dimensions. r refers to replicate(), which is a full copy in this dimension. s(x) refers to shard(x) which is split in the x dimension of the machine in this dimension. p(x) refers to partial(x) which is only part of the data in the x dimension of the tensor.

[0184] refer to Figure 8 , the optimization method can be specifically:

[0185] 1. Get the model calculation graph

[0186] Get the computational graph with complete state after segmentation and derivation.

[0187] 2. Subgraph Matching

[0188] Traverse all operations in the computation graph.

[0189] Identify all-gather and cross_entropy_with_softmax operator logic segments.

[0190] Check how the input tensor is sliced ​​and make sure it is sliced ​​in the column direction.

[0191] 3. Sub-image replacement

[0192] Replace the all-gather and cross_entropy_with_softmax operations with c_softmax_with_cross_entropy and all_reduce operations.

[0193] Insert the c_softmax_with_cross_entropy operator after cross_entropy_with_softmax and redirect the input of the cross_entropy_with_softmax operator in the computation graph to the input of c_softmax_with_cross_entropy.

[0194] Go ahead and insert the all_reduce operation and link the output of c_softmax_with_cross_entropy to the input of all_reduce.

[0195] Update all operations in the computation graph that receive the output of cross_entropy_with_softmax to receive the output of all_reduce.

[0196] Delete the all-gather+cross_entropy_with_softmax operator logic segment in the original calculation graph.

[0197] 4. Modify the segmentation status

[0198] Update the splitting status of c_softmax_with_cross_entropy and all_reduce.

[0199] 5. Re-segmentation derivation

[0200] According to the new segmentation status, traverse the computation graph to update the segmentation status of other related operations to ensure the consistency of the computation graph.

[0201] The automatic parallel distributed marking optimization method proposed in the embodiment of the present disclosure directly optimizes and modifies the calculation graph based on the existing distributed marking of single-card networking, without the need for users to manually modify the model and network segmentation status, or use distributed networking, thereby lowering the threshold for users to apply various optimization strategies when using automatic parallelization to complete distributed training, and improving the ease of use and efficiency of using optimization strategies under automatic parallelization.

[0202] As an implementation of the above-mentioned optimization methods for distributed marking of each computational graph, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned optimization methods for distributed marking of each computational graph.

[0203] Fig. 9 is a structural diagram of an optimization device for distributed marking of a computational graph provided according to an embodiment of the present disclosure. Fig. 9 The device comprises: a graph acquisition module 901, a graph identification module 902 to be optimized, a graph adjustment module 903 and a mark adjustment module 904, wherein:

[0204] A graph acquisition module 901 is used to acquire a computation graph of a distributed operation model and an optimization paradigm of a target optimization strategy;

[0205] A graph identification module 902 to be optimized is used to identify the corresponding computation subgraph to be optimized in the computation graph according to the optimization paradigm;

[0206] A graph adjustment module 903, used to adjust the computational subgraph to be optimized according to the optimization paradigm;

[0207] A tag adjustment module 904, used to adjust the tensor distribution tags in the calculation graph according to the original tensor distribution tags in the calculation graph and the operation logic of the operator;

[0208] Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

[0209] The technical solution of the disclosed embodiment obtains the calculation graph of the distributed operation model and the target calculation subgraph of the target optimization strategy; in the calculation graph, identifies the calculation subgraph to be optimized corresponding to the target calculation subgraph; uses the target calculation subgraph to adjust the calculation subgraph to be optimized; and adjusts the tensor distributed markings in the calculation graph according to the original tensor distributed markings in the calculation graph and the operation logic of the target calculation subgraph. The target calculation subgraph is used to automatically identify and convert into calculation subgraphs of multiple optimization strategies that integrate distributed networking, ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0210] On the basis of the above embodiments, the graph acquisition module 901 is specifically used for:

[0211] Obtain a computational graph of the model and some tensor distributed markers manually added in the computational graph;

[0212] According to the partial tensor distributed labels, automatic parallel splitting calculation is adopted to derive the remaining tensor distributed labels in the calculation graph and add them to the calculation graph.

[0213] On the basis of the above embodiments, the calculation graph before automatic parallel calculation is a model calculation graph supported by a single-card network, and the calculation graph after automatic parallel calculation is a model calculation graph supported by a multi-card network.

[0214] On the basis of the above embodiments, the target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networking.

[0215] On the basis of the above embodiments, the to-be-optimized graph identification module 902 is specifically used for:

[0216] The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

[0217] In an optional embodiment, based on the above embodiments, the to-be-optimized graph identification module 902 may include: a candidate graph matching unit and a to-be-optimized graph determination unit, wherein:

[0218] A candidate graph matching unit, configured to adopt the operator conditions in the optimization conditions to perform traversal matching in the computation graph to determine a matching candidate computation subgraph;

[0219] The unit for determining the graph to be optimized is used to determine the candidate computational subgraph as the computational subgraph to be optimized if it is identified that the segmentation method meets the segmentation requirements in the optimization condition based on the tensor distributed label of the candidate computational subgraph.

[0220] On the basis of the above embodiments, the graph adjustment module 903 is specifically used for:

[0221] According to the optimization operation in the optimization paradigm, inserting and deleting operations of operators are performed in the computation subgraph to be optimized to adjust the computation subgraph to be optimized.

[0222] On the basis of the above embodiments, the mark adjustment module 904 is specifically used for:

[0223] Starting from the adjusted operator in the computational graph, forward traversal and / or backward traversal is performed, and according to the original tensor distributed markings of the computational graph and the operation logic of the operator, the tensor distributed markings in the computational graph are adjusted until the tensor distributed markings determined according to the operation logic are consistent with the original tensor distributed markings.

[0224] In an optional embodiment, based on the above embodiments, the target optimization strategy is an embedding vector layer vocabulary segmentation strategy, and the computational subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator and a communication protocol operator of the word segmentation pattern.

[0225] Further, based on the above embodiments, the to-be-optimized graph identification module 902 may also be specifically used for:

[0226] In the computational graph, when an embedded vector conversion operator is identified and a vector segmentation mode is determined according to the distributed labeling of the input tensor of the embedded vector conversion operator, the embedded vector conversion operator is determined to be a computational subgraph to be optimized.

[0227] Further, based on the above embodiments, the graph adjustment module 903 may be specifically used for:

[0228] Inserting a new embedding vector conversion operator in a word segmentation mode after the original embedding vector conversion operator in a vector segmentation mode, and redirecting the input of the original embedding vector conversion operator to the input of the new embedding vector conversion operator;

[0229] Inserting a communication protocol operator after the new embedding vector conversion operator, and linking the output of the new embedding vector conversion operator to the input of the communication protocol operator;

[0230] Linking the operator input in the computation graph that receives the output of the original embedding vector conversion operator to the output of the communication protocol operator;

[0231] Remove the original embedding vector conversion operator.

[0232] Further, based on the above embodiments, the mark adjustment module 904 may be specifically used for:

[0233] According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the vector segmentation mark in the calculation graph as the original tensor distributed mark is adjusted to a word segmentation mark, and the output tensor mark of the new embedding vector conversion operator is adjusted to a partial segmentation mark;

[0234] According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

[0235] In an optional embodiment, based on the above embodiments, the target optimization strategy is a parallel classification probability and cross entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross entropy operator and a communication protocol operator.

[0236] Further, based on the above embodiments, the to-be-optimized graph identification module 902 may also be specifically used for:

[0237] In the computational graph, when a computational subgraph including a communication acquisition operator and a classification cross entropy operator is identified and a column splitting mode is determined according to the distributed marking of the input tensor of the communication acquisition operator, the computational subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be a computational subgraph to be optimized.

[0238] Further, based on the above embodiments, the graph adjustment module 903 may be specifically used for:

[0239] Inserting a new parallel categorical cross entropy operator in a row splitting mode after the original categorical cross entropy operator in a column splitting mode, and redirecting the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator;

[0240] Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator;

[0241] Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator;

[0242] The communication collection operator and the original classification cross entropy operator are removed.

[0243] Further, based on the above embodiments, the mark adjustment module 904 may be specifically used for:

[0244] According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark in the computation graph, which is the original tensor distributed mark, is adjusted to a row split mark;

[0245] According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

[0246] The above-mentioned product can execute the optimization method for distributed marking of computational graphs provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the optimization method for distributed marking of computational graphs.

[0247] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0248] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0249] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0250] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0251] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0252] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the optimization method for distributed labeling of the computing graph, that is:

[0253] Obtain the computational graph of the distributed operation model and the target computational subgraph of the target optimization strategy;

[0254] In the computation graph, identifying the computation subgraph to be optimized corresponding to the target computation subgraph;

[0255] Adopting the target computation subgraph to adjust the computation subgraph to be optimized;

[0256] According to the original tensor distribution mark in the calculation graph and the operation logic of the target calculation subgraph, the tensor distribution mark in the calculation graph is adjusted.

[0257] For example, in some embodiments, the optimization method for distributed marking of a computational graph may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the optimization method for distributed marking of the computational graph described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the optimization method for distributed marking of the computational graph by any other appropriate means (e.g., by means of firmware).

[0258] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0260] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0262] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0263] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0264] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0265] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0266] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0267] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for optimizing distributed labeling of a computational graph, comprising: Obtain the computational graph of the distributed running model and the optimization paradigm of the target optimization strategy; In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm; Adjust the computation subgraph to be optimized according to the optimization paradigm; Adjust the tensor distribution mark in the calculation graph according to the original tensor distribution mark in the calculation graph and the operation logic of the operator; Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

2. The method according to claim 1, wherein: Obtaining the computational graph of a distributed running model includes: Obtain a computational graph of the model and some tensor distributed markers pre-added in the computational graph; According to the partial tensor distributed labels, automatic parallel computing is used to derive the remaining tensor distributed labels in the computation graph and add them to the computation graph.

3. The method according to claim 2, wherein: The target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networking.

4. The method according to claim 1, wherein: In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

5. The method according to claim 4, wherein: The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine that the matching computation subgraphs to be optimized include: Using the operator conditions in the optimization conditions, traversing and matching in the computation graph to determine a matching candidate computation subgraph; If, according to the tensor distribution mark of the candidate computation subgraph, it is identified that the distribution mode meets the segmentation requirement in the optimization condition, the candidate computation subgraph is determined as the computation subgraph to be optimized.

6. The method according to claim 1, wherein: Adjusting the computation subgraph to be optimized according to the optimization paradigm includes: According to the optimization operation in the optimization paradigm, inserting and deleting operations of operators are performed in the computation subgraph to be optimized to adjust the computation subgraph to be optimized.

7. The method according to claim 1, wherein: According to the original tensor distribution mark in the calculation graph and the operation logic of the operator, adjusting the tensor distribution mark in the calculation graph includes: Starting from the adjusted operator in the computational graph, forward traversal and / or backward traversal is performed, and according to the original tensor distributed markings of the computational graph and the operation logic of the operator, the tensor distributed markings in the computational graph are adjusted until the tensor distributed markings determined according to the operation logic are consistent with the original tensor distributed markings.

8. The method according to any one of claims 1 to 7, wherein: The target optimization strategy is an embedding vector layer vocabulary segmentation strategy, and the computational subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator and a communication protocol operator of a word segmentation pattern.

9. The method according to claim 8, wherein: In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: In the computational graph, when an embedded vector conversion operator is identified and a vector segmentation mode is determined according to the distributed labeling of the input tensor of the embedded vector conversion operator, the embedded vector conversion operator is determined to be a computational subgraph to be optimized.

10. The method according to claim 9, wherein: Adjusting the computation subgraph to be optimized according to the optimization paradigm includes: Inserting a new embedding vector conversion operator in a word segmentation mode after the original embedding vector conversion operator in a vector segmentation mode, and redirecting the input of the original embedding vector conversion operator to the input of the new embedding vector conversion operator; Inserting a communication protocol operator after the new embedding vector conversion operator, and linking the output of the new embedding vector conversion operator to the input of the communication protocol operator; Linking the operator input in the computation graph that receives the output of the original embedding vector conversion operator to the output of the communication protocol operator; Remove the original embedding vector conversion operator.

11. The method according to claim 10, wherein: According to the original tensor distribution mark in the calculation graph and the operation logic of the operator, adjusting the tensor distribution mark in the calculation graph includes: According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the vector segmentation mark in the calculation graph as the original tensor distributed mark is adjusted to a word segmentation mark, and the output tensor mark of the new embedding vector conversion operator is adjusted to a partial segmentation mark; According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

12. The method according to any one of claims 1 to 7, wherein: The target optimization strategy is a parallel classification probability and cross entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross entropy operator and a communication protocol operator.

13. The method according to claim 12, wherein: In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: In the computational graph, when a computational subgraph including a communication acquisition operator and a classification cross entropy operator is identified and a column splitting mode is determined according to the distributed marking of the input tensor of the communication acquisition operator, the computational subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be a computational subgraph to be optimized.

14. The method according to claim 13, wherein: Adjusting the computation subgraph to be optimized according to the optimization paradigm includes: Inserting a new parallel categorical cross entropy operator in a row splitting mode after the original categorical cross entropy operator in a column splitting mode, and redirecting the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator; Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator; Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator; The communication collection operator and the original classification cross entropy operator are removed.

15. The method according to claim 14, wherein: According to the original tensor distribution mark in the calculation graph and the operation logic of the operator, adjusting the tensor distribution mark in the calculation graph includes: According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark in the computation graph, which is the original tensor distributed mark, is adjusted to a row split mark; According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

16. An optimization device for distributed labeling of a computational graph, comprising: The graph acquisition module is used to obtain the computational graph of the distributed operation model and the optimization paradigm of the target optimization strategy; A graph identification module to be optimized, used to identify the corresponding computation subgraph to be optimized in the computation graph according to the optimization; A graph adjustment module, used for adjusting the computational subgraph to be optimized according to the optimization paradigm; A tag adjustment module, used to adjust the tensor distributed tags in the calculation graph according to the original tensor distributed tags in the calculation graph and the operation logic of the operator; Among them, the calculation graph before adjustment is a model calculation graph supported by a single-card network, and the calculation graph after adjustment is a model calculation graph supported by a multi-card network.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

19. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images