An optimization method, device and equipment for distributed labeling of a computation graph, and a medium

Abstract

The present disclosure provides an optimization method, device and medium for distributed labeling of a computation graph, relating to the technical field of computers, and in particular to artificial intelligence, deep learning and distributed training technology. The specific implementation scheme is as follows: obtaining a computation graph of a distributed running model and an optimization paradigm of a target optimization strategy; identifying a corresponding to-be-optimized computation subgraph in the computation graph according to the optimization paradigm; adjusting the to-be-optimized computation subgraph according to the optimization paradigm; and adjusting tensor distributed labeling in the computation graph according to original tensor distributed labeling in the computation graph and operator running logic. The optimization paradigm is used to automatically identify and convert the computation subgraph of the multiple optimization strategies integrated into the distributed networking, ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to artificial intelligence, deep learning, and distributed training technology. Background Art

[0002] As deep neural networks continue to deepen, the number of model parameters continues to expand, showing an exponential growth trend, from millions a few years ago to hundreds of billions today. Consequently, large-scale model operations now mostly rely on distributed systems. Distributed model execution is a technology that breaks down computing tasks and distributes them across multiple computing nodes for parallel execution.

[0003] To run a model in a distributed system consisting of multiple compute nodes, it is necessary to construct a computational graph for the model. This graph includes multiple operators that conform to the model's execution logic. The input and output tensors of these operators must then be marked as distributed within the computational graph to ensure that each operator can run in parallel. Distributed model execution can be achieved using either a single-GPU or multi-GPU network. A single GPU refers to an independent graphics card (GPU). Each physical host can run one or more GPUs, and each GPU is also referred to as a compute node. Multi-GPU networking requires data communication between the GPUs.

[0004] For distributed model execution using a single-card network, distributed tagging of the computation graph is relatively simple and can be added manually. However, for distributed execution using multiple-card networks, distributed tagging is more complex. Some optimization operators cannot even be described using distributed tagging. Therefore, manually adding distributed tags requires a high technical threshold and is labor-intensive. Summary of the Invention

[0005] The present disclosure provides a method, apparatus, device, medium, and program product for generating distributed tags for computation graphs, so as to optimize the manner of adding distributed tags to computation graphs for distributed model operation.

[0006] According to one aspect of the present disclosure, a method for optimizing distributed labeling of computational graphs is provided, comprising:

[0007] Obtain the computational graph of the distributed running model and the optimization paradigm of the target optimization strategy;

[0008] In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm;

[0009] Adjust the computation subgraph to be optimized according to the optimization paradigm;

[0010] Adjusting the tensor distribution marks in the computation graph according to the original tensor distribution marks in the computation graph and the operation logic of the operator;

[0011] Among them, the calculation graph before adjustment is the model calculation graph supported by single-card networking, and the calculation graph after adjustment is the model calculation graph supported by multi-card networking.

[0012] According to another aspect of the present disclosure, there is also provided an optimization device for distributed labeling of a computational graph, comprising:

[0013] The graph acquisition module is used to obtain the computational graph of the distributed operation model and the optimization paradigm of the target optimization strategy;

[0014] A graph identification module to be optimized, configured to identify, in the computation graph, the corresponding computation subgraph to be optimized according to the optimization paradigm;

[0015] A graph adjustment module, configured to adjust the computational subgraph to be optimized according to the optimization paradigm;

[0016] A tag adjustment module, configured to adjust the tensor distribution tags in the computation graph according to the original tensor distribution tags in the computation graph and the operation logic of the operator;

[0017] Among them, the calculation graph before adjustment is the model calculation graph supported by single-card networking, and the calculation graph after adjustment is the model calculation graph supported by multi-card networking.

[0018] According to another aspect of the present disclosure, there is also provided an electronic device, including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the optimization method for distributed marking of computational graphs as described in any one of the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable a computer to execute the optimization method for distributed marking of computational graphs according to any one of the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the optimization method for distributed marking of computational graphs as described in any one of the embodiments of the present disclosure.

[0024] The technical solution of the embodiment of the present disclosure can automatically complete the calculation graph adjustment of the optimization strategy and the derivation change of distributed marking, reducing the difficulty of manual marking.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a schematic diagram of an optimization method for distributed marking of computation graphs provided according to an embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram of another optimization method for distributed marking of computational graphs provided according to an embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram of another method for optimizing distributed marking of computational graphs provided according to an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of another method for optimizing distributed marking of computational graphs provided according to an embodiment of the present disclosure;

[0031] Figure 5 This is a schematic diagram of automatic parallel distributed marking optimization applicable to the embodiments of the present disclosure;

[0032] Figure 6 This is a schematic diagram of distributed marking optimization implemented by a setting tool software applicable to the embodiments of the present disclosure;

[0033] Figure 7 Schematic diagram of a parallel column split embedding vector optimization applicable to the embodiments of the present disclosure;

[0034] Figure 8 Schematic diagram of a parallel column split probability distribution cross entropy optimization applicable to the embodiments of the present disclosure;

[0035] Figure 9 This is a structural diagram of an optimization device for distributed labeling of computational graphs provided according to an embodiment of the present disclosure;

[0036] Figure 10 It is a block diagram of an electronic device used to implement the optimization method of distributed labeling of computational graphs according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] First, the related technologies of the embodiments of the present disclosure are introduced:

[0039] A computation graph is a graph structure that describes the computational flow of a deep neural network. A computation subgraph is a portion of the full computation graph, encompassing part of the model's computational flow. A distributed model is one that splits model execution into multiple tasks and runs them in parallel using multiple compute nodes. Model execution can involve both model training and inference using a trained model. A computation graph for a distributed model can be understood as a graphical representation of the model's computational process in a distributed system, describing how data flows between different compute nodes and how the various computational tasks work together to complete the overall model training or inference process. Operators in the computation graph represent arbitrary tasks, such as computation or communication, and tensor input and output flows between operators.

[0040] Tensors are a generalization of vectors and matrices. Based on the processing stage, they can be divided into input tensors and output tensors. Input tensors can be inputs to be processed or data required for an operator, such as weight parameter matrices. The processing logic of an operator determines the relationship between input and output tensors.

[0041] The placements of a tensor are used to indicate the placement of the distributed tensor in the dimension of the corresponding compute node. There are three optional placements: Replicate, Shard, and Partial. These placements are described in detail below:

[0042] Replicate means that the tensor is kept in full state on all computing nodes.

[0043] Sharding (axis) refers to splitting a tensor along the axis dimension and placing it on different compute nodes. This typically includes 0-dimensional (column-wise) and 1-dimensional (row-wise) sharding.

[0044] Partial means that each computing node only has partial values ​​and needs to be restored to the full data through specified reduction operations.

[0045] Row splitting involves splitting a tensor along its row dimension. For example, a 1k×1k 2D tensor is split into four blocks, each containing 250 rows, but with 1k columns in each row. In a distributed environment, each block can be assigned to different compute nodes for processing.

[0046] Column sharding splits a tensor along its column dimension. For example, for a 1k×1k 2D tensor, the tensor is split into four columns, each containing 250 columns, but all columns have 1k rows. In distributed computing, each column block can be assigned to a different compute node.

[0047] Local slicing is the process of splitting a tensor into blocks of partial dimensions, with each block containing a portion of rows and columns. This is suitable for complex multidimensional data processing. For example, for a 1k×1k dimensional two-dimensional tensor, the tensor can be divided into 25 blocks of 200×200 dimensional data. Preferably, the tensor obtained by local slicing can still be 1k×1k dimensional, retaining only the data corresponding to the 200×200 dimension, and filling the remaining positions with zeros. This allows the final result to be obtained by superimposing the locally split tensors after distributed computation.

[0048] The distribution of tensors directly affects how distributed computing operates and needs to be determined based on the specific computational requirements of the operators. In a single-card network, distributed labels can be added manually when the model generates the computational graph. However, given the enormous workload of labeling every input and output tensor, labeling is typically performed only on key operators, such as large-scale matrix multiplications, convolution operations, or complex, time-consuming, and communication-intensive operations. For other operations, the system automatically derives and adjusts the distributed labels of tensors in the computational graph based on the added distributed labels and the computational graph's operational logic.

[0049] Automatic parallelization technology automatically determines the parallel processing method based on the operator logic in the computation graph. Specifically, it determines the tensor distribution tags for each operator. A small number of distribution tags can be manually added, and the framework automatically completes the remaining tensor distribution tags and inserts the necessary communication operations to achieve distributed training.

[0050] Figure 1 This is a schematic diagram of a method for optimizing distributed marking of computational graphs according to an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to situations where various optimization strategies based on distributed networking are automatically applied. This method can be executed by an optimization device for distributed marking of computational graphs, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 , the method specifically includes the following:

[0051] S101. Obtain a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy;

[0052] S102: In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm;

[0053] S103, adjusting the computation subgraph to be optimized according to the optimization paradigm;

[0054] S104: Adjust the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator;

[0055] Among them, the calculation graph before adjustment is the model calculation graph supported by single-card networking, and the calculation graph after adjustment is the model calculation graph supported by multi-card networking.

[0056] In the above solution, the target optimization strategy refers to a strategy that is not used in the original computation graph but is desired to be optimized. For example, when deploying a model corresponding to a single-card network computation graph to a distributed system with multiple cards, some parallel strategies that are only supported by multiple cards can be adjusted to the original computation graph as the target optimization strategy.

[0057] The optimization paradigm of the target optimization strategy can be specifically understood as: requirements and conditions corresponding to specific optimization strategies, mainly including optimization conditions and optimization operations. Operators that meet the optimization conditions in the calculation graph can be optimized, and the optimization operation is used to specify how to adjust the operators to be optimized to implement the target optimization strategy in the calculation graph. Among them, the target optimization strategy may be an operation strategy with the same operating purpose as the original parallel strategy, but adopts different operators, such as an optimization strategy adopted to reduce the amount of data communication between computing nodes, or to improve the efficiency of parallel processing. The embodiments of the present disclosure do not limit this. For the calculation graph supported by a single-card network, it is generally impossible to consider some more efficient parallel processing strategies that can be implemented on a multi-card network. These parallel processing strategies can be target optimization strategies.

[0058] The computational subgraph to be optimized can be understood as the set of operators in the computational graph that meet the optimization criteria in the optimization paradigm. Optimization criteria can include operator types and tensor attributes of the operator input tensors, such as distributed labels.

[0059] In different networking environments, such as single-SIM and multi-SIM, data reading and processing logic differs due to media and transmission latency. Each model runs independently based on its computational graph logic, involving multiple operator operations, such as convolution, embedding vector conversion, reduction operations, and communication operations, which together constitute the model's computational graph. Even if the basic model is the same, the computational graphs for single-SIM and multi-SIM networks will differ due to different networking methods. Therefore, to optimize performance, the computational subgraph in single-SIM networks needs to be adjusted according to specific optimization strategies to ensure that the computational graph adapts to different network environments and computing requirements. The pre-adjusted computational graph is a model computational graph supported by single-SIM networks. The hardware structure of single-SIM networks is simple, so users are relatively simple to label tensors in the computational graph, with a low technical threshold and minimal workload. Automatic parallel inference technology is used to automatically run operator tasks in parallel in the distributed system of multi-SIM networks. Furthermore, the technical solution of the disclosed embodiment can also adjust the model calculation graph and the tensor distribution labels therein based on the optimization strategy based on automatic parallelization. The adjusted calculation graph is a model calculation graph supported by multi-card networking. As a result, the operation mode of the adjusted model calculation graph can adopt the unique parallelization strategy supported by multi-card networking, making the operation mode more able to give full play to the hardware advantages of multi-card networking and effectively utilize the hardware device resources of multi-card networking.

[0060] Generally speaking, adjusting the computational subgraph to be optimized according to the optimization paradigm may specifically include: performing operator insertion and deletion operations in the computational subgraph to be optimized according to the optimization operations in the optimization paradigm to adjust the computational subgraph to be optimized. Optimization operations generally include operators that need to be inserted or deleted, as well as the order of insertion and deletion operations. For example: the computational subgraph to be optimized includes operator op1 (operator, operator 1), which is the object to be optimized. Operators that can be inserted include the new operator op1_new, so op1 can be replaced with op1_new. Among them, operators op and op1_new only represent computational operation links and do not specify or limit a specific operation. In order to meet the needs of multi-card networking, it may also be necessary to add communication operators, such as communication protocol operators (AllReduce), communication broadcast operators (Broadcast), or communication collection operators (AllGather), to achieve efficient data transmission and synchronization across devices; operator fusion can also be used to reduce the amount of data calculation or communication, reduce communication overhead, and improve computing efficiency.

[0061] After adjusting the computation graph, such changes require adaptive adjustments to the tensor distribution labels of other operators to ensure that the tensor distribution labels of the entire computation graph are correct. Therefore, automatic parallelization technology can be used to automatically complete the derivation and adjustment of tensor distribution labels, realizing the automatic application of various optimization strategies and the derivation of tensor distribution on a multi-card network.

[0062] The technical solution of the embodiment of the present disclosure uses the optimization paradigm to automatically identify and convert operators into multiple optimization strategies that are integrated into distributed networking, and can automatically derive distributed labels of tensors in parallel, ensuring the ease of use of automatic parallelization, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelization, and improving the efficiency and performance of distributed training.

[0063] In an optional implementation of this embodiment, the optimization method for distributed marking of the computation graph may also be configured with a simple switch mechanism.

[0064] The switch mechanism can be specifically understood as a control parameter, which is used to indicate whether to execute the optimization method of distributed marking of the computational graph, and to realize the use of distributed marking optimization based on the existing distributed marking through a simple switch, and to quickly apply those optimization strategies based on distributed networking. The switch mechanism can be implemented through options in the configuration file, environment variables, command line parameters, constants in the code, check boxes in the user interface, or settings in the database. Through these flexible implementation methods, the switch mechanism provides an intuitive control means for administrators. After turning on this function, the administrator can select the target optimization strategy, so that the solution of the embodiment of the present disclosure can be used to automatically complete the adjustment of the computational graph and the annotation of the distributed marking of the tensor based on the target optimization strategy, ensuring the ease of use of automatic parallelism, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0065] In a specific example, a part of the computational graph can be analyzed, modified or replaced by a setting tool software with automatic parallel deduction function to achieve a specific optimization goal. First, a specific optimization strategy subgraph matching pattern needs to be implemented in the setting tool software, and when the corresponding subgraph is identified, the corresponding operator is replaced and the distributed label of the corresponding tensor is adjusted. In the final stage of the setting tool software processing flow, it is also necessary to call the splitting and derivation rules to re-split and deduce the entire input computational subgraph to ensure that the distributed labeling state of the computational graph remains correct. Among them, the splitting and derivation rules refer to the derivation of the input and output tensor splitting states actually required based on the input tensor state and the operator. Automated processing not only improves efficiency but also reduces human errors.

[0066] Figure 2This is a schematic diagram of another method for optimizing distributed labeling of computational graphs provided in accordance with an embodiment of the present disclosure. This embodiment concretizes the operation of "obtaining a computational graph of a distributed operation model" in the above embodiment as follows: "obtaining the computational graph of the model and some pre-added tensor distributed labels in the computational graph; based on the partial tensor distributed labels, using automatic parallel computing to derive the remaining tensor distributed labels in the computational graph and add them to the computational graph."

[0067] Accordingly, reference Figure 2 , the method may specifically include:

[0068] S201: Obtain a computational graph of a model and some tensor distributed labels pre-added in the computational graph.

[0069] Pre-added distributed markings for some tensors can be manually added by the user. Specifically, in a distributed computing environment, due to the need to disperse model execution across multiple computing nodes, to optimize performance and resource utilization or to ensure that the parallel strategy meets the user's preset requirements, the distribution method (such as row splitting, column splitting, and local splitting) of tensors is explicitly specified at specific or key locations in the computation graph. This provides finer control than fully automatic splitting strategies. Typically, this distributed marking can be a small number of tensor splitting method markings for specific locations in the computation graph supported by a single-card network.

[0070] S202: Based on the partial tensor distributed labels, use automatic parallel computing to derive the remaining tensor distributed labels in the computation graph and add them to the computation graph.

[0071] After adding distributed labels to some tensors, the manually specified distributed labels are used as the starting point. According to the distributed labels of known tensors, the networking architecture of computing nodes, and the logical structure of the computational graph, automatic parallelization technology is applied to automatically deduce parallel strategies, thereby realizing the derivation and update of the distributed labels of the unlabeled tensors of the operator, so that the distributed labels of the remaining unlabeled tensors can be completed in the computational graph.

[0072] Generally speaking, since the functions and relationships of operators are well-defined, a small number of splitting markers can be used to infer the splitting methods of other tensors in the entire computation graph. If the splitting methods of some tensors cannot be automatically inferred, it means that the manually added splitting markers are insufficient and more splitting markers need to be added to guide the automatic derivation process.

[0073] S203, obtaining an optimization paradigm of the target optimization strategy;

[0074] S204: In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm.

[0075] S205: Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0076] S206. Adjust the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator.

[0077] In multi-card networks, manual addition of distributed tags is complex and labor-intensive due to the increased data and operations involved. The implementation of distributed operation strategies requires comprehensive consideration of the segmentation of individual operators, as well as the data flow and dependencies between operators. Furthermore, manually added tensor distributed tags in multi-card networks may conflict with other tags, particularly regarding operator distribution and data synchronization, because they fail to account for global optimization and resource allocation requirements.

[0078] Generally speaking, in automatic parallel mode, most optimization strategies can be automatically applied to multi-SIM networks by identifying distributed tags marked in single-SIM networks. However, some special optimization strategies based on multi-SIM networks include distributed-specific optimization operators and are associated with multi-SIM communication operations. Because these operators and communication operations are integrated, these strategies cannot be used in single-SIM networks. At the same time, because these optimization strategies use distributed-specific optimization operators, which, while semantically similar to ordinary operators, are implemented completely differently, users cannot describe these optimization strategies in single-SIM networks by adding distributed tags to them, converting the original ordinary operators into distributed-specific optimization operators. In other words, it is impossible to describe these optimization strategies in single-SIM networks by manually adding distributed tags.

[0079] Optionally, the calculation graph before automatic parallel calculation is a model calculation graph supported by a single-card network, and the calculation graph after automatic parallel calculation is a model calculation graph supported by a multi-card network. The disclosed embodiment can manually add distributed tags based on the distributed tags of the single-card network calculation graph, perform automatic parallel deduction to improve the tensor distributed tags of the entire calculation graph; then use the target optimization strategy to optimize the calculation graph, and update the tensor distributed tags so that the entire calculation graph is consistent with the application of the corresponding optimization strategy. Based on the existing distributed tagging and automatic parallel deduction technology, it is possible to quickly apply optimization strategies based on multi-card networks.

[0080] The technical solution of the disclosed embodiment pre-adds some tensor distributed tags for key operators or special operators in the calculation graph of a single-card network and derives the addition of the remaining tensor distributed tags, and then performs adjustments to the optimization strategy. This enables the use of automatic parallel derivation tools to handle the addition and optimization of distributed tags in a multi-card network. This optimization method can maximize the advantages of automatic parallelization, make better decisions based on the entire calculation graph and network topology, while maintaining human control over key parts, achieve simple and efficient distributed training, ensure the ease of use of automatic parallelization, reduce the difficulty and cost of using optimization strategies, enhance the adaptability and flexibility of automatic parallelization, and improve the efficiency and performance of distributed training.

[0081] In an optional implementation of this embodiment, the target optimization strategy corresponds to a distributed optimization strategy supported by multi-card networking.

[0082] Typically, as mentioned above, given the complex structure of multi-card networks, manually adding distributed tags is labor-intensive and prone to errors. The optimization method for distributed tagging in computational graphs proposed in this disclosure is specifically designed for multi-card networks. Accordingly, the target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networks, thereby reducing the difficulty and cost of using the multi-card network optimization strategy and improving the efficiency and performance of distributed training in multi-card networks. As a result, the original computational graph with manually added distributed tags can be free of the distributed optimization strategy for multi-card networks, thus reducing the complexity of manual addition.

[0083] The optimization paradigm may include the addition of communication operators. In the process of distributed marking based on the single-card networking calculation graph, applying distributed operation strategies, and optimizing the multi-card networking calculation graph, in order to maintain the consistency of distributed training results, model parameters need to be frequently synchronized between different nodes. This synchronization operation needs to be implemented through specific communication operators. Since data and model parameters do not need to be synchronized between multiple computing nodes in a single-card network, communication operators are usually not required. Therefore, the calculation subgraph corresponding to the multi-card network includes communication operators between each graphics card / computing node to ensure data synchronization and parameter consistency across computing nodes and to meet the computing and communication dependencies between each computing node.

[0084] Typically, in a multi-card networking environment, in addition to communication operators, the optimized operator graph can also include some unique operators, such as data sharding operators, task scheduling operators, memory management operators, and load balancing operators.

[0085] In an optional implementation of this embodiment, in the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm may include:

[0086] The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

[0087] By traversing each part of the computational graph based on optimization conditions, the matched data streams have the same composition and contain the computational subgraphs to be optimized of the corresponding operators. The automated traversal method ensures comprehensive identification of optimization opportunities, reduces the need for manual intervention, reduces the complexity and error rate of the optimization work, ensures the ease of use of automatic parallelization, reduces the difficulty and cost of using optimization strategies, and improves the efficiency and performance of distributed training.

[0088] In an optional implementation of this embodiment, the optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine that the matching computation subgraph to be optimized includes:

[0089] Using the operator conditions in the optimization conditions, traversing and matching in the computation graph to determine a matching candidate computation subgraph;

[0090] If, based on the tensor distribution mark of the candidate computation subgraph, it is identified that the distribution mode meets the segmentation requirement in the optimization condition, the candidate computation subgraph is determined as the computation subgraph to be optimized.

[0091] Specifically, a candidate computational subgraph can be understood as one that matches the data flow composition of each part of the computational graph through optimization conditions and contains the corresponding operators. After determining the candidate computational subgraph, it is also necessary to determine whether the tensor distribution in the candidate computational subgraph matches the segmentation requirements in the optimization conditions, such as row segmentation, column segmentation, or local segmentation. Only when these match is the candidate computational subgraph determined as the computational subgraph to be optimized. This ensures consistency and accuracy in distributed computing, improves the performance of distributed training, and avoids inconsistent segmentation and erroneous computation results caused by incorrect identification of the computational subgraph to be optimized.

[0092] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0093] According to the optimization operation in the optimization paradigm, inserting and deleting operators are performed in the computation subgraph to be optimized to adjust the computation subgraph to be optimized.

[0094] The optimization paradigm records the logical structure corresponding to the target optimization strategy, including the types and logical relationships of each operator. After identifying the computational subgraph to be optimized, it is necessary to insert or delete at least one operator in the computational subgraph according to the optimization operation in the optimization paradigm to implement the target optimization strategy into the computational graph.

[0095] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator includes:

[0096] Starting from the adjusted operator in the computation graph, forward traversal and / or backward traversal is performed, and the tensor distributed markings in the computation graph are adjusted according to the original tensor distributed markings of the computation graph and the operation logic of the operator until the tensor distributed markings determined according to the operation logic are consistent with the original tensor distributed markings.

[0097] After adjusting the computational subgraph to be optimized, starting from the adjusted operator in the graph, traverse forward from the operator's input flow and traverse backward from the operator's output flow to determine whether the tensor distributed labels of other operators in the forward and backward flows need to change. During the traversal process, the distributed labels of the corresponding tensors are determined and updated according to the operator's operating logic. Since the logical structure of the adjusted computational subgraph to be optimized remains unchanged except for the adjusted operator, when it is found that the original tensor distributed label of an operator in the computational graph is consistent with the tensor distributed label determined according to the operation logic, there is no need to continue traversal and update. This can effectively reduce unnecessary calculations, optimize the traversal process, enhance the adaptability and flexibility of automatic parallelization, and improve the efficiency and performance of distributed training.

[0098] In a preferred implementation of this embodiment, the target optimization strategy is an embedding vector (Embedding i ng) layer word list segmentation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator and a communication protocol operator of the word segmentation pattern.

[0099] Embedded vectors are a commonly used feature representation in models. Input data needs to be converted into embedded vector outputs, and this is typically accomplished using an embedding vector conversion operator. The embedding vector layer vocabulary is a vector vocabulary used to convert input data into embedded vectors. The vector vocabulary contains all words and their corresponding unique indexes. Each word is assigned an index, which is used to retrieve the corresponding word vector from the embedding matrix. Assume the vocabulary is N*M dimensional, with N rows representing the number of words and M columns representing the dimensions of each word's embedding vector. Of course, the word dimension and embedding vector dimension can also be swapped to M*N dimensions. The word segmentation mode also refers to row segmentation, which divides all words into multiple parts. The vector segmentation mode is generally column segmentation, which divides the entire embedding vector of each word into multiple parts. Communication protocol operators are operations used in distributed computing and parallel processing to exchange and aggregate data between multiple nodes.

[0100] The embedding vector layer vocabulary optimization is an important distributed computing scenario, which solves the problem of high dimensionality and inability to express the relationship between words by mapping the vocabulary to the vector in the high-dimensional space. The optimization method of the scenario can improve the adaptability of automatic parallel distributed training.

[0101] In another preferred embodiment of the present embodiment, the target optimization strategy is a parallel classification probability and cross-entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross-entropy operator and a communication reduction operator.

[0102] The cross-entropy loss is a standard choice for handling multi-classification problems, often combined with neural networks using a softmax activation function. The softmax function converts the raw values output by the neural network into a probability distribution (classification probability), and the cross-entropy loss measures the difference between this probability distribution and the true label distribution. Parallel classification probability and cross-entropy loss calculation refers to the above calculation process, and parallel means that the calculation of the classification probability and cross-entropy loss of each operator in the calculation graph is parallel calculation. The parallel classification cross-entropy operator can be specifically understood as: an operation that converts the calculation method of the classification probability and cross-entropy loss of the operator into other methods, corresponding to the parallel classification probability and cross-entropy loss calculation strategy. It is the key to automatically applying the parallel classification probability and cross-entropy loss calculation strategy in multi-card networking based on the existing distributed labeling of single-card networking.

[0103] In the distributed scenario, due to the need to process large-scale data sets and complex models, the optimization of cross-entropy loss calculation and parallel classification probability is an important distributed computing scenario. The optimization method of this scenario not only helps to improve the efficiency and accuracy of model training, but also effectively synchronizes and aggregates data between multiple nodes, enhances the adaptability of automatic parallel, and improves the efficiency and performance of distributed training.

[0104] Figure 3 is a schematic diagram of another optimization method for distributed labeling of a calculation graph according to an embodiment of the present disclosure. In the present embodiment, the operation of "identifying a corresponding to-be-optimized calculation subgraph in the calculation graph according to the optimization paradigm" in the above preferred embodiment is specifically implemented as: "in the calculation graph, an embedding vector conversion operator is identified, and when the distributed labeling of the input tensor of the embedding vector conversion operator is determined as a vector splitting mode, it is determined that the embedding vector conversion operator is a to-be-optimized calculation subgraph".

[0105] Correspondingly, referring to Figure 3 , the method can specifically include:

[0106] S301, obtaining a calculation graph of a distributed running model, and an optimization paradigm of a target optimization strategy.

[0107] S302: In the computation graph, when an embedded vector conversion operator is identified and a vector segmentation mode is determined based on the distributed labeling of the input tensor of the embedded vector conversion operator, the embedded vector conversion operator is determined to be a computation subgraph to be optimized.

[0108] In the embedding vector layer vocabulary segmentation scenario, the input tensor is a vocabulary, and the vocabulary dimension can be the number of words × the number of vectors corresponding to each word. Accordingly, the vector segmentation mode is column segmentation. Conversely, when the vocabulary dimension can be the number of vectors corresponding to each word × the number of words, the vector segmentation mode is row segmentation.

[0109] Determine the location of all embedding vector operators and check how their input parameters are segmented. Generally speaking, the dimension of the vocabulary can be the number of words × the number of vectors corresponding to each word. Therefore, the vocabulary with column segmentation (vector segmentation) can be optimized and its segmentation mode can be optimized to row segmentation (word segmentation) mode.

[0110] Only when the embedding vector conversion operator is identified and the corresponding tensor is split in vector split mode can the operator be determined as the computational subgraph to be optimized. In a single-card network, vector splitting maintains the consistency of the word embedding model, so that the vector representation of each word remains unchanged before and after splitting, which helps the model understand and process vocabulary. In a multi-card network, since multiple computing nodes are involved, using word splitting allows each node to be responsible for only part of the vocabulary, which can reduce the memory pressure of a single node and can process data in parallel. In addition, word splitting allows each node to process its own data slice independently, which can reduce the data transmission requirements between nodes. Each node only needs to exchange prediction results or model updates with other nodes, rather than the entire vocabulary.

[0111] S303: Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0112] S304: Adjust the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator.

[0113] The technical solution of the embodiment of the present disclosure can realize distributed labeling based on single-card networking through the optimization method of distributed labeling of computational graphs, automatically apply the vocabulary segmentation strategy to multi-card networking, and optimize the corresponding distributed labeling, thereby ensuring the ease of use of automatic parallelism in the vocabulary segmentation scenario of the embedded vector layer, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0114] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0115] Inserting a new embedding vector conversion operator in the word segmentation mode after the original embedding vector conversion operator in the vector segmentation mode, and redirecting the input of the original embedding vector conversion operator to the input of the new embedding vector conversion operator;

[0116] Inserting a communication protocol operator after the new embedding vector conversion operator, and linking the output of the new embedding vector conversion operator to the input of the communication protocol operator;

[0117] Linking the operator input receiving the output of the original embedding vector conversion operator in the computation graph to the output of the communication protocol operator;

[0118] Remove the original embedding vector conversion operator.

[0119] The original embedding vector conversion operator is replaced with a new embedding vector conversion operator (c_embedding) in word segmentation mode and aggregated with the communication protocol operator. The input of the original embedding vector conversion operator is redirected to the input of the new embedding vector conversion operator to receive the input tensor and convert it into row segmentation form. The output of the new embedding vector conversion operator is linked to the input of the communication protocol operator. The operator input that receives the output of the original embedding vector conversion operator is changed to receive the output of the communication protocol operator. The output results of each new embedding vector conversion operator are collected and output externally through the communication protocol operator, realizing the mode of replacing the original embedding vector conversion operator in vector segmentation mode with the new embedding vector conversion operator and communication protocol operator in word segmentation mode. The vector segmentation of single-card networking is automatically replaced with the word segmentation strategy of multi-card networking, ensuring the efficiency and performance of automatic parallel distributed training.

[0120] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator includes:

[0121] According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the vector segmentation mark in the computation graph, which is the original tensor distributed mark, is adjusted to a word segmentation mark, and the output tensor mark of the new embedding vector conversion operator is adjusted to a partial segmentation mark;

[0122] According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

[0123] According to the operating logic of the new embedding vector conversion operator in the word segmentation mode, the original tensor distributed mark is adjusted to the word segmentation mark (row segmentation), the output tensor mark is adjusted to the partial segmentation mark, and the segmentation mark of the output tensor of the communication protocol operator is set to the copy state, that is, each computing node will store a complete copy of the tensor output. By simply superimposing the output tensors of each partial segmentation method, the final output tensor result is obtained, ensuring the automatic parallel generation of the embedding vector results in the form of word segmentation, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelization, and improving the efficiency and performance of distributed training.

[0124] Figure 4 This is a schematic diagram of another optimization method for distributed marking of a computational graph provided according to an embodiment of the present disclosure. This embodiment concretizes the operation of "identifying the corresponding computational subgraph to be optimized according to the optimization paradigm in the computational graph" in another preferred embodiment as follows: "In the computational graph, a computational subgraph including a communication acquisition operator and a classification cross entropy operator is identified, and when the column splitting mode is determined based on the distributed marking of the input tensor of the communication acquisition operator, the computational subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be the computational subgraph to be optimized."

[0125] Accordingly, reference Figure 4 , the method may specifically include:

[0126] S401. Obtain a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy.

[0127] S402. In the computation graph, when a computation subgraph including a communication acquisition operator and a classification cross entropy operator is identified, and when a column splitting mode is determined based on the distributed labeling of the input tensor of the communication acquisition operator, the computation subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be a computation subgraph to be optimized.

[0128] The communication collection operator (all_gather) can be specifically understood as the operation of communicating with other nodes and collecting the calculation results of other nodes. The classification cross entropy operator can be specifically understood as the operation of calculating the classification probability and cross entropy of the data.

[0129] In a single-card network, since data is usually processed in batches and each batch contains multiple samples, column splitting (splitting along the category dimension) can parallelize the calculation of the classification probability and cross-entropy loss for each category, improving the processing efficiency of a single card. In a multi-card network, row splitting (splitting along the sample dimension) means that each computing node processes a portion of the samples in the entire batch and can independently calculate the loss of the samples it is responsible for, reducing the data processing dimension. It also aggregates gradients through backpropagation, reduces inter-node communication, and improves the efficiency of distributed training. Therefore, in the classification cross entropy optimization scenario, the column splitting method can be optimized, that is, the computational subgraph that includes the communication acquisition operator and the classification cross entropy operator, and the input tensor of the communication acquisition operator is distributed and marked as a column split mode, is selected as the computational subgraph to be optimized.

[0130] S403: Adjust the computation subgraph to be optimized according to the optimization paradigm.

[0131] S404: Adjust the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator.

[0132] The technical solution of the embodiment of the present disclosure automatically optimizes the classification cross entropy calculation strategy in the form of column segmentation in a single-card network to a classification cross entropy calculation strategy in the form of row segmentation in a multi-card network, thereby reducing the data processing dimension, reducing communication between nodes, ensuring the ease of use of automatic parallelization, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelization, and improving the efficiency and performance of distributed training.

[0133] In an optional implementation of this embodiment, adjusting the computation subgraph to be optimized according to the optimization paradigm includes:

[0134] Insert a new parallel categorical cross entropy operator in row split mode after the original categorical cross entropy operator in column split mode, and redirect the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator;

[0135] Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator;

[0136] Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator;

[0137] Remove the communication acquisition operator and the original classification cross entropy operator.

[0138] The original categorical cross entropy operator is a column-splitting categorical cross entropy calculation operation suitable for single-SIM networking. The communication collection operator is an operation that collects the output results of the original categorical cross entropy operator, suitable for single-SIM networking. In multi-SIM networking, it needs to be replaced with the new parallel categorical cross entropy operator using row-splitting mode and the corresponding communication protocol operator.

[0139] Insert a new parallel classification cross entropy operator and communication protocol operator in the row split mode, redirect the input of the original classification cross entropy operator to the input of the new parallel classification cross entropy operator to receive data and calculate the classification probability and cross entropy, link the output of the new parallel classification cross entropy operator to the input of the communication protocol operator, and link the operator input that receives the output of the original classification cross entropy operator to the output of the communication protocol operator, so that the calculation result is transferred from the new parallel classification cross entropy operator in the row split mode to the communication protocol operator, remove the communication acquisition operator and the original classification cross entropy operator, and replace the original classification cross entropy operator and communication acquisition operator in the column split mode with the new parallel classification cross entropy operator and communication protocol operator in the row split mode. Automatically replace the column splitting of single-card networking with the row splitting strategy of multi-card networking to ensure the efficiency and performance of automatic parallel distributed training.

[0140] In an optional implementation of this embodiment, adjusting the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator includes:

[0141] According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark used as the original tensor distributed mark in the computation graph is adjusted to a row split mark;

[0142] According to the operation logic of the communication protocol operator, the output tensor of the communication protocol operator is added with a copy state.

[0143] According to the operating logic of the new parallel categorical cross entropy operator in the split mode, the original tensor distribution mark is adjusted to the row split mark, and the split mark of the output tensor of the communication protocol operator is set to the copy state, that is, each computing node will store a complete copy of the tensor output. By storing each output tensor, the output tensor results of each time are obtained, ensuring the automatic parallel generation of the categorical cross entropy calculation results in the form of row splits, reducing the difficulty and cost of using the optimization strategy, enhancing the adaptability and flexibility of automatic parallelism, and improving the efficiency and performance of distributed training.

[0144] In deep learning, the categorical cross-entropy operator (softmax_with_cross_entropy) is a commonly used loss function. Its calculation consists of two parts: softmax and cross-entropy. In model parallel scenarios, the complete model output is usually required on each compute node for loss calculation. This is usually achieved through an all-gather operation, but this incurs significant communication overhead. To optimize this process, the parallel categorical cross-entropy operator (c_softmax_with_cross_entropy) optimizer reduces communication overhead by: 1. Local computation: Softmax and cross-entropy calculations are performed locally on each compute node. 2. Communication reduction: All-gather communication is converted to all-reduce communication. Specifically, partial sums are first calculated on each compute node, and then these results are aggregated through an all-reduce operation. This reduces the communication overhead from b*s*v (batch size * sequence length * vocabulary size) to b*s (batch size * sequence length).

[0145] For ease of understanding, the specific application scenarios applicable to each embodiment of the present disclosure are now described. In the context of the continuous increase in models, it is necessary to give full play to the computing power resources of the hardware to achieve efficient distributed training, which usually requires the application of various distributed optimization strategies. In automatic parallel mode, most optimization strategies can be automatically applied by identifying the distributed tags of tensors marked by users in single-card networking. However, there are still some optimization strategies based on multi-card networking, which contain optimization operators unique to multi-card networking and are associated with multi-card communication operations, which makes them unable to be used in single-card networking; at the same time, these strategies cannot be described by distributed tags, that is, users cannot describe these optimization strategies by distributed tags in single-card networking, which increases the difficulty and cost of using optimization strategies, reduces the adaptability and flexibility of automatic parallelism, and reduces the efficiency and performance of distributed training. To solve the above problems, the embodiments of the present disclosure propose an automatic parallel distributed tag optimization method.

[0146] Distributed tagging inserts distributed tag information into the model's dynamic graph. In automatic parallel mode, users only need to perform distributed tagging on a subset of tensors in the model. Automatic parallelization automatically extends this distributed tag information to the entire computational graph using segmentation inference rules.

[0147] Distributed tag optimization is a technology that automatically applies different distributed optimization strategies by replacing and splitting computational subgraphs within a single-card network computation graph that has been distributedly tagged. Specifically, distributed tag optimization enables the application of various optimization strategies based on distributed networking, especially multi-card networking. Distributed tag optimization performs subgraph matching on the tagged computation graph to identify the computational subgraphs that require optimization. It then replaces operators, adds or subtracts operators, and performs other operations on these computational subgraphs based on the desired optimization strategy. Finally, the entire computation graph is re-split and re-derived. This method allows distributed networking-based optimization strategies to be effectively implemented on single-card networks.

[0148] Figure 5 This is a schematic diagram of an automatic parallel distributed marking optimization applicable to the embodiment of the present disclosure, refer to Figure 5 ,The main process of distributed marking optimization includes subgraph matching, ,operator replacement and re-segmentation deduction.

[0149] 1. Get the model calculation graph

[0150] In automatic parallelization, the forward network with labeled information input by the user is first generated into a forward, backward and optimizer calculation graph with distributed information. This calculation graph is the basis for subsequent optimization steps. Figure 5 As shown in (1), after the user marks the split state of a two-dimensional tensor Y (such as replicate and shard), the framework will automatically generate the split state of the tensor R according to the specific split inference rules.

[0151] 2. Subgraph matching

[0152] Analyze and identify the features of the entire computational graph, and use existing paradigms to match the subgraphs that need to be optimized. Figure 5 As shown in (2), by performing subgraph matching on the entire computational graph, the operators OP1 and OP2 in the box, as well as the corresponding input tensors X and Y, conform to the paradigm. There may be multiple subgraphs in a computational graph that match the paradigm, and all of these subgraphs need to be replaced and optimized.

[0153] 3. Sub-image replacement

[0154] Based on the matched subgraph, the specific operators are replaced to make them conform to the state in the optimization strategy. Figure 5 As shown in (3), for all matched subgraphs, the operators in them are replaced according to the logic of the optimization strategy. Operator OP1 in the figure is replaced with new operator OP1. For more complex cases, operators can be deleted or inserted multiple times until the subgraph meets the optimization strategy.

[0155] 4. Modify the segmentation status

[0156] After replacing a subgraph, the affected tensors need to be re-sliced ​​to ensure the correctness of the computation graph. Figure 5 As shown in (4), after modifying the operator in the subgraph, the tensor segmentation state in the subgraph also needs to be modified. In the figure, the segmentation state of tensor Y is modified to make it meet the input requirements of the new operator OP1.

[0157] 5. Re-segmentation derivation

[0158] Based on the replaced subgraph, the entire computation graph is re-split and deduced, including forward and backward deduction, until the state of a certain operator is consistent with the original state. This ensures that the state of the entire computation graph reaches the optimization goal and all necessary communication operations and data distribution are correctly configured. Figure 5 As shown in (5), after the entire subgraph is replaced, the segmentation state of the entire computational graph needs to be re-derived. In the figure, the segmentation state of the output tensor R of the operator OP2 is modified by re-segmentation derivation.

[0159] After completing the above distributed labeling optimization, the execution module will start distributed training of the model based on the calculation graph after distributed labeling optimization.

[0160] Figure 6 This is a schematic diagram of distributed marking optimization implemented by a setting tool software applicable to the embodiment of the present disclosure. Figure 6 , distributed marking optimization can be implemented by setting tool software (Pass). For example, a technology used in compiler design and optimization can be specifically understood as a stage in the compilation process, during which the compiler performs specific analysis or transformations on the program code. In the deep learning framework, it refers to the gradual optimization of the computation graph, responsible for analyzing, modifying or replacing part of the computation graph to achieve specific optimization goals. Figure 6 , the subgraph matching paradigm for a specific optimization strategy needs to be implemented in the configuration tool software. After matching the subgraph, the operator replacement and tensor segmentation state modification need to be completed in the configuration tool software. At the end of the configuration tool software execution, the segmentation inference rules need to be called to re-segment and infer the entire input computational subgraph to ensure the correctness of the entire computational graph segmentation state. It is worth noting that the configuration tool software needs to replace and modify before loading parameters to avoid loading the parameter tensor with the old segmentation state first and then updating it to the new segmentation state through communication operations.

[0161] In a preferred disclosed embodiment, the automatic parallel distributed labeling optimization method proposed in the disclosed embodiment is applicable to the application scenario of parallel column splitting embedding vector optimization.

[0162] In large-scale distributed training, the vocabulary of the embedding layer can be very large. For example, if 1,000 words are represented by a 300-dimensional vector, the vocabulary is a 1,000-by-300 matrix. The actual vocabulary dimensions are much larger. Performing all embedding calculations on a single GPU would impact both graphics memory and computational efficiency. Computational efficiency can be improved by partitioning the vocabulary and running parallel computations on multiple GPUs. The parallel c_embedding (column splitting) optimization reduces graphics memory and computational overhead by performing row-wise (word-wise) splitting and all_reduce (communication protocol reduction) operations on the vocabulary.

[0163] Figure 7 This is a schematic diagram of a parallel column split embedding vector optimization applicable to the embodiment of the present disclosure, refer to Figure 7 , builtin.parameter refers to built-in parameters, that is, parameters in the model, such as weights or biases, which are learned during the training process. data refers to the data input to the model. embedding refers to the conversion of input data (usually word indexes) into vector representations in a high-dimensional space. transpose refers to the transposition operation, that is, exchanging the dimensions of the tensor, such as swapping the rows and columns of the matrix. all_gather refers to the communication collection operator, which is used to collect data from all nodes to each node in a distributed system. full_int_array refers to a full integer array, that is, an array filled with integers, which is used to restore the dimensions of the original data after the transposition operation. multiply refers to the multiplication operation, which is used to perform element-level multiplication. T[x,x] refers to the splitting state of the tensor Tensor in different dimensions. r refers to replicate(), which is a full copy in this dimension. s(x) refers to shard(x), which is split in the x dimension of the machine in this dimension. p(x) refers to partial(x), which is only partial data in the x dimension of the tensor.

[0164] refer to Figure 7 , the optimization method may specifically include:

[0165] 1. Get the model calculation graph

[0166] Get the computational graph with complete state after segmentation and deduction.

[0167] 2. Subgraph matching

[0168] Based on the optimization strategy logic, the entire computational graph is matched with subgraphs to identify whether the model is a parallel structure. Parallel dimension information is obtained, the locations of all embedding operations are determined, and the input parameter segmentation method is checked. Only column-segmented vocabulary is optimized.

[0169] 3. Sub-image replacement

[0170] The core is to replace the embedding operation with c_embedding and all_reduce operations.

[0171] Insert the c_embedding operator after the embedding operator to redirect the original embedding input to c_embedding. Insert the all_reduce operator after c_embedding to link the output of c_embedding to the input of all_reduce. Update all operations in the computation graph that receive the output of embedding to receive the output of all_reduce. Remove all embedding operations from the original computation graph.

[0172] 4. Modify the segmentation status

[0173] Although embedding and c_embedding use the same vocabulary weights, their sharding states are different. Change the vocabulary weight sharding state of c_embedding to row sharding (sharding), and change the output from column sharding to partial sharding (partial). Mark the output of all_reduce as replicated (replicate).

[0174] 5. Re-segmentation derivation

[0175] For operators outside the subgraph, since the input and output segmentation states of the input subgraph are modified, they also need to be modified synchronously. According to the topological relationship of the computational graph, traverse the computational graph forward or backward from the subgraph, and call the segmentation inference rule of each operator.

[0176] Calculate the split state of its input or output, and determine whether the split state is equal to the current state. If not, replace the current split state with the newly derived result.

[0177] In another preferred disclosed embodiment, the automatic parallel distributed labeling optimization method proposed in the disclosed embodiment is applicable to the application scenario of parallel column segmentation probability distribution cross entropy optimization.

[0178] In deep learning, softmax_with_cross_entropy is a commonly used loss function. Its calculation includes both softmax and cross-entropy parts. In the model parallel scenario, it is usually required to have the complete model output on each GPU for loss calculation. This is usually achieved through the all-gather operation, but this will bring a lot of communication overhead. To optimize this process, the parallel c_softmax_with_cross_entropy optimization reduces the communication amount in the following ways:

[0179] 1. Local calculation

[0180] First, perform local softmax and cross-entropy calculation on each GPU.

[0181] 2. Reduce communication

[0182] Convert all-gather communication to all-reduce communication. Specifically, first calculate the partial sum on each GPU, and then aggregate the results through the all-reduce operation. In this way, the communication amount is reduced from b×s×v (batch size × sequence length × vocabulary size) to b×s (batch size × sequence length).

[0183] Figure 8 is a schematic diagram of a parallel column-cut probability distribution cross-entropy optimization applicable to embodiments of the present disclosure, referring to Figure 8 matmul refers to matrix multiplication, which is used to calculate the product of two matrices. full refers to the padding operation, which is used to create a tensor filled with a specific value. c_softmax_with_cross_entropy: optimized function of softmax with cross entropy in row-cut form. subtract refers to subtraction, which is used to calculate the difference between two tensors. cross_entropy_with_softmax refers to the cross-entropy and softmax function calculation operation, which is used for loss calculation in multi-classification problems. First, apply the softmax function to convert the output into a probability distribution, and then calculate the cross-entropy loss between these probabilities and the true label. T[x,x] refers to the slicing state of the tensor Tensor in different dimensions. r refers to replicate() in this dimension. s(x) refers to shard(x) in the x dimension of the machine. p(x) refers to partial(x) in the x dimension of the tensor.

[0184] Referring to Figure 8 , the optimization method can be specifically:

[0185] 1. Obtain the model computation graph

[0186] Get the computational graph with complete state after segmentation and deduction.

[0187] 2. Subgraph matching

[0188] Traverse all operations in the computation graph.

[0189] Identify all-gather and cross_entropy_with_softmax operator logic segments.

[0190] Check how the input tensor is sliced ​​and make sure it is sliced ​​in the column direction.

[0191] 3. Sub-image replacement

[0192] Replace the all-gather and cross_entropy_with_softmax operations with c_softmax_with_cross_entropy and all_reduce operations.

[0193] Insert the c_softmax_with_cross_entropy operator after cross_entropy_with_softmax and redirect the input of the cross_entropy_with_softmax operator in the computation graph to the input of c_softmax_with_cross_entropy.

[0194] Go ahead and insert the all_reduce operation and link the output of c_softmax_with_cross_entropy to the input of all_reduce.

[0195] Update all operations in the computation graph that receive the output of cross_entropy_with_softmax to receive the output of all_reduce.

[0196] Delete the all-gather+cross_entropy_with_softmax operator logic segment in the original calculation graph.

[0197] 4. Modify the segmentation status

[0198] Update the splitting status of c_softmax_with_cross_entropy and all_reduce.

[0199] 5. Re-segmentation derivation

[0200] According to the new segmentation status, traverse the computation graph to update the segmentation status of other related operations to ensure the consistency of the computation graph.

[0201] The automatic parallel distributed marking optimization method proposed in the embodiment of the present disclosure directly optimizes and modifies the calculation graph based on the existing distributed marking of single-card networking, without the need for users to manually modify the model and network segmentation status, or use distributed networking. This lowers the threshold for users to apply various optimization strategies when using automatic parallelization to complete distributed training, and improves the ease of use and efficiency of using optimization strategies under automatic parallelization.

[0202] As an implementation of the above-mentioned optimization method for distributed marking of each computational graph, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned optimization method for distributed marking of each computational graph.

[0203] Figure 9 This is a structural diagram of an optimization device for distributed marking of a computational graph provided according to an embodiment of the present disclosure. Figure 9 The device includes: a graph acquisition module 901, a graph identification module 902 to be optimized, a graph adjustment module 903 and a mark adjustment module 904, wherein:

[0204] A graph acquisition module 901 is used to acquire a computational graph of a distributed operation model and an optimization paradigm of a target optimization strategy;

[0205] A graph identification module 902 for identifying a corresponding computation subgraph to be optimized in the computation graph according to the optimization paradigm;

[0206] A graph adjustment module 903 is configured to adjust the computation subgraph to be optimized according to the optimization paradigm;

[0207] A tag adjustment module 904 is configured to adjust the tensor distribution tags in the computation graph according to the original tensor distribution tags in the computation graph and the operation logic of the operator;

[0208] Among them, the calculation graph before adjustment is the model calculation graph supported by single-card networking, and the calculation graph after adjustment is the model calculation graph supported by multi-card networking.

[0209] The technical solution of the embodiment of the present disclosure obtains the computational graph of the distributed operation model and the target computational subgraph of the target optimization strategy; in the computational graph, identifies the computational subgraph to be optimized corresponding to the target computational subgraph; uses the target computational subgraph to adjust the computational subgraph to be optimized; and adjusts the tensor distribution labels in the computational graph according to the original tensor distribution labels in the computational graph and the operation logic of the target computational subgraph. The target computational subgraph is used to automatically identify and convert computational subgraphs of multiple optimization strategies that are integrated into distributed networking, ensuring the ease of use of automatic parallelization, reducing the difficulty and cost of using optimization strategies, enhancing the adaptability and flexibility of automatic parallelization, and improving the efficiency and performance of distributed training.

[0210] Based on the above embodiments, the graph acquisition module 901 is specifically configured to:

[0211] Obtain a computational graph of the model and some manually added tensor distribution labels in the computational graph;

[0212] According to the partial tensor distributed labels, automatic parallel split calculation is adopted to derive the remaining tensor distributed labels in the computation graph and add them to the computation graph.

[0213] On the basis of the above embodiments, the calculation graph before automatic parallel calculation is a model calculation graph supported by a single-card network, and the calculation graph after automatic parallel calculation is a model calculation graph supported by a multi-card network.

[0214] On the basis of the above embodiments, the target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networking.

[0215] Based on the above embodiments, the to-be-optimized graph identification module 902 is specifically configured to:

[0216] The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

[0217] In an optional embodiment, based on the above embodiments, the graph to be optimized identification module 902 may include: a candidate graph matching unit and a graph to be optimized determination unit, wherein:

[0218] A candidate graph matching unit, configured to perform traversal matching on the computation graph using the operator conditions in the optimization conditions to determine a matching candidate computation subgraph;

[0219] The unit for determining a graph to be optimized is configured to determine the candidate computation subgraph as the computation subgraph to be optimized if it is identified that a segmentation method satisfies the segmentation requirements in the optimization condition based on the tensor distributed label of the candidate computation subgraph.

[0220] Based on the above embodiments, the graph adjustment module 903 is specifically configured to:

[0221] According to the optimization operation in the optimization paradigm, inserting and deleting operators are performed in the computation subgraph to be optimized to adjust the computation subgraph to be optimized.

[0222] Based on the above embodiments, the mark adjustment module 904 is specifically configured to:

[0223] From the adjusted operator in the computation graph, a forward traversal and / or a backward traversal is performed, and the tensor distribution labels in the computation graph are adjusted according to the original tensor distribution labels of the computation graph and the running logic of the operator, until the tensor distribution labels determined according to the running logic are consistent with the original tensor distribution labels.

[0224] In an optional embodiment, on the basis of each of the above embodiments, wherein the target optimization strategy is an embedding vector layer vocabulary segmentation strategy, the computation subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator of a vocabulary segmentation mode and a communication reduction operator.

[0225] Further, on the basis of each of the above embodiments, wherein the to-be-optimized graph identification module 902 can be specifically used for:

[0226] In the computation graph, an embedding vector conversion operator is identified, and when it is determined that the input tensor distribution label of the embedding vector conversion operator is a vector segmentation mode, the embedding vector conversion operator is determined to be a to-be-optimized computation subgraph.

[0227] Further, on the basis of each of the above embodiments, wherein the graph adjustment module 903 can be specifically used for:

[0228] After the original embedding vector conversion operator of the vector segmentation mode, a new embedding vector conversion operator of a vocabulary segmentation mode is inserted, and the input of the original embedding vector conversion operator is redirected to the input of the new embedding vector conversion operator;

[0229] A communication reduction operator is inserted after the new embedding vector conversion operator, and the output of the new embedding vector conversion operator is linked to the input of the communication reduction operator;

[0230] In the computation graph, the input of an operator receiving the output of the original embedding vector conversion operator is linked to the output of the communication reduction operator;

[0231] The original embedding vector conversion operator is removed.

[0232] Further, on the basis of each of the above embodiments, wherein the label adjustment module 904 can be specifically used for:

[0233] According to the running logic of the new embedding vector conversion operator of the vocabulary segmentation mode, the vector segmentation label as the original tensor distribution label in the computation graph is adjusted to a vocabulary segmentation label, and the output tensor label of the new embedding vector conversion operator is adjusted to a partial segmentation label;

[0234] According to the running logic of the communication reduction operator, a copy label is added to the output tensor of the communication reduction operator.

[0235] In an optional embodiment, based on the above embodiments, the target optimization strategy is a parallel classification probability and cross entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross entropy operator and a communication protocol operator.

[0236] Furthermore, based on the above embodiments, the to-be-optimized graph identification module 902 may be further specifically configured to:

[0237] In the computational graph, when a computational subgraph including a communication acquisition operator and a classification cross entropy operator is identified and a column splitting mode is determined based on the distributed marking of the input tensor of the communication acquisition operator, the computational subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be a computational subgraph to be optimized.

[0238] Furthermore, based on the above embodiments, the graph adjustment module 903 may be further configured to:

[0239] Insert a new parallel categorical cross entropy operator in row split mode after the original categorical cross entropy operator in column split mode, and redirect the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator;

[0240] Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator;

[0241] Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator;

[0242] Remove the communication acquisition operator and the original classification cross entropy operator.

[0243] Furthermore, based on the above embodiments, the mark adjustment module 904 may be further configured to:

[0244] According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark used as the original tensor distributed mark in the computation graph is adjusted to a row split mark;

[0245] According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

[0246] The above-mentioned product can execute the optimization method for distributed marking of computational graphs provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects for executing the optimization method for distributed marking of computational graphs.

[0247] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0248] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0249] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0250] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0251] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0252] The computing unit 1001 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the optimization method for distributed labeling of computational graphs, that is,

[0253] Obtain the computational graph of the distributed operation model and the target computational subgraph of the target optimization strategy;

[0254] In the computation graph, identifying the computation subgraph to be optimized corresponding to the target computation subgraph;

[0255] Adopting the target computation subgraph to adjust the computation subgraph to be optimized;

[0256] Adjust the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the target computation subgraph.

[0257] For example, in some embodiments, the optimization method for distributed labeling of computational graphs may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the optimization method for distributed labeling of computational graphs described above may be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to perform the optimization method for distributed labeling of computational graphs in any other appropriate manner (e.g., by means of firmware).

[0258] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0260] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0262] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0263] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0264] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0265] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0266] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not a limitation herein.

[0267] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for optimizing distributed labeling of computational graphs, comprising: Obtain the computational graph of the distributed running model and the optimization paradigm of the target optimization strategy; In the computation graph, identifying a corresponding computation subgraph to be optimized according to the optimization paradigm; Adjust the computation subgraph to be optimized according to the optimization paradigm; Adjusting the tensor distribution marks in the computation graph according to the original tensor distribution marks in the computation graph and the operation logic of the operator; The calculation graph before adjustment is the model calculation graph supported by a single-SIM card network, and the calculation graph after adjustment is the model calculation graph supported by a multi-SIM card network. The target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networking; Wherein, adjusting the computation subgraph to be optimized according to the optimization paradigm includes: According to the optimization operation in the optimization paradigm, inserting and deleting operators in the computation subgraph to be optimized to adjust the computation subgraph to be optimized; According to the original tensor distribution mark in the computation graph and the operation logic of the operator, adjusting the tensor distribution mark in the computation graph includes: Starting from the adjusted operator in the computation graph, forward traversal and / or backward traversal are performed, and according to the original tensor distribution marks of the computation graph and the operation logic of the operator, the tensor distribution marks in the computation graph are adjusted until the tensor distribution marks determined according to the operation logic are consistent with the original tensor distribution marks; In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

2. The method according to claim 1, wherein Obtaining the computational graph of a distributed running model includes: Obtain the computational graph of the model and some tensor distributed markers pre-added in the computational graph; According to the partial tensor distributed labels, automatic parallel computing is used to derive the remaining tensor distributed labels in the computation graph and add them to the computation graph.

3. The method according to claim 1, wherein Using the optimization conditions in the optimization paradigm, traversing and matching are performed in the computation graph to determine that the matching computation subgraphs to be optimized include: Using the operator conditions in the optimization conditions, traversing and matching in the computation graph to determine a matching candidate computation subgraph; If, based on the tensor distribution mark of the candidate computation subgraph, it is identified that the distribution mode meets the segmentation requirement in the optimization condition, the candidate computation subgraph is determined as the computation subgraph to be optimized.

4. The method according to any one of claims 1 to 3, wherein: The target optimization strategy is an embedding vector layer vocabulary segmentation strategy, and the computational subgraph adjusted according to the optimization paradigm includes an embedding vector conversion operator and a communication protocol operator of a word segmentation pattern.

5. The method according to claim 4, wherein In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: In the computation graph, when an embedded vector conversion operator is identified and a vector segmentation mode is determined based on the distributed labeling of the input tensor of the embedded vector conversion operator, the embedded vector conversion operator is determined to be a computation subgraph to be optimized.

6. The method according to claim 5, wherein: Adjusting the computation subgraph to be optimized according to the optimization paradigm includes: Inserting a new embedding vector conversion operator in the word segmentation mode after the original embedding vector conversion operator in the vector segmentation mode, and redirecting the input of the original embedding vector conversion operator to the input of the new embedding vector conversion operator; Inserting a communication protocol operator after the new embedding vector conversion operator, and linking the output of the new embedding vector conversion operator to the input of the communication protocol operator; Linking the operator input receiving the output of the original embedding vector conversion operator in the computation graph to the output of the communication protocol operator; Remove the original embedding vector conversion operator.

7. The method according to claim 6, wherein: Adjusting the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator includes: According to the operation logic of the new embedding vector conversion operator of the word segmentation mode, the vector segmentation mark in the computation graph, which is the original tensor distributed mark, is adjusted to a word segmentation mark, and the output tensor mark of the new embedding vector conversion operator is adjusted to a partial segmentation mark; According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

8. The method according to any one of claims 1 to 3, wherein: The target optimization strategy is a parallel classification probability and cross entropy loss calculation strategy, and the calculation subgraph adjusted according to the optimization paradigm includes a parallel classification cross entropy operator and a communication protocol operator.

9. The method according to claim 8, wherein In the computation graph, identifying the corresponding computation subgraph to be optimized according to the optimization paradigm includes: In the computational graph, when a computational subgraph including a communication acquisition operator and a classification cross entropy operator is identified and a column splitting mode is determined based on the distributed marking of the input tensor of the communication acquisition operator, the computational subgraph including the communication acquisition operator and the classification cross entropy operator is determined to be a computational subgraph to be optimized.

10. The method according to claim 9, wherein: Adjusting the computation subgraph to be optimized according to the optimization paradigm includes: Insert a new parallel categorical cross entropy operator in row split mode after the original categorical cross entropy operator in column split mode, and redirect the input of the original categorical cross entropy operator to the input of the new parallel categorical cross entropy operator; Inserting a communication protocol operator after the new parallel classification cross entropy operator, and linking the output of the new parallel classification cross entropy operator to the input of the communication protocol operator; Linking the operator input in the computation graph that receives the output of the original classification cross entropy operator to the output of the communication protocol operator; Remove the communication acquisition operator and the original classification cross entropy operator.

11. The method according to claim 10, wherein: Adjusting the tensor distribution mark in the computation graph according to the original tensor distribution mark in the computation graph and the operation logic of the operator includes: According to the operation logic of the new parallel classification cross entropy operator of the row split mode, the column split mark in the computation graph, which is used as the original tensor distributed mark, is adjusted to a row split mark; According to the operation logic of the communication protocol operator, a copy mark is added to the output tensor of the communication protocol operator.

12. An optimization device for distributed labeling of a computational graph, comprising: The graph acquisition module is used to obtain the computational graph of the distributed operation model and the optimization paradigm of the target optimization strategy; A graph identification module to be optimized, configured to identify, in the computation graph, the corresponding computation subgraph to be optimized according to the optimization paradigm; A graph adjustment module, configured to adjust the computational subgraph to be optimized according to the optimization paradigm; A tag adjustment module, configured to adjust the tensor distribution tags in the computation graph according to the original tensor distribution tags in the computation graph and the operation logic of the operator; The calculation graph before adjustment is the model calculation graph supported by a single-SIM card network, and the calculation graph after adjustment is the model calculation graph supported by a multi-SIM card network. The target optimization strategy corresponds to the distributed optimization strategy supported by multi-card networking; The graph adjustment module is specifically used to: According to the optimization operation in the optimization paradigm, inserting and deleting operators in the computation subgraph to be optimized to adjust the computation subgraph to be optimized; The mark adjustment module is specifically used to: Starting from the adjusted operator in the computation graph, forward traversal and / or backward traversal are performed, and according to the original tensor distribution marks of the computation graph and the operation logic of the operator, the tensor distribution marks in the computation graph are adjusted until the tensor distribution marks determined according to the operation logic are consistent with the original tensor distribution marks; The graph recognition module to be optimized is specifically used for: The optimization conditions in the optimization paradigm are adopted to perform traversal matching in the computation graph to determine a matching computation subgraph to be optimized.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images