Automatic parallel strategy search method based on polyhedron modeling and related equipment

Abstract

The present invention discloses an automatic parallel strategy search method based on polyhedron model modeling and related equipment. The method includes: obtaining a model calculation graph of a deep learning algorithm based on a model object input by a user; converting the model calculation graph to obtain a converted model calculation graph; balancing the converted model calculation graph to obtain a balanced calculation graph; creating a polyhedron model instance based on the balanced calculation graph, and outputting a parallel strategy based on the polyhedron model instance; and calling an underlying framework to execute the parallel strategy. In the present invention, by converting and balancing the model calculation graph, and automatically outputting a parallel strategy after creating a polyhedron model instance under the framework of the polyhedron model, different algorithm logics are modeled under the polyhedron model, and the parallel strategy process is automatically output, thereby improving the efficiency of parallel strategy search and reducing the difficulty of distributed training development and efficiency tuning of deep learning algorithms.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an automatic parallel strategy search method based on polyhedron modeling and related equipment. Background Art

[0002] Over the past decade, deep learning technology has continuously broken records in various tasks across fields like vision, natural language, speech, search, and recommendations. The reason for this can be described in one key word: "large scale." Large-scale data allows models to retain sufficient knowledge, models with large parameter counts enable them to retain even more data, and large-scale, high-performance computing power (typically represented by GPUs) increases model training speed by hundreds or even thousands of times. The development of data, models, and computing power has given rise to the field of large-scale deep learning. Key research areas in this direction include how to split tasks across multiple machines, how to configure cluster training resources, how to balance training speed and convergence speed, how to train models that cannot be trained on a single machine, and elastic training and fault tolerance. Distributed training is the most effective way to address these issues and improve training efficiency. The core purpose of distributed training is to accelerate model training.

[0003] At present, mainstream deep learning frameworks such as TensorFlow (TensorFlow is a symbolic mathematical system based on data flow programming, which is widely used in the programming implementation of various machine learning algorithms), Pytorch (PyTorch is an open source Python machine learning library based on Torch, used for applications such as natural language processing), Mindspore (MindSpore is a new open source deep learning training / inference framework suitable for edge-cloud scenarios), and PaddlePaddle (PaddlePaddle is a technologically advanced and fully functional open source deep learning platform that integrates deep learning core framework, tool components and service platform) all have multi-machine distribution. The main parallel modes of distributed training include data parallelism (data parallelism refers to the process of dividing training data samples into multiple computing devices for distributed computing during distributed training of AI models), operator parallelism, and pipeline parallelism (pipeline technology refers to a quasi-parallel processing implementation technology in which multiple instructions overlap to perform operations during program execution). However, these parallel modes require algorithm developers to call the parallel splitting API provided by the AI ​​framework according to the characteristics of the algorithm model. This method increases the technical difficulty of distributed training of AI algorithms. At the same time, due to the algorithm developers' insufficient understanding of the characteristics of the AI ​​framework and computing devices, the efficiency of model parallel training is low. The specific distributed tuning work increases the difficulty of algorithm development and reduces the efficiency of algorithm research.

[0004] To address this challenge, the Mindspore framework offers automatic parallel model training capabilities, the FlexFlow framework proposes a search strategy based on four-dimensional parallel policy space modeling, and the RaNNC framework proposes a pipelined parallel policy automatic search middleware that supports the Pytorch frontend. However, due to the large scale of the parallel policy search space (related to the scale of the computational graph and the resource space), these efforts struggle to achieve practical automatic parallel search efficiency. For example, when implementing pipelined parallel policy search for the 4.9B parameter BERT-enlarge model on a 4-node, 32-GPU system, the RaNNC framework required over four hours of policy search time. This significantly increased debugging and training time during model training and development, reducing efficiency. Therefore, existing technologies still require improvement and enhancement. Summary of the Invention

[0005] The main purpose of the present invention is to provide an automatic parallel strategy search method and related equipment based on polyhedron model modeling, aiming to solve the problem in the prior art that when training large-scale deep learning models, algorithm developers need to configure parallel strategies themselves, which will result in low training efficiency and great development difficulty.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An automatic parallel strategy search method based on polyhedron modeling, comprising:

[0008] Obtain the model calculation graph of the deep learning algorithm based on the model object input by the user;

[0009] Converting the model calculation graph to obtain a converted model calculation graph;

[0010] Performing equalization processing on the converted model calculation graph to obtain a balanced calculation graph;

[0011] Creating a polyhedron model instance according to the balanced calculation graph, and outputting a parallel strategy according to the polyhedron model instance;

[0012] The underlying framework is called to execute the parallel strategy.

[0013] In the automatic parallel strategy search method based on polyhedron modeling, the step of obtaining a model calculation graph of a deep learning algorithm based on a model object input by a user specifically includes:

[0014] Get an algorithm model based on the model object input by the user;

[0015] After randomly inputting a numerical value into the algorithm model, the calculation process of the algorithm model is recorded to obtain the model calculation graph;

[0016] Alternatively, the model object is parsed by a Python interpreter to generate a syntax tree, and then the syntax tree is analyzed to obtain the model calculation graph.

[0017] In the automatic parallel strategy search method based on polyhedron modeling, the step of converting the model calculation graph to obtain the converted model calculation graph specifically includes:

[0018] The model calculation graph is re-expressed using a pre-defined intermediate representation to obtain a converted model calculation graph.

[0019] In the automatic parallel strategy search method based on polyhedron modeling, the step of performing balancing processing on the converted model calculation graph to obtain the balanced calculation graph specifically includes:

[0020] Setting an average computation amount threshold for a node, and comparing the computation amount nodes in the converted model computation graph with the average computation amount threshold;

[0021] Adjacent computation nodes with computation amounts less than the average computation threshold are merged, and computation nodes with computation amounts greater than the average computation threshold are split to obtain a balanced computation graph.

[0022] In the automatic parallel strategy search method based on polyhedron modeling, the steps of creating a polyhedron model instance according to the balanced calculation graph and outputting a parallel strategy according to the polyhedron model instance specifically include:

[0023] Map the equilibrium calculation graph onto the polyhedron model to obtain a polyhedron optimization model;

[0024] Inputting the equilibrium calculation graph into the polyhedron optimization model to obtain a polyhedron model instance;

[0025] Output a parallel strategy according to the polyhedron model instance and the number of computing resources input by the user.

[0026] In the automatic parallel strategy search method based on polyhedron modeling, the step of calling the underlying framework to execute the parallel strategy specifically includes:

[0027] The execution API of the underlying framework is called to execute the parallel strategy.

[0028] In the automatic parallel strategy search method based on polyhedron modeling, the model object refers to a stand-alone training code of a deep learning algorithm pre-defined by a user.

[0029] In the automatic parallel strategy search method based on polyhedron modeling, the predefined intermediate representations include: IRType, IRValue, IRNode and IRGraph.

[0030] In the automatic parallel strategy search method based on polyhedron modeling, the parallel strategy includes a data parallel segmentation dimension and a pipeline parallel segmentation dimension.

[0031] In the automatic parallel strategy search method based on polyhedron modeling, the execution API is the run manager in the underlying AI framework.

[0032] An automatic parallel strategy search system, the automatic parallel strategy search system further comprising:

[0033] The computational graph generation module is used to obtain the model computational graph of the deep learning algorithm based on the model object input by the user;

[0034] A calculation graph conversion module, used to convert the model calculation graph to obtain a converted model calculation graph;

[0035] The calculation graph balancing module is used to balance the converted model calculation graph to obtain a balanced calculation graph;

[0036] A parallel strategy search module, configured to create a polyhedron model instance according to the balanced calculation graph, and output a parallel strategy according to the polyhedron model instance;

[0037] The parallel strategy execution module is used to call the underlying framework to execute the parallel strategy.

[0038] A controller comprises: a memory, a processor, and an automatic parallel strategy search program based on polyhedron modeling stored in the memory and executable on the processor. When the automatic parallel strategy search program based on polyhedron modeling is executed by the processor, the steps of the automatic parallel strategy search method based on polyhedron modeling as described above are implemented.

[0039] A computer-readable storage medium stores an automatic parallel strategy search program based on polyhedron modeling. When the automatic parallel strategy search program based on polyhedron modeling is executed by a processor, the steps of the automatic parallel strategy search method based on polyhedron modeling as described above are implemented.

[0040] Compared with the prior art, the present invention provides an automatic parallel strategy search method based on polyhedron model modeling and related equipment. The automatic parallel strategy search method based on polyhedron model modeling includes: obtaining a model calculation graph of a deep learning algorithm according to a model object input by a user; converting the model calculation graph to obtain a converted model calculation graph; balancing the converted model calculation graph to obtain a balanced calculation graph; creating a polyhedron model instance according to the balanced calculation graph, and outputting a parallel strategy according to the polyhedron model instance; calling the underlying framework to execute the parallel strategy. In the present invention, by converting and balancing the generated model calculation graph, and creating a polyhedron model instance based on the framework of the polyhedron model, so as to automatically output a parallel strategy according to the polyhedron model instance, the algorithm logic under different frameworks is modeled under the polyhedron model, and a process of automatically outputting a parallel strategy that can be efficiently executed is realized, thereby effectively improving the efficiency of searching for parallel strategies, while reducing the difficulty of distributed training development and efficiency tuning of deep learning algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a preferred embodiment of the automatic parallel strategy search method based on polyhedron modeling provided by the present invention;

[0042] Figure 2 Flowchart of step S100 in a preferred embodiment of the automatic parallel strategy search method based on polyhedron modeling provided by the present invention;

[0043] Figure 3 Flowchart of step S300 in a preferred embodiment of the automatic parallel strategy search method based on polyhedron modeling provided by the present invention;

[0044] Figure 4 Schematic diagram of node splitting and node aggregation provided by the present invention;

[0045] Figure 5 Flowchart of step S400 in a preferred embodiment of the automatic parallel strategy search method based on polyhedron modeling provided by the present invention;

[0046] Figure 6 A schematic diagram of computing graph segmentation in the data parallel mode provided by the present invention;

[0047] Figure 7 A schematic diagram of computing graph segmentation in the pipeline parallel mode provided by the present invention;

[0048] Figure 8 A functional block diagram of the automatic parallel strategy search system provided by the present invention;

[0049] Figure 9A diagram showing the relationship between the Pytorch framework and the automatic parallel strategy search system provided by the present invention;

[0050] Figure 10 A schematic diagram of the operating environment of a preferred embodiment of the controller provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0054] First of all, distributed training of deep learning models based on multiple machines and multiple graphics cards has become the most important technical solution for accelerating model training efficiency. The main distributed training modes include data parallelism, model parallelism, optimizer parallelism, pipeline parallelism, and hybrid parallelism. These parallel methods are also the main parallel functions supported by current mainstream deep learning frameworks (such as TensorFlow, Pytorch, Mindspore, and PaddlePaddle). Different AI frameworks only differ in ease of use and efficiency. However, almost all of the above parallel methods require algorithm developers to implement them by calling the framework API and manually tune the training efficiency. This is very difficult for algorithm developers who do not understand the underlying AI framework implementation mechanism and cluster communication characteristics. Complex implementation and debugging lead to a significant reduction in efficiency.

[0055] On the other hand, since the efficiency of automatic parallel strategy search is exponentially related to the AI ​​model calculation graph and cluster resource scale, the search efficiency of some existing automatic parallel search tasks is difficult to meet the efficiency requirements of large-parameter models on large-scale clusters. Therefore, it is necessary to design and implement a highly efficient multi-machine and multi-card distributed training parallel strategy search method to solve the problem of automatic parallel training strategy search for large models, allowing algorithm developers to focus only on the development of algorithm logic and quickly implement distributed training on AI clusters.

[0056] To address the above-mentioned problems in the prior art, the present invention provides an automatic parallel strategy search method and related equipment based on polyhedron modeling. The present invention converts and balances the generated model calculation graph, balances the calculation graph, and creates a polyhedron model instance based on the balanced calculation graph within the framework of the polyhedron model, so as to automatically output a parallel strategy based on the polyhedron model instance. This method realizes the process of modeling algorithm logic under different frameworks within the polyhedron model and automatically outputting a parallel strategy that can be executed efficiently, thereby effectively improving the efficiency of searching for parallel strategies and reducing the difficulty of distributed training development and efficiency tuning of deep learning algorithms.

[0057] The following describes the design of an automatic parallel strategy search method based on polyhedron modeling through specific exemplary embodiments. It should be noted that the following embodiments are only used to illustrate the technical solution of the invention and are not intended to be limiting.

[0058] See also Figure 1 The present invention provides an automatic parallel strategy search method based on polyhedron modeling, comprising:

[0059] S100. Obtain a model calculation graph of a deep learning algorithm based on a model object input by a user.

[0060] The model object refers to the stand-alone training code of the deep learning algorithm pre-defined by the user, such as the BERT model, the GPT3 model (GPT-3 was built by OpenAI, an independent AI research and deployment company, and is a large-scale natural language model currently running on Microsoft Azure), etc.

[0061] Specifically, under different deep learning model frameworks (such as TensorFlow, Pytorch, Mindspore, PaddlePaddle), the corresponding model objects (algorithm logic pre-defined by the user) are different, so it is necessary to adopt the corresponding method and obtain the model calculation graph of the deep learning algorithm based on the single-machine training code of the deep learning algorithm defined by the AI ​​framework input by the user. Among them, the model calculation graph refers to a representation of the deep learning algorithm in the AI ​​framework. At present, a directed acyclic graph (DAG) is generally used to represent the calculation process of a deep learning algorithm. The node of the calculation graph represents a calculation operation, and the edge represents the tensor data dependency between the algorithm calculation operations. The purpose of the above-mentioned process of generating the model calculation graph is to represent the calculation process of an algorithm.

[0062] Further, see Figure 2 , step 100 specifically includes:

[0063] S110, obtaining an algorithm model according to the model object input by the user;

[0064] S120, after randomly inputting a numerical value into the algorithm model, recording the calculation process of the algorithm model to obtain the model calculation graph;

[0065] S130, or using a Python interpreter to parse the model object to generate a syntax tree, and then analyzing the syntax tree to obtain the model calculation graph.

[0066] Specifically, in the present invention, taking the Pytorch framework as an example, the model calculation graph defined by the algorithm developer can be obtained through two methods: jit.trace (tracing technology) or jit.script (source code conversion technology); wherein, the jit.trace method is based on vector calculation tracing. After obtaining an algorithm model according to the model object input by the user, a numerical value is randomly input according to the input type requirement of the algorithm, and then each calculation process of the algorithm is recorded. These recorded calculation processes are constructed into a calculation graph; for example, the algorithm model inputs a 32X32 picture, and the output is the classification label of this picture, then the random input refers to the random input of a 32X32 data. The jit.script method is based on source code conversion. Its basic principle is to first parse the model object (user-defined algorithm logic) into a syntax tree (a tree structure in which a computer describes the truth of the world) through a python interpreter, and then obtain the model calculation graph by analyzing the syntax tree.

[0067] Please continue reading Figure 1 , S200, convert the model calculation graph to obtain a converted model calculation graph.

[0068] Specifically, since there are multiple ways to define computational graphs under different frameworks, that is, the computational graph representations under different frameworks are different, for example, the Pytorch framework uses the Torch computational graph (Torch Graph), so it is necessary to convert the computational graphs of different frameworks into a computational graph in a common intermediate representation form. Usually, the computational graph intermediate representation method is manually defined by experts, which is a process of manually establishing rule mapping. Therefore, the conversion of different computational graph intermediate representations is carried out based on their respective rules. In the present invention, a new computational graph representation method is defined for parallel strategy search, so it is necessary to convert the upper-level framework computational graph into the computational graph used in parallel strategy search.

[0069] Furthermore, step S200 specifically includes:

[0070] S210: Re-represent the model calculation graph using a predefined intermediate representation to obtain a converted model calculation graph.

[0071] Specifically, the computational graphs under different frameworks are re-represented using an intermediate representation pre-defined by the user, thereby converting the computational graphs under different frameworks into the computational graphs used in the parallel strategy search in the present invention. The use of the intermediate representation is a general method, and the specific definition in the C++ implementation is four class definitions; among them, the pre-defined intermediate representations are: IRType (type), IRValue (value), IRNode (node) and IRGraph (graph).

[0072] Please continue reading Figure 1 , S300, balance the converted model calculation graph to obtain a balanced calculation graph.

[0073] Specifically, since the computational load of many different node operations in the computational graph is different, the computational graph is unbalanced. In order to obtain small nodes of balanced size when splitting the nodes in subsequent steps, and to improve the efficiency of automatic search for parallel strategies, it is necessary to balance the nodes in the computational graph, that is, to perform node splitting and node aggregation operations; among them, the purpose of the computational graph balancing process is to balance the computational graph in the horizontal and vertical dimensions. The purpose of computational balancing is to reaggregate or split the nodes in the computational graph, so that the converted computational graph can achieve computational balance in two dimensions, avoiding uneven computational distribution during parallel strategy search.

[0074] Further, see Figure 3 , step S300 specifically includes:

[0075] S310: Setting an average computational cost threshold for a node, and comparing the computational cost nodes in the converted model computation graph with the average computational cost threshold;

[0076] S320: Merge adjacent nodes whose computational amount is less than the average computational threshold, and split nodes whose computational amount is greater than the average computational threshold, to obtain a balanced computation graph.

[0077] Specifically, the balancing process mainly includes aggregating or splitting the nodes in the model calculation graph; before the node aggregation or node differential operation, first, it is necessary to set an average computational amount threshold of a node, traverse the computational amount nodes in the converted model calculation graph in node order, and compare the computational amount nodes in the converted model calculation graph with the average computational amount threshold; then, the node aggregation operation is: merging adjacent computational amount nodes that are less than the average computational amount threshold; and the node splitting operation is: according to the average computational amount threshold, splitting the computational amount nodes that are greater than the average computational amount threshold into multiple nodes, usually splitting the matrix multiplication operator that is greater than the average computational amount threshold, so that the computational amount of the fused and split nodes is equivalent to the average computational amount threshold; after completing the node aggregation or node splitting operation, a balanced calculation graph can be obtained.

[0078] In the present invention, by splitting the large nodes in the calculation graph or aggregating multiple adjacent small nodes, balanced nodes are obtained, that is, the balanced calculation graph is obtained, so that the calculation amount between each node is balanced, which can effectively avoid the problem of uneven calculation distribution during parallel strategy search and improve the efficiency of node segmentation.

[0079] Among them, node aggregation is to convert the calculation process of multiple nodes into a node in terms of representation. For example, if three computing nodes are aggregated, the three nodes before aggregation are three IRNode objects, and a new IRNode object is formed after aggregation. In addition, the splitting of nodes with large computing volume needs to be determined according to the operator type of the specific node. Not all nodes can be split. For example, matrix multiplication is a node that can be split, and it is also a basic operator commonly used in AI models. The specific operation process of node splitting and node aggregation is as follows: Figure 4 As shown in Figure 1, it represents the process of splitting nodes with large computational load and aggregating adjacent nodes with small computational load.

[0080] Please continue reading Figure 1 , S400, create a polyhedron model instance according to the balanced calculation graph, and output a parallel strategy according to the polyhedron model instance.

[0081] Specifically, this process involves training and modeling under a model based on polyhedron optimization, as well as a parallel strategy search process. The training and modeling process involves initializing the model under the established polyhedron optimization model framework based on the balanced computational graph to create a polyhedron optimization model instance, i.e., modeling it into a specific object. Parallel strategy search then involves searching for a pipeline parallel strategy based on the polyhedron optimization model instance, according to the specified parallel mode (the specified parallel mode refers to the parallel mode required by the developer, such as data parallelism, model parallelism, optimizer parallelism, pipeline parallelism, hybrid parallelism, etc.). The search for a pipeline parallel strategy is an example of searching for strategy values ​​for the number of pipelines and the number of mini-batches based on the pipeline mode specified by the developer, and generating the final parallel strategy. The parallel mode is a large framework. For example, data parallelism refers to the process of distributing training data samples to multiple computing devices for distributed computing during distributed training of an AI model, while the parallel strategy refers to the distributed training partitioning strategy for a specific algorithm model during execution.

[0082] Polyhedron modeling is a common method for compiler optimization of for loops. By representing the loop in the polyhedron model space, the parallel computing optimization of the loop can be directly achieved through the mapping table of the polyhedron model to improve the efficiency of training. In this invention, the training calculation process of a deep learning model is represented as a multi-layer loop operation, and then how to represent this training process through the polyhedron model is defined. Finally, the common data parallelism, model parallelism, and pipeline parallelism are unified under the polyhedron modeling framework.

[0083] Suppose a deep learning model is represented as a computational graph D(N, E) via a directed acyclic graph (DAG), where N is the set of nodes and E is the set of edges. Therefore, the deep learning model training computational process (actually a modeling process) is represented as follows (in Python), mapping the for loop to the polyhedron optimization model:

[0084] for e in range(Epoch_num): / / Epoch_num: the number of model training rounds, that is, the round number range (domain) of the for loop;

[0085] for b in range(Batch_num): / / Batch_num: The number of sample batches for one round of model training, that is, the batch number range of the for loop;

[0086] for node in nodes: / / node: a node of the model; nodes: a set of nodes in the model calculation graph;

[0087] out=Forward(node,b) / / Forward(): represents the forward calculation of the model;

[0088] for node in reverse(nodes): / / reverse(): indicates to sort the model calculation nodes in reverse order;

[0089] grad=Backward(node, b) / / grad: represents the gradient of model parameters; Backward(): represents the reverse gradient calculation of the model;

[0090] Update(node) / / update(): represents the parameter update process of the model.

[0091] Further, see Figure 5 , step S400 specifically includes:

[0092] S410, creating a polyhedron optimization model;

[0093] S420: Initialize the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance;

[0094] S430: Output a parallel strategy based on the polyhedron model instance and the number of computing resources input by the user, wherein the parallel strategy includes data parallel segmentation dimension, pipeline parallel segmentation dimension, and pipeline arrangement.

[0095] Specifically, the balanced calculation graph is first mapped onto the polyhedron model to obtain a polyhedron optimization model, where "mapping" here can be understood as an affine transformation of the coordinate space, such as translation, rotation and other operations, and then the balanced calculation graph is input into the polyhedron optimization model to obtain a polyhedron model instance (specific object), that is, the for loop is mapped onto the polyhedron optimization model to obtain a polyhedron model instance (finding a distributed optimization calculation model), and then the data parallel splitting dimension or pipeline parallel splitting dimension of distributed training is found through geometric linear transformation, and combined with the number of computing resources input by the user, for example, the computing resources are 8 GPU cards, so the data parallelism will be split into 8 mini-batches from the data parallel splitting dimension and calculated on 8 cards respectively, and finally the parallel strategy (data parallel splitting dimension, pipeline parallel splitting dimension and pipeline arrangement, etc.) is comprehensively output. The specific data parallel splitting dimension and pipeline parallel splitting dimension splitting process are as follows. Figure 6 and Figure 7 As shown in the figure, the horizontal axis represents the number of sample batches for model training, and the vertical axis represents the number of layers of model parameters. For example, the calculation process of the BERT basic model has 24 layers. The points in the first quadrant represent the calculation process of a sample batch on a certain layer of the model. The arrows represent the data dependency or time dependency of the model training calculation process. The dotted box represents an example of model segmentation during distributed training. Figure 6 This shows an example of splitting a computational graph in data parallel mode, where the graph is split directly in the batch direction. Figure 7 This figure shows an example of computing graph segmentation in pipeline parallel mode, where segmentation is performed in a staggered horizontal direction.

[0096] The present invention provides a unified description of the training calculation process of the deep learning model and describes the calculation process into a polyhedron model. Through the polyhedron model, commonly used distributed parallel training modes such as data parallelism and pipeline parallelism can be unified through the obtained polyhedron optimization model, so that feasible parallel strategies can be searched through a simple mapping transformation of the polyhedron model.

[0097] Please continue reading Figure 1 S500: Call the underlying framework to execute the parallel strategy. The execution API is the runtime manager (runtime interface) in the underlying AI framework.

[0098] Specifically, the data parallel partitioning dimension or pipeline parallel partitioning dimension for distributed training is found under the polyhedron model, and combined with the number of computing resources input by the user, a parallel strategy is automatically output. Finally, the parallel strategy needs to be executed by calling the underlying framework execution API. The API is an interactive interface.

[0099] Furthermore, step S500 specifically includes:

[0100] S510: Calling an execution API of the underlying framework to execute the parallel strategy, wherein the execution API is a run manager in the underlying AI framework.

[0101] Specifically, after creating a polyhedron model instance under the polyhedron model, the data parallel segmentation dimension or pipeline parallel segmentation dimension of distributed training is found through the polyhedron model instance, and combined with the number of computing resources input by the user, the distributed training execution strategy of the algorithm model is automatically output. Then, the underlying framework execution API needs to be called to implement the execution of the distributed training strategy of the algorithm model.

[0102] For further information, see Figure 8 Based on the above-mentioned automatic parallel strategy search method based on polyhedron model modeling, the present invention also provides an automatic parallel strategy search system accordingly, and the automatic parallel strategy search system also includes: a calculation graph generation module 100, a calculation graph conversion module 200, a calculation graph balancing module 300, a parallel strategy search module 400 and a parallel strategy execution module 500.

[0103] Specifically, the calculation graph generation module 100 is used to obtain a model calculation graph of a deep learning algorithm based on a model object input by a user; the calculation graph conversion module 200 is used to convert the model calculation graph to obtain a converted model calculation graph; the calculation graph balancing module 300 is used to balance the converted model calculation graph to obtain a balanced calculation graph; the parallel strategy search module 400 is used to create a polyhedron model instance based on the balanced calculation graph, and output a parallel strategy based on the polyhedron model instance; the parallel strategy execution module 500 is used to call the underlying framework to execute the parallel strategy.

[0104] The implementation of the automatic parallel strategy search method based on polyhedron modeling proposed in this invention can be regarded as a middleware that supports the AI ​​framework to realize the automatic parallel function. Figure 9 The diagram shows the relationship between its functions and the Pytorch framework. First, users use the Pytorch front-end API to define the algorithm. In fact, users do not need to consider the parallel segmentation of the algorithm, but only need to consider the single-machine implementation logic of the algorithm; therefore, users only need to pay attention to the API usage method provided by the Pytorch framework for the algorithm, and implement the single-machine algorithm through the API, such as Figure 9 As shown, the vertical row on the left is the basic architecture of the Pytorch framework, and the right part is the automatic parallel strategy search system. The main interface between the automatic parallel strategy search system and the Pytorch framework is the algorithm's computational graph representation and the underlying framework runtime interface. Users are almost unaware of the automatic parallel strategy search system.

[0105] The user-defined algorithm then outputs a computational graph through Pytorch's JIT (Just-in-Time Compilation) module. The computational graph is then fed into the automatic parallel strategy search system and subsequently fed into the computational graph generation module, computational graph conversion module, computational graph balancing module, and parallel strategy search module. Finally, the computational subgraph segmented according to the parallel strategy is output and executed by calling Pytorch's underlying Runtime API (the internal functional API of the Pytorch framework).

[0106] Furthermore, the present invention also provides a controller, which includes a processor 10 , a memory 20 and a display 30 . Figure 10 Only some of the components of the controller are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may alternatively be implemented.

[0107] In some embodiments, the memory 20 may be an internal storage unit of the controller, such as a hard disk or memory of the controller. In other embodiments, the memory 20 may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the memory 20 may also include both an internal storage unit of the controller and an external storage device. The memory 20 is used to store application software and various types of data installed on the controller. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an automatic parallel strategy search program 40 is stored on the memory 20, and the automatic parallel strategy search program 40 can be executed by the processor 10, thereby realizing the automatic parallel strategy search method based on polyhedron model modeling in the present invention.

[0108] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the automatic parallel strategy search method based on polyhedron modeling.

[0109] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the device and to display a visual user interface. The components 10-30 of the device communicate with each other via a system bus.

[0110] In one embodiment, when the processor 10 executes the automatic parallel strategy search program 40 in the memory 20, the following steps are implemented:

[0111] Obtain the model calculation graph of the deep learning algorithm based on the model object input by the user;

[0112] Converting the model calculation graph to obtain a converted model calculation graph;

[0113] Performing equalization processing on the converted model calculation graph to obtain a balanced calculation graph;

[0114] Creating a polyhedron model instance according to the balanced calculation graph, and outputting a parallel strategy according to the polyhedron model instance;

[0115] The underlying framework is called to execute the parallel strategy.

[0116] The step of obtaining a model calculation graph of a deep learning algorithm based on a model object input by a user specifically includes:

[0117] After randomly inputting a numerical value into the algorithm model, the calculation process of the algorithm model is recorded to obtain the model calculation graph; wherein, the model object refers to the stand-alone training code of the deep learning algorithm pre-defined by the user.

[0118] Alternatively, the model object is parsed by a Python interpreter to generate a syntax tree, and then the syntax tree is analyzed to obtain the model calculation graph.

[0119] The step of converting the model calculation graph to obtain the converted model calculation graph specifically includes:

[0120] The model calculation graph is re-expressed using a pre-defined intermediate representation to obtain a converted model calculation graph.

[0121] The step of performing equalization processing on the converted model calculation graph to obtain the equalized calculation graph specifically includes:

[0122] A balanced calculation graph is obtained by splitting the large nodes in the converted model calculation graph and aggregating the adjacent small nodes in the converted model calculation graph.

[0123] The steps of creating a polyhedron model instance according to the balanced calculation graph and outputting a parallel strategy according to the polyhedron model instance specifically include:

[0124] Creating a polyhedron optimization model; initializing the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance;

[0125] Output a parallel strategy according to the polyhedron model instance and the number of computing resources input by the user.

[0126] The step of calling the underlying framework to execute the parallel strategy specifically includes:

[0127] The execution API of the underlying framework is called to execute the parallel strategy.

[0128] Furthermore, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an automatic parallel strategy search program 40, and when the automatic parallel strategy search program 40 is executed by the processor, it implements the steps of the automatic parallel strategy search method based on polyhedron model modeling as described above; since the steps of the automatic parallel strategy search method based on polyhedron model modeling have been described in detail above, they will not be repeated here.

[0129] In summary, the present invention provides an automatic parallel strategy search method based on polyhedron model modeling and related equipment. The automatic parallel strategy search method based on polyhedron model modeling includes: obtaining a model calculation graph of a deep learning algorithm according to a model object input by a user; converting the model calculation graph to obtain a converted model calculation graph; balancing the converted model calculation graph to obtain a balanced calculation graph; creating a polyhedron model instance according to the balanced calculation graph, and outputting a parallel strategy according to the polyhedron model instance; calling the underlying framework to execute the parallel strategy. In the present invention, by converting and balancing the model calculation graph, and automatically outputting a parallel strategy after creating a polyhedron model instance under the framework of the polyhedron model, it is realized that different algorithm logics are modeled under the polyhedron model, and the parallel strategy process is automatically output, thereby improving the efficiency of parallel strategy search and reducing the difficulty of distributed training development and efficiency tuning of deep learning algorithms.

[0130] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An automatic parallel strategy search method based on polyhedron modeling, characterized in that: The automatic parallel strategy search method based on polyhedron modeling includes: Obtain the model calculation graph of the deep learning algorithm based on the model object input by the user; Converting the model calculation graph to obtain a converted model calculation graph; Perform node splitting and node aggregation operations on the nodes of the converted model calculation graph to obtain a balanced calculation graph; Creating a polyhedron model instance according to the balanced calculation graph, and outputting a parallel strategy according to the polyhedron model instance; Calling the underlying framework to execute the parallel strategy; The step of obtaining a model calculation graph of a deep learning algorithm based on a model object input by a user specifically includes: Get an algorithm model based on the model object input by the user; After randomly inputting a numerical value into the algorithm model, the calculation process of the algorithm model is recorded to obtain the model calculation graph; or parsing the model object through a Python interpreter to generate a syntax tree, and then analyzing the syntax tree to obtain the model calculation graph; The step of converting the model calculation graph to obtain a converted model calculation graph specifically includes: Re-expressing the model calculation graph using a pre-defined intermediate representation to obtain a converted model calculation graph; The step of performing node splitting and node aggregation operations on the nodes of the converted model calculation graph to obtain a balanced calculation graph specifically includes: Setting an average computation amount threshold for a node, and comparing the computation amount nodes in the converted model computation graph with the average computation amount threshold; Adjacent computation amount nodes with a computation amount less than the average computation amount threshold are merged, and computation amount nodes with a computation amount greater than the average computation amount threshold are split to obtain a balanced computation graph.

2. The automatic parallel strategy search method based on polyhedron modeling according to claim 1, characterized in that: The step of creating a polyhedron model instance according to the balanced calculation graph and outputting a parallel strategy according to the polyhedron model instance specifically includes: Create a polyhedron optimization model; Initialize the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance; Output a parallel strategy according to the polyhedron model instance and the number of computing resources input by the user.

3. The automatic parallel strategy search method based on polyhedron modeling according to claim 1, characterized in that: The step of calling the underlying framework to execute the parallel strategy specifically includes: The execution API of the underlying framework is called to execute the parallel strategy.

4. The automatic parallel strategy search method based on polyhedron modeling according to claim 1, characterized in that: The model object refers to the stand-alone training code of the deep learning algorithm pre-defined by the user.

5. The automatic parallel strategy search method based on polyhedron modeling according to claim 1, characterized in that: The predefined intermediate representations include: IRType, IRValue, IRNode and IRGraph.

6. The automatic parallel strategy search method based on polyhedron modeling according to claim 2, characterized in that: The parallel strategy includes a data parallel segmentation dimension and a pipeline parallel segmentation dimension.

7. The automatic parallel strategy search method based on polyhedron modeling according to claim 3, characterized in that: The execution API is the run manager in the underlying AI framework.

8. An automatic parallel strategy search system, characterized in that: The automatic parallel strategy search system is applied to the automatic parallel strategy search method based on polyhedron modeling according to any one of claims 1 to 7, and the automatic parallel strategy search system includes: The computational graph generation module is used to obtain the model computational graph of the deep learning algorithm based on the model object input by the user; The step of obtaining a model calculation graph of a deep learning algorithm based on a model object input by a user specifically includes: Get an algorithm model based on the model object input by the user; After randomly inputting a numerical value into the algorithm model, the calculation process of the algorithm model is recorded to obtain the model calculation graph; or parsing the model object through a Python interpreter to generate a syntax tree, and then analyzing the syntax tree to obtain the model calculation graph; A calculation graph conversion module, used to convert the model calculation graph to obtain a converted model calculation graph; The computation graph balancing module is used to split and aggregate the nodes of the converted model computation graph to obtain a balanced computation graph. A parallel strategy search module, configured to create a polyhedron model instance according to the balanced calculation graph, and output a parallel strategy according to the polyhedron model instance; The parallel strategy execution module is used to call the underlying framework to execute the parallel strategy.

9. A controller, characterized in that: The controller includes: a memory, a processor, and an automatic parallel strategy search program based on polyhedron model modeling stored in the memory and runnable on the processor. When the automatic parallel strategy search program based on polyhedron model modeling is executed by the processor, the steps of the automatic parallel strategy search method based on polyhedron model modeling are implemented as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an automatic parallel strategy search program based on polyhedron model modeling, and when the automatic parallel strategy search program based on polyhedron model modeling is executed by a processor, the steps of the automatic parallel strategy search method based on polyhedron model modeling according to any one of claims 1 to 6 are implemented.

Images