Cross-domain parallel strategy generation technology based on constraint guidance and redundancy calculation

Abstract

The invention discloses a cross-domain parallel strategy generation technology based on constraint guidance and redundancy calculation, and aims to research how a cross-domain distributed calculation framework dynamically generates a cloud edge end cross-domain training parallel strategy. The method comprises the following steps: firstly, researching a runtime parallel plan regeneration mechanism, fitting cloud edge end environment change to reduce a search space by constructing a constraint model and a performance model, and applying an optimized graph matching algorithm to obtain final runtime matching; and secondly, researching a parallel execution mechanism of multiple parallel plans during operation, determining a specific switching plan by constructing a video memory constraint and parallel coverage model, obtaining a new Micro-Batch parallel plan, and completing switching and execution of the related parallel plans. Through the mechanism, the parallel strategy generation is dynamically and quickly generated according to the dynamic change of the environment, the strategy search time is shortened, the switching influence is reduced, and the dynamic nature of the cloud edge end computing environment is effectively dealt with.

Description

Technical Field

[0001] The present invention relates to the computer field, and mainly relates to a cross-domain parallel strategy generation technology based on constraint guidance and redundant calculation. Background Art

[0002] In the training process of deep learning models, as the scale of the model increases, a single node gradually cannot accommodate all model parameters. A good parallel strategy and model placement method are needed to run the training tasks of deep learning models on multiple nodes. In the cloud-edge computing environment, the connection relationship between each computing node and the dynamic nature of computing power create obstacles for generating and adjusting parallel strategies. Therefore, how to dynamically adjust the cloud-edge cross-domain training parallel strategy has become an important issue in building a cross-domain distributed computing framework. Currently, researchers have proposed many methods for finding or approaching the optimal parallel strategy. The main strategies are based on classical algorithms (such as recursive algorithms, dynamic programming algorithms, integer linear programming algorithms, breadth-first search algorithms, etc.) and machine learning methods (such as Monte Carlo methods, Markov chain methods, reinforcement learning algorithms, etc.). The following is a review of these two aspects.

[0003] 1. Classical method

[0004] Li et al. proposed a layer-by-layer parallel strategy OptCNN. OptCNN constructs a computational graph and a device graph to form an evaluation model to estimate the cost under different tensor parallel partitions, and executes a dynamic graph search algorithm to obtain a tensor parallel strategy for a linear CNN model. Wang et al. proposed a technology called ToFu that can search for tensor parallel strategies for nonlinear models. ToFu coarsens the computational operations in the computational graph, that is, it merges nonlinear nodes with other nodes to make them linear, and then uses a dynamic programming algorithm to generate tensor parallel strategies, and considers communication costs to narrow the search space, which shortens the search time by a hundred times. Cai et al. proposed an extended OptCNN algorithm TensorOpt. TensorOpt extends the dynamic search algorithm in OptCNN so that the algorithm can be executed in a nonlinear computational graph. In addition, TensorOpt groups operators and uses multi-threaded parallelization to generate different tensor parallel strategies. Narayanan et al. proposed PipeDream, an automatic parallel solution that supports asynchronous pipeline training. This technology uniquely analyzes the model, estimates the execution time, activation value size and model parameter size of each model layer, and applies a dynamic programming algorithm through a cost model to achieve load balancing of pipeline partitioning. In addition, Narayanan et al. also expanded PipeDream-2BW based on PipeDream, reducing memory consumption by applying activation value recalculation. Fiddle researchers extended PipeDream and proposed DNN-Partitioning to support arbitrary computational graphs. Tarnawski et al. proposed Piper, a method that can perform tensor parallelism and pipeline parallelism at the same time. Piper uses a two-level dynamic programming algorithm to generate strategies. First, an external dynamic programming algorithm is used to generate hundreds of NP-hard knapsack subproblems, and then these problems are formed into subgraphs, and the Band-Per-Buck heuristic algorithm is used to accelerate the search of subgraphs to generate parallel strategies. In addition, there are methods such as D-Rec and PaSE based on static symbolic analysis to generate parallel strategies. These methods have fast generation speeds, but cannot capture information such as cache and computation-communication overlap, and the automatic generation of parallel strategies is poor. Song et al. proposed AccPar, which can perform data parallelism and 2D tensor parallelism. AccPar takes into account the communication cost between each working node and introduces partition ratio to support heterogeneous clusters. Alpa formalizes the partitioning of Intra-P as an integer programming problem and Inter-P as a dynamic programming problem, and comprehensively generates parallel strategies. Alpa also supports tensor parallelism, pipeline parallelism, and data parallelism, and can work in any computational graph. Lai et al. proposed Merak, an automatic 3D parallel method designed for basic large models.Merak uses graph sharding algorithms and model graph algorithms based on proxy nodes to build an automatic model partitioner, and uses automatic model parallelism to deploy these models on different proxy nodes. Lin et al. proposed a constraint-guided parallel strategy generator nnScaler. nnScaler proposed a new search space consisting of three primitives, which can capture model conversion models and time series, and convert parallelization plans into spatial scheduling of models, which greatly accelerates the generation of parallel plans and computational efficiency.

[0005] 2. Machine Learning Methods

[0006] Placeto and GDP use graph neural networks to model computational graphs. It is worth noting that Placeto's model can only generate the layout of one operator each time it runs; while GDP is an end-to-end model specifically for Transformer, which can directly generate the layout for the Transformer backbone network. Paliwal et al. proposed REGAL, a computational graph optimization algorithm using a reinforced genetic algorithm. REGAL uses a GNN strategy to predict multiple node-specific rough parallel schemes, and then runs these strategies using a biased random key genetic algorithm to find the best solution within a certain iteration. Liu et al. proposed HeterPS to generate parallel strategies on heterogeneous devices. HeterPS uses an LSTM model to predict which heterogeneous device the model layer should run on, providing higher throughput and resource utilization than homogeneous devices. Wang et al. proposed Auto-MAP, which uses a deep Q network to search for hybrid parallel strategies. Auto-MAP sets rewards, states, and operations for all data parallelism, tensor parallelism, and pipeline parallelism to indicate the DQN search strategy, and combines the topology of the device and network to generate a parallel strategy with a higher running speed. Zhang et al. proposed the TAG system to accelerate DNN training. TAG uses the DNN computation graph and device topology graph as the input of the graph neural network, and combines the graph neural network with a search-based method to quickly identify optimized distributed training strategies.

[0007] The existing automatic search and adjustment of parallel strategies are mainly achieved through classical search methods and machine learning methods, which realize the generation and adjustment of model distributed training plans under large-scale models and complex topologies, and obtain more efficient parallel plans than manual ones. In the future, the automatic search and generation of parallel strategies are expected to focus on the search strategy search acceleration and how to generate model distributed training parallel plans under complex conditions. In the cloud-edge computing environment, there is currently a lack of related work on parallel strategy search and adjustment, but there is an urgent need for collaborative model training in the cloud-edge computing environment, and this research direction still needs to be further developed and explored.

[0008] In summary, for dynamic parallel strategies in cloud-edge computing environments, there are two main types of work: research on parallel strategies themselves and automatic search algorithms for parallel strategies. However, existing parallel strategies are not designed for cloud-edge computing environments, and the search efficiency of automatic search algorithms is low, making it difficult to adapt to the dynamic cloud-edge environment. Specifically, parallel strategies need to be designed in combination with specific computing and communication environments. Most existing parallel strategies are designed for centralized training in cloud computing centers, and lack structural support for the dynamic nature of cloud-edge computing environments. In terms of dynamic search strategies, the search methods of existing classical algorithms take too long and the search results are unstable. The stability and effect of machine learning methods are poor, and it is difficult to respond to dynamic changes in the environment. These problems make it difficult for existing parallel strategies to exert their optimization capabilities, limiting their effectiveness in cloud-edge computing environments. Therefore, it is extremely important to study a parallel strategy suitable for cloud-edge computing environments. How to dynamically generate cloud-edge cross-domain training parallel strategies has become an urgent problem to be solved.

[0009] In summary, around the distributed computing framework, the industry has conducted a lot of research and exploration in heterogeneous hardware training, communication optimization strategies, dynamic parallel strategies, etc. However, for the new features of cloud-edge-end cross-domain intelligent computing hardware heterogeneity, high communication cost between cloud-edge-end cross-domain nodes, and strong dynamics of cloud-edge-end cross-domain parallel training environment presented in the cross-domain distributed computing scenario of cloud-edge-end collaboration, existing methods are still difficult to effectively deal with, and there is an urgent need to study the cross-domain distributed training framework and its key technologies for cloud-edge-end collaboration. Summary of the invention

[0010] In response to the above problems, the present invention proposes a cross-domain parallel strategy generation technology based on constraint guidance and redundant computing. First, through the constraint-guided runtime parallel strategy adjustment technology, migration cost estimation, cloud-edge environment fitting and rapid parallel strategy regeneration are realized. According to the changes in the environment, a parallel strategy suitable for the current environment can be generated to solve the challenge that fixed hybrid parallel strategies are difficult to adapt to dynamic environments; secondly, through the runtime multi-parallel strategy parallel execution mechanism, seamless switching between new and old parallel strategies is achieved, and the calculation pause time of parallel strategy switching in the cloud-edge environment is reduced to meet the challenges of dynamically applying parallel strategies.

[0011] Specifically, the present invention consists of the following steps:

[0012] Step (1): Build a training task transfer model.

[0013] Step (2): Run the GRU network to evaluate the communication status.

[0014] Step (3): Run the graph partitioning algorithm.

[0015] Step (4): Construct a parallelism evaluation model for the parallel plan.

[0016] Step (5): Execute the switch between the old and new batches.

[0017] Steps (1), (2), and (3) constitute a larger working component: the runtime parallel plan regeneration mechanism based on two original graph partitioning is mainly. First, the migration cost estimation mechanism is introduced. The present invention models the migration cost between computing nodes, comprehensively considers the weight migration cost and data migration cost, and passes the generated migration cost to the graph partitioning algorithm for plan generation; secondly, the environment is simulated and modeled through GRU to identify the dynamic changes of computing nodes in the cloud-edge environment. When the changes of computing nodes are identified, the graph partitioning parallel plan generation algorithm is triggered; finally, when the graph partitioning parallel plan generation algorithm is triggered, the computational graph is constructed according to the migration cost, and the binary graph partitioning algorithm is run to obtain a new parallel plan.

[0018] Step (1) The present invention constructs a training task migration model, considering two aspects: weight migration cost and data migration cost:

[0019]

[0020] This method uses d to represent the migration cost of data and p to represent the migration cost of weights, and comprehensively evaluates the cost of task migration between nodes.

[0021] Step (2) The present invention uses GRU to evaluate the status of computing nodes in the cloud edge environment and predict the status of cloud edge computing nodes through network communication conditions. Specifically, the calculation process of GRU is as follows:

[0022] r t =σ(W ir x t +b ir +W hr h (t-1) +b hr )

[0023] z t =σ(W iz x t +b iz +W hz h (t-1) +b hz )

[0024] n t =tanh(W in x t +b in +r t ⊙(W hn h(t-1) +b hn ))

[0025] h t =(1-z t )⊙n t +z t ⊙h (t-1)

[0026] In step (3), when the environment fitting algorithm finds fluctuations in cloud edge nodes, the graph partitioning algorithm is triggered. The migration computation graph of the task is constructed based on the running results of the migration cost estimation. A binary connection graph is constructed between the migration task and the computing node, and the migration strategy shown in the figure is obtained through the minimum graph matching and partitioning algorithm. A new parallel strategy is formulated to cope with the dynamic changes in the cloud edge computing environment.

[0027] Step (4) and step (5) together constitute a parallel execution mechanism of parallel plans based on redundant execution. First, the invention defines the coverage of concurrent execution of two parallel plans, so as to execute the new parallel plan as much as possible while minimizing the micro-batches of the existing parallel strategy; secondly, the main process of parallel execution of the parallel strategy is introduced, and how to transition from the old parallel plan to the new parallel plan is explained.

[0028] Step (4) First, the present invention constructs a parallelism evaluation model for two parallel plans to execute the new parallel plan as much as possible while minimizing the micro-batch of the existing parallel strategy:

[0029]

[0030] Among them, this method uses C to represent the execution time of the parallel plan, and uses L1 and L2 to represent the number of micro-batches of the old and new parallel strategies. The most suitable concurrent execution plan is obtained by searching the values ​​of L1 and L2. At the same time, this method searches the optimization space with the video memory capacity and the coverage of the old and new parallel plans as constraints. First, the coverage constraint is defined:

[0031]

[0032] Through the above formula, this method defines the coverage of the new and old parallel plans, that is, to ensure that the execution time of the old parallel plan completely covers the execution time of the new parallel plan. At the same time, this method defines the video memory constraint:

[0033]

[0034] Through the above formula, the present invention defines existing constraints to ensure that the capacity of the video memory of the cloud-edge computing device is not exceeded when parallel plans are executed concurrently.

[0035] Step (5) After the search and scheduling of the micro-batch plan is completed, the switch between the old and new batches will be executed. The present invention will first execute the old parallel plan. When the computing node receives a parallel plan switching request, the old and new parallel plans will be executed in parallel at the same time, and the old and new parallel plans will be ensured to be completed almost at the same time. AllReduce will be used to synchronize the weight information of the old parallel plan to each node. After the synchronization of the weight information is completed, the new parallel plan will be executed.

[0036] The main advantages of the present invention are:

[0037] By proposing constraint-guided parallel strategy search and redundant computing-based parallel strategy migration, a cloud-edge cross-domain parallel strategy search module is established to dynamically generate cloud-edge cross-domain training parallel strategies. This method achieves dynamic generation and adjustment of parallel strategies by using constraint guidance to narrow the search space and redundant computing to achieve smooth migration. Compared with existing methods, it has the advantages of fast strategy generation and smooth parallel plan migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the overall technical solution for cross-domain parallel strategy generation based on constraint guidance and redundant computing proposed by the present invention.

[0039] Figure 2 This is the runtime parallel plan regeneration mechanism proposed by the present invention.

[0040] Figure 3 This is the runtime parallel plan concurrent redundant execution mechanism proposed by the present invention. DETAILED DESCRIPTION

[0041] Combined with the attached figure ( Figure 1-Figure 3 ) The present invention is further described in detail.

[0042] In order to solve the problem of how the cross-domain distributed computing framework can dynamically generate cloud-edge cross-domain training parallel strategies, this paper studies the cross-domain parallel strategy generation technology based on constraint guidance and redundant computing. First, the runtime parallel plan regeneration mechanism is studied. By building constraint models and performance models, the search space is narrowed down according to the changes in the cloud-edge environment, and the optimized graph matching algorithm is applied to obtain the final runtime match. Secondly, the runtime multi-parallel plan parallel execution mechanism is studied. By building video memory constraints and parallel coverage models, the specific switching plan is determined, the new Micro-Batch parallel plan is obtained, and the switching and execution of related parallel plans are completed. Through the above mechanism, according to the dynamic changes of the environment, the parallel strategy generation is generated dynamically and quickly, the strategy search time and switching impact are shortened, and the dynamic nature of the cloud-edge computing environment is effectively responded to.

[0043] Figure 1The overall scheme of cross-domain parallel strategy generation technology based on constraint guidance and redundant computing is shown, including steps (1) to (5). First, the runtime parallel plan regeneration mechanism is studied. By building constraint models and performance models, the search space is narrowed down according to the changes in the cloud-edge environment, and the optimized graph matching algorithm is applied to obtain the final runtime match; secondly, the runtime multi-parallel plan parallel execution mechanism is studied. By building video memory constraints and parallel coverage models, the specific switching plan is determined, the new Micro-Batch parallel plan is obtained, and the switching and execution of related parallel plans are completed. Through the above mechanism, according to the dynamic changes of the environment, the parallel strategy generation is generated dynamically and quickly, the strategy search time and switching impact are shortened, and the dynamic nature of the cloud-edge computing environment is effectively responded to.

[0044] Figure 2 This paper demonstrates the runtime parallel plan regeneration mechanism based on binary graph partitioning, which includes steps (1) to (3). First, the migration cost estimation mechanism is introduced. This method models the migration cost between computing nodes, comprehensively considers the weight migration cost and data migration cost, and passes the generated migration cost to the graph partitioning algorithm for plan generation; secondly, the environment is fitted and modeled through GRU to identify the dynamic changes of computing nodes in the cloud-edge environment. When the changes of computing nodes are identified, the graph partitioning parallel plan generation algorithm is triggered; finally, when the graph partitioning parallel plan generation algorithm is triggered, the computational graph is constructed according to the migration cost, and the binary graph partitioning algorithm is run to obtain a new parallel plan. Figure 2 In the figure, green squares represent migration tasks and purple circles represent computing nodes. Based on the migration cost, a binary connection graph is constructed between the migration tasks and computing nodes. The migration strategy shown in the figure is obtained through the minimum graph matching and partitioning algorithm. A new parallel strategy is formulated to cope with the dynamic changes of the cloud-edge computing environment.

[0045] Figure 2 It shows that after the search and scheduling of the micro-batch plan is completed, the switch between the old and new batches will be executed, including step (5). This method will first execute the old parallel plan. When the computing node receives the parallel plan switch request, it will execute the old and new parallel plans in parallel and ensure that the old and new parallel plans are completed almost at the same time. AllReduce is used to synchronize the weight information of the old parallel plan to each node. After the synchronization of the weight information is completed, the new parallel plan is started.

[0046] The present invention proposes a cross-domain parallel strategy generation technology based on constraint guidance and redundant computing. First, through the constraint-guided runtime parallel strategy adjustment technology, migration cost estimation, cloud-edge environment fitting and rapid parallel strategy regeneration are realized. According to the changes in the environment, a parallel strategy suitable for the current environment can be generated to solve the challenge that fixed hybrid parallel strategies are difficult to adapt to dynamic environments; secondly, through the runtime multi-parallel strategy parallel execution mechanism, seamless switching between new and old parallel strategies is achieved, and the computing pause time of parallel strategy switching in the cloud-edge environment is reduced to meet the challenges of dynamically applying parallel strategies.

[0047] Specifically, the present invention consists of the following steps:

[0048] Step (1): Build a training task transfer model.

[0049] Step (2): Run the GRU network to evaluate the communication status.

[0050] Step (3): Run the graph partitioning algorithm.

[0051] Step (4): Construct a parallelism evaluation model for the parallel plan.

[0052] Step (5): Switch between the old and new batches.

[0053] Step (1) and step (3) need to be protected.

[0054] Step (1) The present invention constructs a training task migration model, considering two aspects: weight migration cost and data migration cost:

[0055]

[0056] This method uses d to represent the migration cost of data and p to represent the migration cost of weights, and comprehensively evaluates the cost of task migration between nodes.

[0057] In step (3), when the environment fitting algorithm finds fluctuations in cloud edge nodes, the graph partitioning algorithm is triggered. The migration computation graph of the task is constructed based on the running results of the migration cost estimation. A binary connection graph is constructed between the migration task and the computing node, and the migration strategy shown in the figure is obtained through the minimum graph matching and partitioning algorithm. A new parallel strategy is formulated to cope with the dynamic changes in the cloud edge computing environment.

[0058] Steps (4) and (5) require protection.

[0059] Step (4) First, the present invention constructs a parallelism evaluation model for two parallel plans to execute the new parallel plan as much as possible while minimizing the micro-batch of the existing parallel strategy:

[0060]

[0061] Among them, this method uses C to represent the execution time of the parallel plan, and uses L1 and L2 to represent the number of micro-batches of the old and new parallel strategies. The most suitable concurrent execution plan is obtained by searching the values ​​of L1 and L2. At the same time, this method searches the optimization space with the video memory capacity and the coverage of the old and new parallel plans as constraints. First, the coverage constraint is defined:

[0062]

[0063] Through the above formula, this method defines the coverage of the new and old parallel plans, that is, to ensure that the execution time of the old parallel plan completely covers the execution time of the new parallel plan. At the same time, this method defines the video memory constraint:

[0064]

[0065] Through the above formula, the present invention defines existing constraints to ensure that the capacity of the video memory of the cloud-edge computing device is not exceeded when parallel plans are executed concurrently.

[0066] Step (5) After the search and scheduling of the micro-batch plan is completed, the switch between the old and new batches will be executed. The present invention will first execute the old parallel plan. When the computing node receives a parallel plan switching request, the old and new parallel plans will be executed in parallel at the same time, and the old and new parallel plans will be ensured to be completed almost at the same time. AllReduce will be used to synchronize the weight information of the old parallel plan to each node. After the synchronization of the weight information is completed, the new parallel plan will be executed.

Claims

1. A cross-domain parallel strategy generation technology based on constraint guidance and redundant computing, characterized in that: This includes constraint-guided runtime parallel strategy adjustment technology to achieve migration cost estimation, cloud-edge environment fitting, and rapid parallel strategy regeneration. It generates parallel strategies suitable for the current environment based on environmental changes to solve the problem that fixed hybrid parallel strategies are difficult to adapt to dynamic environments. It also implements a runtime multi-parallel strategy parallel execution mechanism to achieve seamless switching between old and new parallel strategies, reduce the computing pause time of parallel strategy switching in cloud-edge environments, and dynamically apply parallel strategies. The following steps are involved: Step (1): Build a training task transfer model; Step (2): Run the GRU network to evaluate the communication status; Step (3): Run the graph partitioning algorithm; Step (4): construct a parallelism evaluation model for parallel planning; Step (5): Execute the switch between the old and new batches.

2. The cross-domain parallel strategy generation technology according to claim 1 is characterized in that: Step (1) includes building a training task migration model, considering the weight migration cost and data migration cost: Use d to represent the migration cost of data, p to represent the migration cost of weight, and comprehensively evaluate the cost of task migration between nodes; Step (3) includes triggering the graph partitioning algorithm when the environment fitting algorithm discovers fluctuations in the cloud edge nodes, and constructing the migration calculation graph of the task through the running results of the migration cost estimation; constructing a binary connection graph between the migration tasks and the computing nodes, and obtaining the migration strategy shown in the figure through the minimum graph matching and partitioning algorithm, and formulating a parallel strategy to cope with the dynamic changes in the cloud edge computing environment.

3. The cross-domain parallel strategy generation technology according to claim 1 is characterized in that: Step (4) includes constructing a parallelism evaluation model for the two parallel plans to execute the new parallel plan as much as possible while minimizing the micro-batches of the existing parallel strategy: Among them, C is used to represent the execution time of the parallel plan, L1 and L2 are used to represent the number of micro-batches of the old and new parallel strategies, and the most suitable concurrent execution plan is obtained by searching the values ​​of L1 and L2. At the same time, the optimization space is searched with the video memory capacity and the coverage of the old and new parallel plans as constraints. The coverage constraint is defined as follows: The above formula is used to define the coverage of the new and old parallel plans, that is, to ensure that the execution time of the old parallel plan completely covers the execution time of the new parallel plan. At the same time, this method defines the video memory constraint as follows: The above formula is used to define the existing constraints to ensure that the concurrent execution of parallel plans does not exceed the capacity of the video memory of the cloud-edge computing device; Step (5) includes switching between the old and new batches after completing the search and scheduling of the micro-batch plan, and executing the old parallel plan. When the computing node receives a parallel plan switching request, it executes the old and new parallel plans in parallel and ensures that the old and new parallel plans are completed almost at the same time. It also uses AllReduce to synchronize the weight information of the old parallel plan to each node. After completing the synchronization of the weight information, it starts executing the new parallel plan.

Images