Distributed training task allocation method based on communication requirement and network resource matching

Abstract

The application provides a distributed training task allocation search method based on communication demand and network resource matching, which is divided into parallel strategy search and task placement search. The parallel strategy search provides a method of superimposing multiplication steps and then superimposing addition steps to quickly search for parallel configuration. The task placement search is based on distributed training communication behavior modeling, generates a point-to-point traffic matrix for distributed training, and matches it with network resources. Efficient task allocation is achieved, the parallel strategy search and task placement problem on the training cluster are solved, and the deployment scheme of parallel training can be effectively given.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, distributed training, and computer network task scheduling, and particularly to a method, apparatus, storage medium, and client for distributed training task allocation based on communication requirements and network resource matching. Background Technology

[0002] In recent years, large-scale deep neural networks have been widely used in many fields, and the number of model parameters and training datasets has grown rapidly. Currently, distributed training has become the mainstream method for training large models. Common distributed training methods mainly include data parallelism, model parallelism, pipelined parallelism, and hybrid parallelism using multiple dimensions. The distributed training process involves a large amount of data transmission of models, training sets, and gradients, and the performance of the training network has become one of the key factors restricting the improvement of distributed training performance.

[0003] The topology, network bandwidth, and communication algorithms of a training cluster all affect communication performance, and inappropriate allocation of training tasks can lead to severe network congestion. Currently, large-scale deep learning training often involves thousands or even more nodes, and numerous parallelization strategies are available. However, the network bandwidth and latency vary significantly between different training chips and different training nodes within a large-scale training cluster, making it difficult to achieve ideal training performance using commonly used experience-based task deployment methods. Therefore, determining a suitable distributed training task deployment strategy has become a key issue in solving the current challenges of training large models.

[0004] Initially, task deployment strategies were primarily based on practical deployment experience with training clusters. For example, weight parameters with less data parallel communication were chosen for convolutional layers, and activation functions with less model parallel communication were chosen for fully connected layers. With the increase in the number of training nodes and parallel methods, as well as the rise in model complexity, more automated parallel deployment methods have gradually emerged. These include: fusing data parallelism and pipelined parallelism, partitioning pipelines based on minimum communication volume, and simultaneously using data parallelism to achieve load balancing; employing dynamic programming to sort nodes based on network bandwidth to allocate stages, achieving pipeline acceleration; layering accelerator arrays to minimize the total communication volume of DNN training, and achieving linear time complexity for the search by limiting each layer to only one parallel method and using layered dynamic programming; using integer linear programming to search for the maximum acceleration that model parallelism can provide for a specific model; and fusing pipelined parallelism and data parallelism to design a parallel strategy planner to solve partitioning and layout problems. Another approach involves studying the dimensions, operations, attributes, and parameters of samples to expand the parallelism to four dimensions, making the search space more comprehensive. For example, hierarchical parallelism allows each layer in the network to use a separate parallelization strategy, and a series of search space pruning strategies are proposed to find the optimal parallelization strategy for each layer by solving the graph search problem. Markov chain Monte Carlo methods are used to search for the optimal parallelism strategy in a larger search space to accelerate training. These algorithms can explore the optimal solution achievable through hybrid parallelism in a large search space, finding a better approach than empirical layout methods. However, their drawback is high time complexity, which hinders scaling to large-scale systems.

[0005] Recent research also utilizes reinforcement learning to reduce the complexity of parallel task allocation in heterogeneous computing resources. Reinforcement learning processes large amounts of layout information to find the optimal partitioning and layout of tasks and devices. For example, neural networks and reinforcement learning are used to partition the TensorFlow computation graph, using a sequence-to-sequence model to output the devices each operation should be placed on, accelerating the training process of the target network. Pipeline parallel task allocation based on deep reinforcement learning divides the solution search into two processes: task partitioning and task allocation. Task partitioning is based on a feedforward neural network, while the task allocation network uses an attention-based Seq2Seq model. The training objective is to obtain a task allocation plan that minimizes the total training time of a single training batch. Furthermore, a DL2 scheduler (DL-driven scheduler for DL ​​clusters) based on supervised learning and reinforcement learning has been proposed, using reinforcement learning to fine-tune job scheduling in deep learning clusters to determine job resource allocation online.

[0006] Efficient task allocation algorithms for distributed training need to consider numerous factors such as the training model, parameter size, node computing power, network, and system scale. In large-scale clusters, existing parallel configuration search algorithms that combine large search spaces and high scalability typically have high time complexity; while low-complexity algorithms usually consider fewer factors, and their search dimensions cannot adequately cover various parallel strategies and cluster topologies. The reasons for these problems are analyzed below:

[0007] (1) Different parallel strategies produce different communication behaviors and there is a dependency between computation and communication logic, making it difficult to search for multiple parallel strategies;

[0008] (2) Different models have different computation and communication requirements, making it difficult to perform general analysis on models with multiple structures.

[0009] (3) Different training clusters have different resource conditions, making it difficult to automatically match training tasks.

[0010] Therefore, distributed training task allocation methods need to have reasonable search targets and make some trade-offs in the search space in order to achieve a balance between search space and time complexity. Summary of the Invention

[0011] The purpose of this invention is to design a deep learning training task allocation method that matches the communication requirements of distributed training with the network resources of the training cluster. By considering the communication requirements of training and the network resources of the cluster, optimizing the parallel strategy and task placement strategy, the communication performance of large-scale distributed training tasks (such as the pre-training of large models) is improved, thereby accelerating distributed training.

[0012] Specifically, this invention proposes a distributed training task allocation method based on communication requirements and network resource matching, characterized by matching communication requirements and network resource matrices, including:

[0013] The communication behavior analysis steps involve analyzing the communication behavior in parallel mode and counting the communication traffic based on the model to be trained and the parallel configuration strategy. The computation-communication task graph of the model is output, and the traffic matrix and communication count matrix are obtained. The traffic matrix represents the distribution of communication traffic between all task nodes of the model to be trained in one iteration. Each point in the traffic matrix represents the total communication traffic between two task nodes. The delay matrix is ​​the number of communications initiated by the nodes during training.

[0014] The task placement search step involves determining the communication requirements between two task nodes based on the traffic matrix and the communication frequency matrix. Then, based on the communication structure of the computing nodes in the distributed training cluster, the network resources between the two computing nodes, including inter-node bandwidth and latency, are determined. Based on these communication requirements and network resources, and with the goal of minimizing training completion time, all task nodes of the model to be trained are allocated to computing nodes to complete the training of the model.

[0015] The distributed training task allocation method based on communication requirements and network resource matching further includes communication behavior analysis based on the training model and parallel configuration, as well as communication behavior analysis combining multiple parallel training methods, specifically including:

[0016] This paper analyzes the distributed training iteration process using data parallelism, tensor parallelism, and pipeline parallelism, and statistically analyzes the computation and communication occurring in the training model, including forward propagation, back propagation, and parameter updates, to obtain the computation-communication task graph. Combining the computation-communication task graph and the communication algorithm, the communication flow matrix during the iteration process is derived, including an overlapping flow matrix and a non-overlapping flow matrix. The overlapping flow matrix records the overlapping computation and communication flows, which come from the communication requirements when using data parallelism, while the non-overlapping flow matrix records the non-overlapping computation and communication flows, which come from the communication requirements when using tensor parallelism and pipeline parallelism.

[0017] It also includes a parallel strategy search step. The distributed training cluster includes multiple computing nodes. Based on the number of cluster nodes in the distributed training cluster, with the goal of minimizing the training completion time, the computation-communication task graph is searched in three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The number and distribution of machines used in the three parallel dimensions are given as the parallel configuration strategy.

[0018] The distributed training task allocation method based on communication requirements and network resource matching is characterized in that,

[0019] The parallel strategy search steps include:

[0020] Step 301: Initialize the parallelism allocation state dp=mp=pp=0; perform prime factorization on the total number of computing nodes N in the distributed training cluster to obtain all factors of N; during the search process, retain the configuration with the minimum task completion time found so as to compare with the new scheme;

[0021] Step 302: Before the allocated computing nodes reach the preset z%, use multiplication steps to stack; enumerate the factors of N from largest to smallest, and accumulate the multiplication to the parallel configuration; calculate the benefit of adding the largest factor to each type of parallelism under the current allocation state, select the case with the highest benefit, and update the allocation state.

[0022] Step 303: Repeat step 302 until the number of allocated computing nodes exceeds N×z%

[0023] Step 304: After allocating computing nodes to reach z%, use the addition step to stack; add a value to the parallelism of each parallelism in the current allocation state, and calculate the number of computing nodes added and the benefits that can be obtained; at the same time, the minimum number of computing nodes added each time is greater than the number of remaining nodes × a%; according to the speedup ratio, retain the case with the highest benefit and the branch that can reach the current best performance x% and update the allocation state for subsequent search.

[0024] Step 305: Repeat step 304 until the allocation is completed or the predetermined search time is reached, and output the parallelism allocation state with the highest benefit at this time as the parallel configuration strategy.

[0025] The parallel strategy search steps include:

[0026] Step 401: Iterate through the initial state, setting the search process to no more than m times, with an initial search count of 0;

[0027] Step 402: Increment the search count by 1 and determine whether the search count is greater than m. If so, output the current optimal task placement as the allocation result; otherwise, proceed to step 403.

[0028] Step 403: Update the scoring function in each iteration to change the influence ratio of overlapping and non-overlapping communication. In the initial state, the scoring function treats overlapping and non-overlapping communication equally, only considering the amount of communication, and prioritizes tasks with large amounts of communication data to nodes with large interconnection bandwidth. Subsequently, the importance of non-overlapping communication is gradually increased.

[0029] Step 404, Scoring Function Design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume (a>1);

[0030] Step 405: Based on the communication requirements between the two task nodes, including communication volume and latency requirements, and sort the communication requirements from largest to smallest according to the scoring function results; sort the network resources between the two computing nodes, including bandwidth and latency, so as to allocate the task communication requirements and network performance in order to obtain the current solution.

[0031] Step 406: Traverse the computation and communication task graph using the current scheme to obtain the completion time of the iteration;

[0032] Step 407: Determine whether the completion time of the current solution is better than the current optimal solution. If so, update the optimal task placement to the current solution; otherwise, proceed to step 402.

[0033] The distributed training task allocation method based on communication requirements and network resource matching is characterized in that the model to be trained is an image classification model used to perform an image classification task, and the task placement search step includes: using the trained model to perform the image classification task to obtain the semantic information or category of the classified image.

[0034] This invention also proposes a distributed training task allocation device based on communication requirements and network resource matching, characterized in that matching the communication requirements and network resource matrix includes:

[0035] The communication behavior analysis module analyzes the communication behavior in parallel mode and counts the communication traffic based on the model to be trained and the parallel configuration strategy. It outputs the computation-communication task graph of the model and obtains the traffic matrix and the number of communication matrices. The traffic matrix represents the distribution of communication traffic between all task nodes of the model to be trained in one iteration. Each point in the traffic matrix represents the total communication traffic between two task nodes. The delay matrix counts the number of communications initiated by the nodes during training.

[0036] The task placement search module obtains the communication requirements between two task nodes based on the traffic matrix and the communication frequency matrix. Based on the communication structure of the computing nodes in the distributed training cluster, it obtains the network resources between the two computing nodes, including inter-node bandwidth and latency. Based on these communication requirements and network resources, and with the goal of minimizing training completion time, all task nodes of the model to be trained are allocated to computing nodes to complete the training of the model.

[0037] The aforementioned distributed training task allocation device based on communication requirements and network resource matching includes a communication behavior analysis module that further analyzes communication behavior based on the training model and parallel configuration, and combines communication behavior from multiple parallel training methods. Specifically, this includes:

[0038] This paper analyzes the distributed training iteration process using data parallelism, tensor parallelism, and pipeline parallelism, and statistically analyzes the computation and communication occurring in the training model, including forward propagation, back propagation, and parameter updates, to obtain the computation-communication task graph. Combining the computation-communication task graph and the communication algorithm, the communication flow matrix during the iteration process is derived, including an overlapping flow matrix and a non-overlapping flow matrix. The overlapping flow matrix records the overlapping computation and communication flows, which come from the communication requirements when using data parallelism, while the non-overlapping flow matrix records the non-overlapping computation and communication flows, which come from the communication requirements when using tensor parallelism and pipeline parallelism.

[0039] It also includes a parallel strategy search module. The distributed training cluster includes multiple computing nodes. Based on the number of cluster nodes in the distributed training cluster, with the goal of minimizing the training completion time, the module searches the computing-communication task graph in three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The module then provides the number and distribution of machines used in each of the three parallel dimensions as the parallel configuration strategy.

[0040] The distributed training task allocation device based on communication requirements and network resource matching is characterized in that,

[0041] The parallel strategy search module includes:

[0042] Module 301: Initialize the parallelism allocation state dp=mp=pp=0; perform prime factorization on the total number of computing nodes N in the distributed training cluster to obtain all factors of N; during the search process, retain the configuration with the minimum task completion time found so as to compare it with the new scheme;

[0043] Module 302: Before the allocated computing nodes reach the preset z%, use multiplication steps to stack; enumerate the factors of N from largest to smallest, and accumulate the multiplications to the parallel configuration; calculate the benefit of adding the largest factor to each type of parallelism under the current allocation state, select the case with the highest benefit, and update the allocation state.

[0044] Module 303, repeat module 302 until the number of allocated computing nodes exceeds N×z%;

[0045] Module 304: After allocating computing nodes to reach z%, use an additive step to stack; add a value to the parallelism of each parallelism in the current allocation state, and calculate the number of additional computing nodes and the resulting benefit; at the same time, the minimum number of additional computing nodes each time is greater than the number of remaining nodes × a%; based on the speedup ratio, retain the case with the highest benefit and the branch that can reach the current best performance x% and update the allocation state for subsequent search;

[0046] Module 305: Repeat module 304 until the allocation is completed or the predetermined search time is reached, and output the parallelism allocation state with the highest benefit at this time as the parallel configuration strategy.

[0047] The parallel strategy search module includes:

[0048] Module 401, Iteration Initial State: Set the search process to no more than m times, and the initial search count is 0;

[0049] Module 402: Increment the search count by 1 and determine if the search count is greater than m. If so, output the current optimal task placement as the allocation result; otherwise, execute module 403.

[0050] Module 403: The scoring function is updated in each iteration, changing the influence ratio of overlapping and non-overlapping communication. In the initial state, the scoring function treats overlapping and non-overlapping communication equally, only considering the amount of communication, and prioritizing tasks with large amounts of communication data to nodes with large interconnection bandwidth. Subsequently, the importance of non-overlapping communication is gradually increased.

[0051] Module 404, Scoring Function Design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume (a>1);

[0052] Module 405: Based on the communication requirements between the two task nodes, including communication volume and latency requirements, and sort the communication requirements from largest to smallest according to the results of the scoring function; sort the network resources between the two computing nodes, including bandwidth and latency, so as to allocate the task communication requirements and network performance in order to obtain the current solution.

[0053] Module 406: Traverse the computation and communication task graph using the current scheme to obtain the completion time of the iteration;

[0054] Module 407: Determine whether the completion time of the current solution is better than the current optimal solution. If so, update the optimal task placement to the current solution; otherwise, execute module 402.

[0055] The distributed training task allocation device based on communication requirements and network resource matching is characterized in that the model to be trained is an image classification model for performing an image classification task, and the task placement search module includes: using the trained model to perform the image classification task to obtain the semantic information or category of the classified image.

[0056] The present invention also proposes a storage medium for storing a program that executes any of the distributed training task allocation methods.

[0057] The present invention also proposes a client for the aforementioned distributed training task allocation device.

[0058] As can be seen from the above solutions, the advantages of the present invention are:

[0059] This invention achieves efficient task allocation by adapting the distributed training communication requirements to the network resources of the training cluster, solving the problems of searching for parallel strategies and placing tasks on the training cluster. It can effectively provide a deployment scheme for parallel training and also help researchers to conduct performance evaluation. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the overall operation of the task allocation system of the present invention.

[0061] Figure 2 This is a flowchart illustrating the communication behavior analysis process of the present invention.

[0062] Figure 3 This is an analysis diagram of the computation-communication task in a hybrid parallel iteration according to the present invention.

[0063] Figure 4 This is a flowchart illustrating the parallel configuration search process of the present invention.

[0064] Figure 5 The search flowchart for the task of this invention is shown below;

[0065] Figure 6 This is a flowchart of the task placement process for this invention. Detailed Implementation

[0066] While researching distributed training task allocation, the inventors discovered that the high time complexity of existing technologies stemmed from complex factors influencing training performance (such as training models, parallel methods, training clusters, and communication algorithms). Through research on the search target and process, the inventors optimized the allocation method, transforming the complex task allocation problem into a problem of matching communication needs with network resources. Because of the dependencies between computation and communication logic, the inventors incorporated consideration of whether computation and communication can overlap when prioritizing communication needs, and incorporated latency considerations when prioritizing network resources.

[0067] To achieve the above-mentioned technical effects, the present invention includes the following key technical points:

[0068] Key Point 1: Communication Behavior Analysis. This involves analyzing and statistically analyzing communication requirements for distributed deep learning training scenarios, specifically steps 200 to 204. This method statistically analyzes the communication required by parallel methods used in distributed training and combines this with the communication algorithms employed to obtain the point-to-point traffic matrix for the entire training process, as well as the number of non-overlapping communications. Point-to-point communication refers to communication between devices involved in the training computation, such as between training devices, between nodes in CPU training, and between GPU cards in GPU training.

[0069] Key point 2: Parallel configuration search, which solves the problem of efficiently searching for possible node distributions in three dimensions—data parallelism, tensor parallelism, and pipeline parallelism—given the number of cluster nodes.

[0070] Key point 3: Task placement search. Based on the communication requirements in distributed training, the bandwidth and latency requirements between nodes are matched with network resources to the greatest extent possible.

[0071] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0072] The task allocation method in this invention comprises three core parts: communication behavior analysis, parallel strategy search, and task placement. Figure 1 This describes the overall operation flow of the task allocation system.

[0073] Step 100. Communication Behavior Analysis: Based on the characteristics of the model to be trained and the parallel configuration strategy, output the traffic matrix and delay matrix for communication. The traffic matrix represents the distribution of communication traffic between all devices during an iteration, with each point (element) representing the total communication traffic between the corresponding two nodes. The delay matrix represents the number of non-overlapping point-to-point communications. The communication behavior analysis model can theoretically analyze all parallel configurations. The initial parallel configuration is determined during the search process; for example, it can start by searching all data in parallel and gradually add other parallel methods. The training process includes multiple network layers, and each layer, even down to the fine-grained level of each operator, may generate traffic. The total communication traffic here is the traffic generated by the distributed training of the entire neural network during an iteration.

[0074] Step 101. Parallel Strategy Search: A search is performed across three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The number and distribution of machines used for each of the three parallelism dimensions are given. Each search simultaneously explores three branches: increasing the parallelism of data parallelism, model parallelism, and pipeline parallelism respectively, and finding the branch with the greatest acceleration. The model can simultaneously perform data, tensor, and pipeline parallelism, and the parallelism degree for each parallelism can be different. For example, 64 machines can use 2-way data parallelism × 4-way model parallelism × 8-way pipeline parallelism; the parallelism degrees given here are 2-4-8. (The text then repeats itself, so the translation stops here.) Figure 1 The basic training allocation can still be completed without relying on step 101, that is, initially all data can be used in parallel. However, this parallel strategy is not necessarily the optimal solution, so a search of step 101 is required.

[0075] Step 102. Task Placement Search: This step optimizes the matching of communication requirements and network resources to the greatest extent possible, ultimately providing a method for placing training tasks onto machines. This placement method offers a superior parallel configuration to accelerate training.

[0076] Figure 2 This is a detailed breakdown of the communication behavior analysis in step 100. The communication behavior analysis section models the communication behavior under different parallel strategies, including precise analysis of the dependencies and temporal relationships between computation and communication. The input to this module is the model being trained and the parallel configuration during distributed training. It analyzes the communication behavior under parallel methods and statistically analyzes communication traffic, outputting the model's computation-communication task graph and a point-to-point traffic matrix.

[0077] Step 200. Communication behavior analysis. Figure 3 Taking a simple three-layer neural network as an example, this paper illustrates a distributed training iteration process that simultaneously utilizes data parallelism, tensor parallelism, and pipeline parallelism. This includes the computational and communication steps occurring during training forward propagation, back propagation, and parameter updates. Arrows indicate the temporal relationships between these steps. The communication behavior in this process comprises sub-processes 201-203.

[0078] Step 201. Data parallelism occurs during backpropagation through communication. The communication algorithm is Allreduce, and the communication volume is the parameter size of each layer. Communication and computation can overlap.

[0079] Step 202. Tensor parallelism enables communication between each neural network layer during forward and backward propagation. The communication algorithm is AllReduce / AllReduce, and communication and computation must not overlap.

[0080] Step 203. Pipeline parallelism occurs between partitioned layers for communication. The communication algorithm is Send. Communication and computation cannot overlap.

[0081] Step 204. Traffic Statistics: Combining the computation-communication task graph and the specific ensemble communication algorithm used, a traffic matrix is ​​calculated for the entire iteration process. Each element in the traffic matrix represents the communication volume between the corresponding nodes, including an overlapping traffic matrix and a non-overlapping traffic matrix. The overlapping traffic matrix records the computation and communication traffic that can overlap, originating from the communication requirements when using data parallelism. The non-overlapping traffic matrix records the computation and communication traffic that cannot overlap, originating from the communication requirements when using tensor parallelism and pipelined parallelism.

[0082] Figure 4 This is a detailed breakdown of step 101, specifically the execution steps for performing the parallel strategy search. This step determines the parallel method to be used.

[0083] Step 300. The goal of the search algorithm is to minimize the training completion time, which can be obtained by traversing the computation and communication task graph in step 200. Since the allocation of parallelism is discrete, we enumerate available resources and use a "multiplication step followed by addition step" strategy. This involves large, fast steps for multiplication and small, refined steps for addition to refine the search. If the number of nodes available for training in the cluster is N, there are three available parallel schemes: data parallelism, tensor parallelism, and pipelined parallelism. Let the parallelism of each be dp, mp, and pp, respectively. Then, the total number of nodes used in the final three allocations is dp × mp × pp <= N. The specific steps of the search are 301~305.

[0084] Step 301. Initialize the parallelism allocation state: dp=mp=pp=0. Perform prime factorization on the available computing nodes N to obtain all factors of N. During the search, retain the configuration with the shortest task completion time found so far for comparison with new solutions.

[0085] Step 302. Before the number of allocated nodes reaches z%, use multiplication steps to accumulate. Enumerate the factors of N from largest to smallest, and accumulate the multiplications to the parallel configuration. Calculate the benefit of adding the largest factor to each parallel configuration under the current allocation state, select the case with the highest benefit, and update the allocation state.

[0086] Step 303. Repeat step 302 until the number of assigned nodes exceeds N×z.

[0087] Step 304. After the node allocation reaches z%, use an additive step to accumulate the parallelism. Add a value to the parallelism of each parallelism in the current allocation state. This value can be greater than the number of remaining nodes × a% of the total number of nodes. For example, if the number of remaining nodes is 64 and the weight a% is 12.5%, then 8, 9, 10, or more nodes can be added. Calculate the number of nodes added and the resulting benefit. To ensure convergence speed, the minimum number of nodes added each time should be greater than the number of remaining nodes × a%. Based on the speedup ratio, retain the case with the highest benefit and the branch that achieves the current best performance x% and update the allocation state for subsequent searches. Adjusting x% can achieve a balance between increasing the search space and accelerating convergence speed.

[0088] Step 305. Repeat step 304 until allocation is complete (all available nodes are allocated, or the predetermined maximum search time is reached) or the predetermined search time is reached, and output the optimal parallel configuration at this point (parallelism degree of each of the three parallelism types). The optimal configuration refers to the parallelism allocation state with the highest benefit, and the metric can be the highest speedup ratio.

[0089] Figure 5 This is a detailed breakdown of step 102, namely the specific execution steps for task placement search. The task placement search algorithm is communication-centric, aiming to match communication requirements with network resources as much as possible.

[0090] Step 400. The goal of the task placement strategy is to minimize the completion time. Since communication tasks can be overlapped or non-overlapping, a scoring function is designed to score communication needs and assess the importance of each type of communication. A greedy algorithm is used for task placement search, while a global optimality pruning strategy is employed to avoid exponential complexity. Specifically, the design is 401~408.

[0091] Step 401. Iterate the initial state. Set the initial state, for example, allocating nodes only based on traffic.

[0092] Step 402. The search process shall not exceed m times.

[0093] Step 403. In each iteration, update the scoring function, changing the influence ratio of overlapping and non-overlapping communication to increase the search space for task placement. Initially, the scoring function can treat both equally, considering only communication volume and prioritizing tasks with large communication data volumes on nodes with high interconnect bandwidth. Subsequently, the importance of non-overlapping communication can be gradually increased.

[0094] Step 404. Scoring Function Design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume (a>1)

[0095] Step 405. Task Placement Method: Consider the communication requirements between two task nodes (considering traffic and latency requirements), and sort the communication requirements from largest to smallest according to the scoring function results; sort the network performance (including bandwidth and latency) between the two nodes, typically: intra-machine network performance > same rack > cross-rack > cross-LAN. Allocate tasks according to their communication requirements and network performance in that order. For example... Figure 6 As shown.

[0096] Step 406. Calculate the completion time of an iteration, which can be obtained by traversing the computation communication task graph in step 200.

[0097] Step 407. Determine whether the completion time of the current solution is better than the current optimal solution. If yes, jump to 408; otherwise, jump to 402 to continue the next iteration.

[0098] Step 408. Find a better solution and update the current optimal task placement.

[0099] The main models used in this application for testing are image classification models such as AlexNet, ResNet, and VGG, used to perform image classification tasks. The process of completing large-scale cluster multi-task training is as follows: First, when deploying a specific neural network model, a computational task graph for training is generated. Then, possible parallel configurations are searched. For each configuration, a communication demand matrix (including a point-to-point traffic matrix and a communication frequency matrix) is generated using the communication behavior analysis method proposed in this invention. The communication demand matrix and network resource matrix (including point-to-point bandwidth and latency matrices) are then adapted to determine the most suitable parallel configuration. Finally, this parallel configuration is deployed to the training environment for model training. For example, in distributed training with 16 nodes, the optimal parallel configuration might be 4-way data parallelism - 2-way tensor parallelism - 2-way pipeline parallelism, and this configuration is then deployed in the cluster.

[0100] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0101] This invention also proposes a distributed training task allocation device based on communication requirements and network resource matching, characterized in that matching the communication requirements and network resource matrix includes:

[0102] The communication behavior analysis module analyzes the communication behavior in parallel mode and counts the communication traffic based on the model to be trained and the parallel configuration strategy. It outputs the computation-communication task graph of the model and obtains the traffic matrix and the number of communication matrices. The traffic matrix represents the distribution of communication traffic between all task nodes of the model to be trained in one iteration. Each point in the traffic matrix represents the total communication traffic between two task nodes. The delay matrix counts the number of communications initiated by the nodes during training.

[0103] The task placement search module obtains the communication requirements between two task nodes based on the traffic matrix and the communication frequency matrix. Based on the communication structure of the computing nodes in the distributed training cluster, it obtains the network resources between the two computing nodes, including inter-node bandwidth and latency. Based on these communication requirements and network resources, and with the goal of minimizing training completion time, all task nodes of the model to be trained are allocated to computing nodes to complete the training of the model.

[0104] The aforementioned distributed training task allocation device based on communication requirements and network resource matching includes a communication behavior analysis module that further analyzes communication behavior based on the training model and parallel configuration, and combines communication behavior from multiple parallel training methods. Specifically, this includes:

[0105] This paper analyzes the distributed training iteration process using data parallelism, tensor parallelism, and pipeline parallelism, and statistically analyzes the computation and communication occurring in the training model, including forward propagation, back propagation, and parameter updates, to obtain the computation-communication task graph. Combining the computation-communication task graph and the communication algorithm, the communication flow matrix during the iteration process is derived, including an overlapping flow matrix and a non-overlapping flow matrix. The overlapping flow matrix records the overlapping computation and communication flows, which come from the communication requirements when using data parallelism, while the non-overlapping flow matrix records the non-overlapping computation and communication flows, which come from the communication requirements when using tensor parallelism and pipeline parallelism.

[0106] It also includes a parallel strategy search module. The distributed training cluster includes multiple computing nodes. Based on the number of cluster nodes in the distributed training cluster, with the goal of minimizing the training completion time, the module searches the computing-communication task graph in three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The module then provides the number and distribution of machines used in each of the three parallel dimensions as the parallel configuration strategy.

[0107] The distributed training task allocation device based on communication requirements and network resource matching is characterized in that,

[0108] The parallel strategy search module includes:

[0109] Module 301: Initialize the parallelism allocation state dp=mp=pp=0; perform prime factorization on the total number of computing nodes N in the distributed training cluster to obtain all factors of N; during the search process, retain the configuration with the minimum task completion time found so as to compare it with the new scheme;

[0110] Module 302: Before the allocated computing nodes reach the preset z%, use multiplication steps to stack; enumerate the factors of N from largest to smallest, and accumulate the multiplications to the parallel configuration; calculate the benefit of adding the largest factor to each type of parallelism under the current allocation state, select the case with the highest benefit, and update the allocation state.

[0111] Module 303, repeat module 302 until the number of allocated computing nodes exceeds N×z%;

[0112] Module 304: After allocating computing nodes to reach z%, use an additive step to stack; add a value to the parallelism of each parallelism in the current allocation state, and calculate the number of additional computing nodes and the resulting benefit; at the same time, the minimum number of additional computing nodes each time is greater than the number of remaining nodes × a%; based on the speedup ratio, retain the case with the highest benefit and the branch that can reach the current best performance x% and update the allocation state for subsequent search;

[0113] Module 305: Repeat module 304 until the allocation is completed or the predetermined search time is reached, and output the parallelism allocation state with the highest benefit at this time as the parallel configuration strategy.

[0114] The parallel strategy search module includes:

[0115] Module 401, Iteration Initial State: Set the search process to no more than m times, and the initial search count is 0;

[0116] Module 402: Increment the search count by 1 and determine if the search count is greater than m. If so, output the current optimal task placement as the allocation result; otherwise, execute module 403.

[0117] Module 403: The scoring function is updated in each iteration, changing the influence ratio of overlapping and non-overlapping communication. In the initial state, the scoring function treats overlapping and non-overlapping communication equally, only considering the amount of communication, and prioritizing tasks with large amounts of communication data to nodes with large interconnection bandwidth. Subsequently, the importance of non-overlapping communication is gradually increased.

[0118] Module 404, Scoring Function Design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume (a>1);

[0119] Module 405: Based on the communication requirements between the two task nodes, including communication volume and latency requirements, and sort the communication requirements from largest to smallest according to the results of the scoring function; sort the network resources between the two computing nodes, including bandwidth and latency, so as to allocate the task communication requirements and network performance in order to obtain the current solution.

[0120] Module 406: Traverse the computation and communication task graph using the current scheme to obtain the completion time of the iteration;

[0121] Module 407: Determine whether the completion time of the current solution is better than the current optimal solution. If so, update the optimal task placement to the current solution; otherwise, execute module 402.

[0122] The distributed training task allocation device based on communication requirements and network resource matching is characterized in that the model to be trained is an image classification model for performing an image classification task, and the task placement search module includes: using the trained model to perform the image classification task to obtain the semantic information or category of the classified image.

[0123] The present invention also proposes a storage medium for storing a program that executes any of the distributed training task allocation methods.

[0124] The present invention also proposes a client for the aforementioned distributed training task allocation device.

[0125] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A distributed training task allocation method based on communication requirements and network resource matching, characterized in that... Matching communication needs with network resource matrices, including: The communication behavior analysis steps involve analyzing the communication behavior in parallel mode and counting the communication traffic based on the model to be trained and the parallel configuration strategy. The computation-communication task graph of the model is output, and the traffic matrix and communication count matrix are obtained. The traffic matrix represents the distribution of communication traffic between all task nodes of the model to be trained in one iteration. Each point in the traffic matrix represents the total communication traffic between two task nodes. The communication count matrix counts the number of communications initiated by the nodes during training. The parallel strategy search steps involve a distributed training cluster comprising multiple computing nodes. Based on the number of cluster nodes in the distributed training cluster, and with the goal of minimizing the training completion time, a search is performed on the computation-communication task graph in three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The number and distribution of machines used in each of the three parallel dimensions are given as the parallel configuration strategy. The task placement search step obtains the communication requirements between two task nodes based on the traffic matrix and the communication frequency matrix, and obtains the network resources between two computing nodes, including inter-node bandwidth and latency, based on the communication structure of computing nodes in the distributed training cluster. Based on the communication requirements and network resources, with the goal of minimizing the training completion time, all task nodes of the model to be trained are assigned to computing nodes to complete the training of the model to be trained.

2. The distributed training task allocation method based on communication requirements and network resource matching as described in claim 1, wherein the communication behavior analysis step further includes: The distributed training iteration process using data parallelism, tensor parallelism, and pipeline parallelism is analyzed. The computation and communication that occur in the training forward propagation, back propagation, and parameter update of the model are statistically analyzed, and the computation-communication task graph is obtained. Combining the computation-communication task graph and the communication algorithm, the communication flow matrix during the iteration process is derived, including an overlapping flow matrix and a non-overlapping flow matrix. The overlapping flow matrix records the overlapping flow of computation and communication, which comes from the communication requirements when using data parallelism. The non-overlapping flow matrix records the non-overlapping flow of computation and communication, which comes from the communication requirements when using tensor parallelism and pipelined parallelism.

3. The distributed training task allocation method based on communication requirements and network resource matching as described in claim 2, characterized in that, The parallel strategy search steps include: Step 301: Initialize the parallelism allocation state dp=mp=pp=0; perform prime factorization on the total number of computing nodes N in the distributed training cluster to obtain all factors of N; during the search process, retain the configuration with the minimum task completion time found so as to compare with the new scheme; Step 302: Before the allocated computing nodes reach the preset z%, use multiplication steps to stack; enumerate the factors of N from largest to smallest, and accumulate the multiplication to the parallel configuration; calculate the benefit of adding the largest factor to each type of parallelism under the current allocation state, select the case with the highest benefit, and update the allocation state. Step 303: Repeat step 302 until the number of allocated computing nodes exceeds N×z% Step 304: After allocating computing nodes to reach z%, use the addition step to stack; add a value to the parallelism of each parallelism in the current allocation state, and calculate the number of computing nodes added and the benefits that can be obtained; at the same time, the minimum number of computing nodes added each time is greater than the number of remaining nodes × a%; according to the speedup ratio, retain the case with the highest benefit and the branch that can reach the current best performance x% and update the allocation state for subsequent search. Step 305: Repeat step 304 until the allocation is completed or the predetermined search time is reached, and output the parallelism allocation state with the highest benefit at this time as the parallel configuration strategy. The task placement search steps include: Step 401: Iterate through the initial state, setting the search process to no more than m times, with an initial search count of 0; Step 402: Increment the search count by 1 and determine whether the search count is greater than m. If so, output the current optimal task placement as the allocation result; otherwise, proceed to step 403. Step 403: Update the scoring function in each iteration to change the influence ratio of overlapping and non-overlapping communication. In the initial state, the scoring function treats overlapping and non-overlapping communication equally, only considering the amount of communication, and prioritizes tasks with large amounts of communication data to nodes with large interconnection bandwidth. Subsequently, the importance of non-overlapping communication is gradually increased. Step 404, Scoring function design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume, a>1; Step 405: Based on the communication requirements between the two task nodes, including communication volume and latency requirements, and sort the communication requirements from largest to smallest according to the scoring function results; sort the network resources between the two computing nodes, including bandwidth and latency, so as to allocate the task communication requirements and network performance in order to obtain the current solution. Step 406: Traverse the computation and communication task graph using the current scheme to obtain the completion time of the iteration; Step 407: Determine whether the completion time of the current solution is better than the current optimal solution. If so, update the optimal task placement to the current solution; otherwise, proceed to step 402.

4. The distributed training task allocation method based on communication requirements and network resource matching as described in claim 1, characterized in that, The model to be trained is an image classification model used to perform an image classification task. The task placement search step includes: using the trained model to perform the image classification task to obtain the semantic information or category of the classified image.

5. A distributed training task allocation device based on communication requirements and network resource matching, characterized in that... Matching communication needs with network resource matrices, including: The communication behavior analysis module analyzes the communication behavior in parallel mode and counts the communication traffic based on the model to be trained and the parallel configuration strategy. It outputs the computation-communication task graph of the model and obtains the traffic matrix and the number of communication matrices. The traffic matrix represents the distribution of communication traffic between all task nodes of the model to be trained in one iteration. Each point in the traffic matrix represents the total communication traffic between two task nodes. The number of communication matrices counts the number of communications initiated by the nodes during training. The parallel strategy search module is used in a distributed training cluster that includes multiple computing nodes. Based on the number of cluster nodes in the distributed training cluster, and with the goal of minimizing the training completion time, the module searches the computation-communication task graph in three dimensions: data parallelism, tensor parallelism, and pipeline parallelism. The module then provides the number and distribution of machines used in each of the three parallel dimensions as the parallel configuration strategy. The task placement search module obtains the communication requirements between two task nodes based on the traffic matrix and the communication frequency matrix, and obtains the network resources between two computing nodes, including inter-node bandwidth and latency, based on the communication structure of computing nodes in the distributed training cluster. Based on the communication requirements and network resources, and with the goal of minimizing the training completion time, all task nodes of the model to be trained are assigned to computing nodes to complete the training of the model to be trained.

6. The distributed training task allocation device based on communication requirements and network resource matching as described in claim 5, wherein the communication behavior analysis module further includes: The distributed training iteration process using data parallelism, tensor parallelism, and pipeline parallelism is analyzed. The computation and communication that occur in the training forward propagation, back propagation, and parameter update of the model are statistically analyzed, and the computation-communication task graph is obtained. Combining the computation-communication task graph and the communication algorithm, the communication flow matrix during the iteration process is derived, including an overlapping flow matrix and a non-overlapping flow matrix. The overlapping flow matrix records the overlapping flow of computation and communication, which comes from the communication requirements when using data parallelism. The non-overlapping flow matrix records the non-overlapping flow of computation and communication, which comes from the communication requirements when using tensor parallelism and pipelined parallelism.

7. The distributed training task allocation device based on communication requirements and network resource matching as described in claim 6, characterized in that, The parallel strategy search module includes: Module 301: Initialize the parallelism allocation state dp=mp=pp=0; perform prime factorization on the total number of computing nodes N in the distributed training cluster to obtain all factors of N; during the search process, retain the configuration with the minimum task completion time found so as to compare with the new scheme; Module 302: Before the allocated computing nodes reach the preset z%, use multiplication steps to stack; enumerate the factors of N from largest to smallest, and accumulate the multiplications to the parallel configuration; calculate the benefit of adding the largest factor to each type of parallelism under the current allocation state, select the case with the highest benefit, and update the allocation state. Module 303, repeat module 302 until the number of allocated computing nodes exceeds N×z%; Module 304: After allocating computing nodes to reach z%, use an additive step to stack; add a value to the parallelism of each parallelism in the current allocation state, and calculate the number of additional computing nodes and the resulting benefit; at the same time, the minimum number of additional computing nodes each time is greater than the number of remaining nodes × a%; based on the speedup ratio, retain the case with the highest benefit and the branch that can reach the current best performance x% and update the allocation state for subsequent search; Module 305: Repeat module 304 until the allocation is completed or the predetermined search time is reached, and output the parallelism allocation state with the highest benefit at this time as the parallel configuration strategy. This task includes placing the search module, which includes: Module 401, Iteration Initial State: Set the search process to no more than m times, and the initial search count is 0; Module 402: Increment the search count by 1 and determine if the search count is greater than m. If so, output the current optimal task placement as the allocation result; otherwise, execute module 403. Module 403: The scoring function is updated in each iteration, changing the influence ratio of overlapping and non-overlapping communication. In the initial state, the scoring function treats overlapping and non-overlapping communication equally, only considering the amount of communication, and prioritizing tasks with large amounts of communication data to nodes with large interconnection bandwidth. Subsequently, the importance of non-overlapping communication is gradually increased. Module 404, Scoring Function Design: Overlapping communication task score = communication volume; Non-overlapping communication task score = a × communication volume, a > 1; Module 405: Based on the communication requirements between the two task nodes, including communication volume and latency requirements, and sort the communication requirements from largest to smallest according to the results of the scoring function; sort the network resources between the two computing nodes, including bandwidth and latency, so as to allocate the task communication requirements and network performance in order to obtain the current solution. Module 406: Traverse the computation and communication task graph using the current scheme to obtain the completion time of the iteration; Module 407: Determine whether the completion time of the current solution is better than the current optimal solution. If so, update the optimal task placement to the current solution; otherwise, execute module 402.

8. The distributed training task allocation device based on communication requirements and network resource matching as described in claim 5, characterized in that, The model to be trained is an image classification model used to perform image classification tasks. The task placement search module includes: using the trained model to perform image classification tasks to obtain semantic information or category of the classified image.

9. A storage medium for storing a program that executes the distributed training task allocation method as described in any one of claims 1 to 4.

10. A client for use with the distributed training task allocation apparatus of any one of claims 5 to 8.

Images