A load balancing based model automatic parallel method, device and storage medium

Abstract

This invention discloses a load-balanced automatic parallel modeling method, device, and storage medium. First, it analyzes the key factors affecting the execution performance of operators and models (operator in-degree, tensor shape, and tensor data type), and proposes a method for constructing a performance evaluation model based on operator characteristics to assess the computational, communication, and overall costs of operators, as well as the training performance costs of the model. Then, with the goal of load balancing across devices, a layer-by-layer partitioning method based on topological sorting is used to quickly divide the neural network model into multiple sub-models with balanced overall costs, achieving coarse-grained partitioning. Finally, based on the model's training performance evaluation model, and according to the communication characteristics between operators, a fine-grained model partitioning and scheduling method based on communication optimization is used to fine-grainedly adjust the coarse-grained sub-models, reducing the amount of cross-device communication tensor transmission to achieve optimal global model scheduling.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing technology and relates to an automatic parallelization method, device and storage medium based on load balancing model. Technical Background

[0002] In recent years, with the informatization of industries and the rise of the internet, the scale of data has exploded, reaching the exabyte (EB) level globally. This massive amount of data has brought new opportunities to the field of artificial intelligence. How to fully utilize this vast amount of unlabeled data to drive the rapid development of fields such as finance, medicine, recommender systems, and molecular dynamics has become an urgent problem for both industry and academia. Deep learning technology, due to its powerful learning capabilities, can extract features and value from unlabeled data and has been widely applied in various fields, changing people's lifestyles and improving productivity. For example, DeepMind's Alphafold, leveraging the powerful learning capabilities of AI models, accurately predicts protein structures, accelerating the development of the biomedical field.

[0003] As data volumes increase, deep learning models become increasingly complex, with parameter sizes growing from tens of thousands (ResNet) to hundreds of millions (Bert_large) and trillions (GPT-3, Pangu, Wudao 2.0, PaML, etc.). The ever-growing training datasets and model parameters limit the training of AI models to hardware performance constraints; single devices can no longer meet the training needs of these large AI models. Therefore, leveraging distributed technologies to train large deep learning models across devices has become crucial for improving model training efficiency and accelerating the deployment of AI applications.

[0004] Current mainstream distributed machine learning systems such as PyTorch, TensorFlow, and MindSpore typically describe neural network models as data flow graphs. They achieve distributed parallel execution of AI models by partitioning and scheduling these data flow graphs. When a single device's resources meet the computational and storage requirements of a neural network model, the data parallelism method is primarily used to accelerate model training. This method copies the model to multiple device nodes and divides the dataset into multiple subsets, scheduling them across multiple devices for parallel execution. While data parallelism performs parallel execution along the data dimension and can solve the problem of training massive amounts of data, it cannot handle scenarios where a single device's resources are insufficient to meet the computational and storage requirements of large deep learning models. Therefore, parallel training of large deep learning models has become an inevitable trend, which involves partitioning the large deep learning model into multiple sub-models and scheduling them across different devices for parallel execution.

[0005] Designing and implementing a model parallelism strategy for large deep learning models to optimize distributed parallel execution performance is a challenging problem in model parallelism. Current model parallelism strategies primarily rely on expert experience, requiring in-depth understanding of model structure characteristics and device architecture. This necessitates knowledge across multiple domains, including AI algorithms, architecture, and distributed parallel computing, posing a significant challenge to developers. In practical applications, AI algorithm engineers struggle to design optimal model parallelism solutions, severely hindering the rapid application and development of AI technology. For example, Wu and Sutskever et al. proposed methods for horizontal or vertical partitioning across layers, scheduling LSTM, Attention, and SoftMax layers to different computing devices. Krizhevsky developed a hybrid parallelism strategy based on the characteristics of convolutional and fully connected layers, employing data parallelism for convolutional layers with small parameters and high computational cost, and model parallelism for fully connected layers with large parameters and low computational cost. These methods require developers to have a deep understanding of model characteristics and device architecture. Furthermore, as the model size increases, the number of combinations of distributed strategies grows exponentially, resulting in a very large search space. It is difficult to quickly design and implement high-performance model parallel strategies based on expert experience.

[0006] To improve the efficiency of designing, implementing, and executing distributed parallel strategies for deep learning models, academia and industry have proposed automatic parallelization methods for deep learning models. These methods automatically divide large models into multiple smaller models and schedule them for execution on different devices. Currently, the main types of automatic parallelization methods are machine learning-based and graph algorithm-based.

[0007] Machine learning-based automatic model parallelization methods extract features from neural network models and device cluster topology, leveraging the learning capabilities of machine learning algorithms to find optimal distributed parallel strategies for deep learning models, aiming at optimal model scheduling and execution. For example, the Google team proposed the reinforcement learning-based automatic model parallelization frameworks ColorRL and Hierarchical, which extract features from AI models and training devices to segment and schedule models, and adjust reinforcement learning parameters based on feedback to find optimal distributed training strategies. However, these methods require frequent sampling in real-world environments, making the search process costly. Subsequently, Ravichandra et al. from MIT proposed Placeto, which uses graph embedding to learn the model's topology information, making the strategy somewhat portable. Placeto introduces an execution simulator to replace the real environment during the search process, accelerating the strategy search speed. Later, Wang et al., building on Placeto, found that by changing the model traversal order, the input order of reverse topology sorting better helps the network learn the relationship between parent and child nodes in the model to be scheduled, thereby improving the convergence performance of reinforcement learning models. Liu J et al. proposed the HeterPS framework, which, based on the characteristics of different layers of a DNN neural network model, utilizes reinforcement learning to schedule each layer onto suitable training devices, minimizing overall cost while satisfying throughput constraints. Wu et al. proposed Trinity, an adaptive distributed parallel training method based on reinforcement learning. This method utilizes a proximal policy optimization approach to extend the offline learning capability of the policy network and designs an execution simulator to predict the single-step execution time under a specified policy, thereby accelerating the policy search speed.

[0008] Automatic parallelization based on graph algorithms has become another mainstream approach. Compared to automatic parallelization based on machine learning, it can quickly search for model parallelization strategies without requiring multiple iterations. Jia et al. proposed the OptCNN framework, which uses dynamic programming to evaluate the model by cost and quickly search for the optimal parallelization strategy within the search space. However, its coarse-grained model partitioning method limits the performance improvement of the searched strategies. Subsequently, Jia et al. proposed the FlexFlow framework, which divides tensors into four dimensions: Sample, Operator, Attribute, and Parameter. Based on these dimensions, it establishes a high-dimensional SOAP search space and then uses the Markov chain Monte Carlo algorithm to search for the optimal parallelization strategy within this space. Beomyeol et al. proposed the Baechi framework, which incorporates three graph algorithms: topological sorting, earliest start time, and minimum communication. It has wide applicability and fast search speed, and can find model parallelization strategies for most neural network models within tens of seconds. Zheng et al. proposed the Alpa framework, which uses integer linear programming to find the tensor parallel partitioning method within operators and dynamic programming to find the optimal pipelined parallelism method between operators, achieving hybrid tensor and pipeline parallelism. Zeng et al. proposed the FD-DPS algorithm, which improves parallel performance by splitting computationally intensive operators into tensor dimensions based on operator properties, and then determines the critical path based on dynamic priority and achieves optimal scheduling of operators on the critical path.

[0009] In summary, current operator-level automatic model parallelization methods mainly fall into two categories: machine learning-based and graph algorithm-based methods. Machine learning-based methods rely on the learning and iterative updating of parameters, requiring significant computational resources and time. While graph algorithm-based methods offer faster solutions, they necessitate consideration of operator structure, execution performance, and device topology, employing algorithms like dynamic programming and shortest path search. The resulting distributed parallel strategies are hampered by model structure and device topology, leading to high search difficulty and poor portability. Both of these mainstream methods suffer from complex search processes and low efficiency for models with complex structures and large parameter sizes. Therefore, this invention addresses these shortcomings by designing and implementing a load-balancing-based automatic model parallelization method. Summary of the Invention

[0010] This invention designs and implements a load-balanced automatic parallel model training method, device, and storage medium, providing an optimal parallel training method for large-scale complex neural networks.

[0011] The overall technical concept of this invention is as follows: First, based on the structure and computational attributes of the neural network model, this invention analyzes the key characteristics affecting operator performance (such as the in-degree and out-degree of operators, the shape of tensors, and the data type of tensors). It proposes a scheme for constructing a performance evaluation model based on operator characteristics to assess the cost of operators and model training performance within the model. Then, guided by operator cost overhead and aiming at balanced overall cost across devices, a layer-by-layer partitioning scheme based on topological sorting is used to coarsely partition the neural network, quickly dividing the model into multiple sub-models with balanced overall cost. Finally, a fine-grained model partitioning and scheduling scheme based on communication optimization is proposed. For the coarsely partitioned sub-models, fine-grained adjustments are made with the goal of reducing cross-device communication tensor transmission, thereby achieving optimal global model scheduling.

[0012] Based on the above overall technical concept, the first aspect of the present invention provides an automatic parallel model method based on load balancing, comprising the following steps:

[0013] Step 1: Based on the neural network model structure, computational attributes, and other dimensions, quantitatively analyze the key features that affect the execution performance of operators, and design a scheme to build a performance evaluation model based on operator features in order to evaluate the cost of operators in the model and the training performance cost of the model.

[0014] First, the neural network model structure is extracted and device resource groups are assembled, and the model data flow graph is abstracted into a computation graph. Abstracting the structure, resources, and other information of computing devices into a device topology diagram.

[0015] Then, the factors affecting the operator's execution performance are analyzed, and key features such as the operator's in-degree, the shape of the input and output tensors, and the data type of the tensors are extracted. The communication cost S of the operator is constructed using the operator's in-degree, the shape of the input and output tensors, and the data type of the tensors; the communication cost S of the operator is then used... in and output communication cost S out The computational cost C of establishing the operator; the comprehensive cost T of constructing the model using the communication cost S and computational cost C of the operator, as well as the proportion λ of memory-intensive operators.

[0016] Finally, using the computational cost C of the operator and the sum of the cross-device transfer tensors of the model, a model training performance evaluation model E(R;G,T) is constructed, representing the performance evaluation of the model given the computation graph G and the computation device graph. Evaluation of distributed training performance of policy R under the given conditions.

[0017] Step 2: Based on the structural characteristics of the neural network model, guided by the cost generated in the performance evaluation in Step 1, and with the goal of balancing the overall cost load of each device, a layer-based partitioning scheme based on topological sorting is used to achieve coarse-grained segmentation of the neural network model, quickly dividing the neural network model into multiple sub-models with balanced performance and cost.

[0018] 1) First, based on the comprehensive cost of the operators, calculate the sum of the comprehensive costs of all operators in the neural network model;

[0019] 2) Then, based on the sum of the overall costs and the number of computing devices in the cluster, calculate the average cost that each computing device needs to bear and use it as the load threshold;

[0020] 3) Finally, starting from the structural characteristics of the neural network model, with the goal of reducing cross-device tensor transmission, the neural network model is traversed layer by layer according to the topological sorting order and placed on each computing device in turn until the computing device reaches the load threshold and then switches to the next idle computing device, thereby quickly dividing the model into multiple sub-models with balanced comprehensive costs.

[0021] Step 3: Based on the communication characteristics between operators, design a fine-grained model segmentation and scheduling scheme based on communication optimization. Adjust the upper limit of device load using the correction threshold, and perform fine-grained adjustments on the coarse-grained sub-models with the goal of reducing cross-device communication tensor transmission, so as to achieve global model optimal scheduling.

[0022] 1) First, sort all the tensors currently being transmitted across devices and extract the tensor with the highest transmission cost.

[0023] 2) Then, based on the transmission tensor with the highest cost and the corrected load threshold, the scheduling information of local operators in the sub-model is adjusted to reduce the size of the tensor transmitted across devices and the performance cost of model training.

[0024] 3) Finally, repeat the above process until the specified number of adjustments is reached or all tensors in the model no longer meet the conditions for forward and backward adjustments.

[0025] A second aspect of the present invention provides a load-balanced model automatic parallel device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned load-balanced model automatic parallel method.

[0026] A third aspect of the present invention provides a computer-readable storage medium storing a computer program for executing the above-described automatic parallel method based on load balancing.

[0027] The beneficial effects of this invention are:

[0028] This invention analyzes key features affecting operator execution performance based on neural network model structure and computational attributes. It constructs evaluation models for operator communication cost, computational cost, and overall cost, as well as a model training performance evaluation model, based on features such as operator in-degree, input / output tensor shape, and tensor data type, without requiring data collection in a real-world environment. Based on the dependencies between operators in the model, it utilizes topological sorting-based layer-by-layer partitioning to achieve coarse-grained segmentation of the neural network model while maintaining load balancing of the overall cost across devices, thus accelerating model segmentation and scheduling. A fine-grained model segmentation and scheduling scheme based on communication optimization adjusts the model's local scheduling strategy with the goal of minimizing cross-device tensor transmission, effectively improving strategy execution performance. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of direct correlation operators and indirect correlation operators;

[0030] Figure 2 This is a schematic diagram of a layer-by-layer partitioning scheme based on topological sorting;

[0031] Figure 3 This is a diagram illustrating forward and backward adjustments;

[0032] Figure 4 This is a structural diagram of an automatic parallel device based on a load balancing model. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and specific implementation steps:

[0034] In one embodiment, the automatic parallelization method for load balancing models involved in this application includes the following steps:

[0035] Step 1: Based on the neural network model structure, computational attributes and other dimensions, the key features affecting the execution performance of operators are quantitatively analyzed, and a scheme for constructing a performance evaluation model based on operator features is proposed to evaluate the cost of operators in the model and the training performance cost of the model.

[0036] As a preferred example:

[0037] First, the neural network model structure is extracted and a device resource group is assembled to abstract the computation graph. and device topology diagram In the computation graph In the diagram, vertex O represents a neural network operator, and E represents a directed edge between vertices; device topology diagram. Mid-vertex Q represents the computing device (e.g., CPU or GPU). Edges Indicates device q i and equipment q j Connections between them (such as NV Link, PCI-E or other connection methods).

[0038] Then, the factors affecting the execution performance of the operator are analyzed, and key features such as the operator's in-degree, the shape of the input and output tensors, and the data type of the tensors are extracted. Based on these key features, the cost model for the operator's communication, computation, and synthesis is defined as follows:

[0039] In this embodiment, the operator communication cost is calculated using the shape and data type of all output tensors of the operator, as shown in formula (1), since the communication cost is closely related to the tensor transfer between operators.

[0040]

[0041] Where K represents the number of output tensors, H1, H2, ..., H... h The size of the tensor is h-dimensional, and sizeof(type) represents the number of bytes of data in the format of type.

[0042] In this embodiment, the operator computation cost refers to the overhead incurred by tensor computation, which reflects the tensor transformation process. Therefore, the computation cost is calculated based on the input and output tensors of the operator, as shown in formula (2):

[0043]

[0044] Where S in and S out Let R represent the sum of the input and output tensors of the operator, respectively. When the difference between the input and output tensors is large, it means that the calculation process of the operator is also more complex and the calculation cost of the operator is also greater. R represents the cost conversion rate.

[0045] In this embodiment, the operator comprehensive cost is an index that comprehensively evaluates the operator by combining its structural characteristics, computational cost, and communication cost, as shown in formula (3).

[0046]

[0047] Where S represents the operator communication cost calculated by formula (1); C represents the operator computation cost calculated by formula (2); ∑P para ∑P represents the total number of memory-intensive operators. compThis represents the total number of computationally intensive operators; λ is a proportionality coefficient that indicates the proportion of memory-intensive operators to the total number of memory-intensive and computationally intensive operators. Based on this operator synthesis cost calculation scheme, models with different structural characteristics can automatically generate operator synthesis costs that conform to their own model characteristics.

[0048] Then, based on these cost models, a training performance evaluation model for the strategy is further established.

[0049] In some embodiments, the training performance evaluation model is calculated from the computational cost of the operator and the size of the cross-device transmission tensor in the model, as shown in formula (4):

[0050]

[0051] Where R represents the distributed parallel strategy, and E(R; G, D) represents the distributed training performance evaluation model of strategy R given the computation graph G and the computational device graph D. N represents the number of operators in the model, M represents the number of tensors transferred between operators in the model; C n Y represents the computational cost of the nth operator. m This represents the size of the m-th tensor. The ζ(·) method indicates whether the tensor needs to be transmitted across devices. If it needs to be transmitted across devices, it is 1; otherwise, it is 0.

[0052] Step 2: Based on the structural characteristics of the neural network model, guided by the cost generated by the performance evaluation method in Step 1, and with the goal of balancing the overall cost load of each device, a layer-based partitioning scheme based on topological sorting is used to achieve coarse-grained segmentation of the neural network model, quickly dividing the neural network model into multiple sub-models with balanced overall cost.

[0053] For most dataflow graph-based neural network frameworks, operators are the basic units of computation and scheduling in the neural network model. Each operator receives the output tensors of its directly connected operators, computes new tensors based on these tensors, and outputs these new tensors to its directly connected operators, and so on, driving the computation of the neural network model layer by layer. To clearly describe the structural dependencies between operators, the following definitions are provided:

[0054] In this embodiment, the direct correlation operator is defined as follows: if operator L and operator P have a direct dependency relationship, then operator L and operator P are directly correlated operators.

[0055] In this embodiment, the indirect correlation operator is defined as follows: if operator L and operator P do not have a direct dependency relationship, then operator L and operator P are indirect correlation operators.

[0056] like Figure 1As shown, for operator P, the gray operators directly connected to it are its directly related operators, and the black operators indirectly connected to it are its indirectly related operators.

[0057] Based on the structural characteristics of the above model, step 2, based on the cost of operators and load balancing strategies, utilizes a layer-by-layer partitioning method based on topological sorting to divide and schedule the neural network model layer by layer. This aims to place operators and their directly related operators on the same computing device as quickly as possible, thereby reducing the impact of critical tensors on the model's computation. Figure 2 As shown.

[0058] As a preferred example:

[0059] First, the computational cost, communication cost, and overall cost of each operator are generated using the method of constructing a performance evaluation model based on operator features in step 1.

[0060] Then, the sum of the comprehensive costs of each operator in the entire model is calculated and divided by the number of computing devices to obtain the average load that each computing device needs to bear, which is used as the load threshold of the device.

[0061] The load threshold is calculated as shown in equation (5):

[0062]

[0063] Among them, T n Let A represent the overall cost of the nth operator, Z represent the number of computing devices, and A be the load threshold, representing the average overall cost that the current model needs to bear on each device.

[0064] Finally, starting from the structural characteristics of the model, with the goal of reducing cross-device tensor transmission, the neural network model is traversed layer by layer according to the topological sorting order and placed on each computing device in turn until the computing device reaches the load threshold A, and then switched to the next idle computing device. In this way, the model is quickly divided into multiple sub-models with balanced comprehensive costs.

[0065] Step 3: Based on the communication characteristics between operators, a fine-grained model segmentation and scheduling scheme based on communication optimization is proposed. The upper limit of device load is adjusted by adjusting the correction threshold, and fine-grained adjustments are made for the coarse-grained sub-models with the goal of reducing cross-device communication tensor transmission, so as to achieve global model optimal scheduling.

[0066] Based on the model training performance evaluation model proposed in Step 1, it can be seen that the cross-device communication tensor between operators is a key factor affecting model execution performance. Therefore, in order to adjust and optimize the coarse-grained sub-model after partitioning using the topology sorting-based layer-by-layer partitioning scheme in Step 2, the transmission of cross-device communication tensors between operators should be minimized.

[0067] This embodiment proposes a fine-grained model segmentation and scheduling scheme based on communication optimization.

[0068] First, sort all the tensors currently being transmitted across devices and extract the tensor t with the highest transmission cost. ij , indicating operator o i Transmission to operator o j The tensor is used as the object of adjustment.

[0069] Then, based on the target tensor t ij Adjusting backward or forward can reduce the size of the tensor transmitted across devices. Adjusting backward refers to reducing the tensor t... ij The operator it points to is placed in tensor t ij The device on which the operator resides, and forward adjustment refers to adjusting the tensor t. ij The operator to which it belongs is placed in tensor t ij The device on which the pointed-to operator resides.

[0070] With tensor t ij Taking backward adjustment as an example, the operator o is calculated. j Placed in operator o i The total cost S of cross-device tensor transfer will be generated on the device in question. j Then compare the tensor t ij The total cost S that will be generated by the new placement method j The size of S j Greater than t ij And operator o i If the cost load of the device does not exceed the correction threshold, then the operator o will be... j Placed in operator o i On the device in question, otherwise, based on the above ideas, further adjustments are made to determine if the target tensor t... ij If the judgment of forward or backward adjustment is not met, then the original state shall be maintained.

[0071] In this embodiment, the correction threshold is a load balancing upper limit set to improve the adjustability of the model after initial partitioning. Its calculation method is shown in formula (6), where A represents the load threshold calculated by formula (5), and ε represents the magnitude coefficient of the correction threshold (ε≥1). It is usually based on experience. If the equipment load pressure is small, ε can be set larger to give the algorithm more adjustment space; if the equipment load pressure is large, it can be set smaller to control the equipment load balance and avoid OOM (out of memory) situation.

[0072] A * =A×ε#(6)

[0073] Finally, repeat the above process until the specified number of adjustments is reached or all tensors in the model no longer meet the conditions for forward and backward adjustments.

[0074] Adjust backwards as Figure 3 As shown in Figure (1), operator o1 is placed on device 1, and operators o2, o3, and o4 are placed on device 2. 12 This represents the selected cross-device transfer tensor from operator o1 on device 1 to operator o2 on device 2. If the load on device 1 does not reach the correction threshold and satisfies cost(t)... 12 cost(t) 23 )+cost(t 24 If ), then placing operator o2 on device 1 reduces the total cost required to transfer tensors across devices within the model. Similarly, the forward adjustment is as follows: Figure 3 As shown in (2), operators o1, o2, and o3 are placed on device 1, and operator o4 is placed on device 2. 34 This represents the selected cross-device transfer tensor from operator o3 on device 1 to operator o4 on device 2. If device 1 has not reached the correction threshold and satisfies cost(t)... 13 )+cost(t 23 cost(t) 34 If ), it means that placing operator o3 on device 2 can reduce the total cost required to transfer tensors across devices within the model.

[0075] Based on the above steps, the automatic parallel model method based on load balancing proposed in this embodiment is specifically described as shown in Algorithm 1:

[0076] In the first row, the generate_cost(G) method represents the communication cost, computation cost, and overall cost of generating all operators in the computation graph G based on the cost modeling method.

[0077] The second line indicates the calculation of the average load that will be allocated to each device.

[0078] Lines 3-7 describe the layer-by-layer partitioning method based on topological sorting in step 2, topo_order(G * ) indicates obtaining G * The topological sorting order is determined by the `is_placeable(node,d,cost_per_device)` method, which indicates whether placing the operator `node` on device `d` would exceed the average load size of the device, and `place(node,d)` indicates that the operator `node` is placed on device `d`.

[0079] Lines 8-27 describe the fine-grained model partitioning and scheduling scheme based on communication optimization in step 3, where MAX represents the maximum number of adjustments, and get_all_out_tensors(G * The method represents obtaining the current computation graph G. * The `sort_by_tensor_size(tensor_list)` method sorts the `tensor_list` collection by tensor size. Lines 16-27 describe the process of adjusting backwards and forwards. That is, if the size of the newly generated cross-device transfer tensor is smaller than the size of the current target tensor `target_tensor`, then the operators at both ends of the `target_tensor` tensor are placed on a computing device.

[0080]

[0081]

[0082] Another embodiment of this application illustrates a model-based automatic parallel device with load balancing; the device includes:

[0083] The cost calculation and evaluation module, based on the neural network model structure, computational attributes and other dimensions, quantitatively analyzes the key features affecting the execution performance of operators, and designs a scheme to build a performance evaluation model based on operator features in order to evaluate the cost of operators in the model and the training performance cost of the model.

[0084] The segmentation module, based on the structural characteristics of the neural network model, guided by cost and aiming at balancing the overall cost load of each device, uses topological sorting-based layer-by-layer partitioning to achieve coarse-grained segmentation of the neural network model, quickly dividing the neural network model into multiple sub-models with balanced performance and cost.

[0085] The optimal scheduling module designs a fine-grained model segmentation and scheduling scheme based on communication optimization according to the communication characteristics between operators. It adjusts the upper limit of device load by adjusting the correction threshold, and performs fine-grained adjustments on the obtained sub-models with the goal of reducing cross-device communication tensor transmission, so as to achieve global model optimal scheduling.

[0086] Embodiments of the device of the present invention can be applied to network devices. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the device loading corresponding computer program instructions from non-volatile memory into memory for execution, wherein the computer program is used to execute a load-balancing-based model-based automatic parallel method. From a hardware perspective, such as... Figure 4The diagram shown is a hardware structure diagram of the automatic parallel device for load balancing models according to the present invention. Except for... Figure 4 In addition to the processor, network interface, memory, and non-volatile memory shown, the device may typically include other hardware for hardware-level expansion. On the other hand, this application also provides a computer-readable storage medium storing a computer program that executes a model-based automatic parallel method for load balancing.

[0087] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and those skilled in the art can understand and implement them without creative effort.

[0088] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only.

[0089] It should also be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A model automatic parallelization method based on load balancing, characterized in that... The method includes the following steps: (1) Determine the operator features that affect the performance of the operator execution, and evaluate the cost of the operators in the model and the training performance cost of the model based on the operator features; The operator features include the operator's in-degree and out-degree magnitudes, the shapes of the input and output tensors, and the data types of the tensors; the cost of the operator includes computational cost, communication cost, and synthesis cost; The comprehensive cost of an operator refers to an index that comprehensively evaluates an operator by combining its structural characteristics, computational cost, and communication cost. Its calculation formula is as follows: ； in Indicates the cost of operator communication; This indicates the computational cost of the operator; This represents the total number of memory-intensive operators. This represents the total number of computationally intensive operators; It is a proportionality coefficient that represents the proportion of memory-intensive operators to the total number of memory-intensive and computation-intensive operators. The training performance cost is calculated from the computational cost of the operator and the size of the cross-device transmission tensor in the model, and the calculation formula is as follows: in This represents a distributed parallel strategy. Indicates that in a given computation graph And computing device diagram In this case, strategy Distributed training performance evaluation model, Indicates the number of operators in the model, This represents the number of tensors transferred between operators in the model; Indicates the first The computational cost of an operator Indicates the first The size of a tensor Determine whether the tensor needs to be transferred across devices; return 1 if it does, otherwise return 0. (2) Calculate the sum of the comprehensive costs of each operator in the whole model and divide it by the number of computing devices to obtain the average load that each computing device needs to bear, and use it as the load threshold of the device; Starting from the structural characteristics of the model, with the goal of reducing cross-device tensor transmission, the neural network model is traversed layer by layer according to the topology sorting order and placed on each computing device in turn until the computing device reaches the load threshold and then switches to the next idle computing device, so as to quickly divide the model into multiple sub-models with balanced comprehensive costs. (3) Based on the communication characteristics between operators, a fine-grained model segmentation and scheduling scheme based on communication optimization is implemented, specifically as follows: 1) Sort all tensors currently being transmitted across devices and extract the tensor with the highest transmission cost; 2) Based on the load threshold, set a load correction threshold. Based on the transmission tensor with the highest cost and the correction threshold, adjust the scheduling information of local operators in the sub-model to reduce the size of the tensor transmitted across devices and the performance cost of model training. 3) Repeat the above process until the specified number of adjustments is reached or all tensors in the model no longer meet the conditions for forward and backward adjustments; The backward adjustment refers to placing the operator pointed to by the tensor onto the device where the tensor belongs to the operator. The forward adjustment refers to placing the operator to which the tensor belongs onto the device where the operator pointed to by the tensor is located; The forward adjustment condition is that the device load does not reach the correction threshold and the tensor transmission cost is greater than the total cross-device tensor transmission cost newly generated after operator migration. The backward adjustment condition is that the target device load does not reach the correction threshold and the cross-device tensor transmission cost is greater than the total cross-device tensor transmission cost newly generated after operator migration.

2. A model-based automatic parallel device based on load balancing, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the automatic parallel model method based on load balancing as described in claim 1.

3. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the automatic parallel model method based on load balancing as described in claim 1.

Images