Automatic Parallel Policy Search Method, Media, and Device Based on Network-Level Simulation

Abstract

This invention provides an automatic parallel strategy search method, medium, and device based on network-level simulation. The method includes: acquiring a neural network computation graph, and splitting an initial network model based on the neural network computation graph to obtain multiple node sub-network models, forming a small cluster network model; acquiring the time cost of each node sub-network model; simulating the time cost of a large cluster network model based on the time cost of each node sub-network model; and performing parallel strategy search based on the time cost of the simulated large cluster network model to output the optimal parallel strategy. This invention not only supports simulation in both model parallel and pipeline parallel scenarios, covering all existing parallel training scenarios, but also improves the accuracy and speed of simulation through a performance evaluation scheme for whole-network splitting.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to the field of neural network model technology. Background Technology

[0002] With the continuous development of deep learning, neural network models are being optimized for larger parameter counts. Large-scale pre-trained language models have become a new paradigm in natural language processing. Language models with hundreds of billions of parameters, such as GPT-3, have demonstrated powerful performance in natural language understanding and generation, and have been proven to achieve excellent performance in various natural language tasks. However, training large models is difficult to implement on a single GPU; therefore, distributed training is the core of large model training systems. As the scale of models increases, various deep learning frameworks such as PyTorch and TensorFlow have begun to incorporate distributed training capabilities.

[0003] Distributed training faces the problem of low training efficiency. Existing distributed training strategies mainly include data-parallel training, model-parallel training, pipelined parallel training, and optimizer parallel training. Data parallelism expands the training scale on sample data, thus improving training efficiency. Other parallel training modes, besides data parallelism, aim to separate the training parameters (model parameters, optimizer parameters) from the model training parameters, allowing a portion of the training to be completed on a single GPU, thereby enabling distributed training of large models, but without improving training efficiency. Therefore, combining multiple parallel strategies to maximize training efficiency while ensuring cluster training capabilities is a crucial issue in distributed training.

[0004] Due to the large number of parallel strategy combinations and the time-consuming and labor-intensive nature of manual search, some automated parallel strategy search methods have been proposed in recent years. However, existing automated parallel strategy search schemes are based on empirical models for search and verification, resulting in low accuracy. Performance testing on large real-world models is also quite challenging in practice. Therefore, a precise and lightweight parallel strategy verification scheme is needed.

[0005] Existing automatic parallel policy search techniques lack a precise and efficient scheme for verifying the performance of parallel policies. One existing scheme simulates parallel distributed policies at the operator level, employing operator performance evaluation, computational graph construction, and computational graph simulation. It also modifies and replaces the computational graph based on various existing optimization schemes. Another scheme simulates parallel policies at the operator level, creating a dependency between computation and data transmission in a data-parallel scenario. This approach has several problems. First, this simulation scheme only supports simulations in data-parallel scenarios; second, the optimization schemes it supports only intra-layer operator optimization; third, because the operator level is too low, the performance evaluation process is too cumbersome and time-consuming. Finally, performance evaluation may require performing it on the entire network, resulting in significant time overhead.

[0006] Another approach in existing technologies transforms the automatic parallel policy search problem into an integer linear programming problem. By constructing a cost model and solution space, the merits of parallel policies are evaluated and verified, and a genetic algorithm is used to search for and optimize the parallel policies. The main challenge of this approach is the accuracy of the cost model. While parallel policy verification schemes based on empirical models such as cost models offer the advantage of lightweight operation, their accuracy is relatively low. Furthermore, when computing hardware and network topology change, the parallel training strategies obtained through cost model search cannot guarantee the original training efficiency. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an automatic parallel policy search method, medium and device based on network-level simulation, to solve the technical problems that the prior art cannot support training scenarios with model parallelism and pipeline parallelism, and that the simulation overhead is large.

[0008] To achieve the above and other related objectives, this invention provides an automatic parallel strategy search method based on network-level simulation. The method includes: acquiring a neural network computation graph, and splitting an initial network model based on the neural network computation graph to obtain multiple node sub-network models, forming a small cluster network model; acquiring the time cost of each node sub-network model; simulating the time cost of a large cluster network model based on the time cost of each node sub-network model; performing parallel strategy search based on the time cost of the simulated large cluster network model, and outputting the optimal parallel strategy.

[0009] In one embodiment of the present invention, the splitting of the initial network model based on the neural network computation graph includes a model splitting method in a model parallel mode, comprising: splitting the weights of each neural network layer of the initial network model into multiple parts, assigning each part to a device to perform computation on the same input, and after the computation is completed, summarizing the outputs into a complete output through communication.

[0010] In one embodiment of the present invention, the splitting of the initial network model based on the neural network computation graph includes a pipelined parallel mode model splitting method, which includes: splitting the initial network model into several cascaded node sub-network models, with each device computing one node sub-network model, which includes one or more computation layers, and the devices completing the whole network computation in a relay manner.

[0011] In one embodiment of the present invention, the step of splitting the initial network model based on the neural network computation graph includes: determining a parallel strategy and determining the degree of parallelism of the model parallel mode and the pipeline parallel mode under the parallel strategy; performing intra-layer partitioning of the operators in the initial network model based on the model parallel mode to obtain operators adapted to model parallelism; and dividing the initial network model into several cascaded node sub-network models based on the pipeline parallel mode and each of the operators.

[0012] In one embodiment of the present invention, the time overhead includes: computation time overhead and communication time overhead; the computation time overhead includes one or more of forward computation time, backward computation time, and parameter update time; the communication time overhead includes at least computational value transmission time, computational value summarization time, and model gradient summarization time.

[0013] In one embodiment of the present invention, the method further includes: simulating model parallelism and pipeline parallelism in the forward and backward phases, and simulating data parallelism in the parameter update phase.

[0014] In one embodiment of the present invention, the parallel strategy is searched using a grid search method, and the search content is the pipeline parallelism, model parallelism and data parallelism determined by the cluster size.

[0015] In one embodiment of the present invention, the parallel strategy search based on the time cost of the simulated large cluster network model and the output of the optimal parallel strategy includes: inputting the cluster size, model partitioning and time cost data of the large cluster network model; simulating the time cost of parallel strategies in turn, and determining the parallel strategy with the lowest time cost as the optimal parallel strategy.

[0016] To achieve the above and other related objectives, the present invention also provides a computer storage medium storing program instructions, which, when executed, implement the steps of the automatic parallel strategy search method based on network-level simulation as described above.

[0017] To achieve the above and other related objectives, the present invention also provides an electronic device, including a memory for storing a computer program; and a processor for running the computer program to implement the steps of the automatic parallel strategy search method based on network-level simulation as described above.

[0018] As described above, the automatic parallel policy search method, medium, and device based on network-level simulation of the present invention have the following beneficial effects:

[0019] This invention not only supports simulation in both model-parallel and pipeline-parallel scenarios, covering all existing parallel training scenarios, but also improves the accuracy and speed of simulation through a performance evaluation scheme for whole-network splitting. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 The diagram shows the overall flow of an automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0022] Figure 2 The diagram shows the principle flow of an automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0023] Figure 3 The diagram shows the overall flowchart of the network-level parallel policy simulation verification scheme in the automatic parallel policy search method based on network-level simulation in one embodiment of this application.

[0024] Figure 4 The diagram shown is a schematic representation of model splitting in an automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0025] Figure 5 The diagram shows the principle of the forward priority pipeline algorithm in the automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0026] Figure 6 The diagram shows the principle of the reverse priority pipeline algorithm in the automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0027] Figure 7 The diagram shown is an overall modeling flowchart of an automatic parallel policy search method based on network-level simulation in one embodiment of this application.

[0028] Figure 8 The diagram shown is a flowchart of pipeline parallel modeling in an automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0029] Figure 9 The diagram shown is an automatic parallel strategy search flowchart in an automatic parallel strategy search method based on network-level simulation in one embodiment of this application.

[0030] Figure 10 The diagram shown is a schematic block diagram of an electronic device according to an embodiment of this application.

[0031] Component designation explanation

[0032] 101 Electronic Devices

[0033] 1001 processor

[0034] 1002 Memory

[0035] S100~S400 Steps Detailed Implementation

[0036] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0037] The purpose of this embodiment is to provide an automatic parallel strategy search method, medium, and device based on network-level simulation, which solves the technical problem that the prior art cannot effectively diagnose battery performance quickly.

[0038] To ensure accurate analysis results from the network-level simulation-based automatic parallel policy search method in this embodiment, it is necessary to sample the time overhead of the neural network model. This embodiment employs sub-network segmentation to process the time overhead data of small cluster networks, and finally accumulates the sub-network time overhead data to simulate the time overhead data of a large cluster network. The following will detail the principles and implementation methods of the network-level simulation-based automatic parallel policy search method, medium, and device of this invention, enabling those skilled in the art to understand the invention without creative effort.

[0039] Example 1

[0040] This embodiment provides an automatic parallel policy search method based on network-level simulation, specifically, as follows: Figure 1 As shown, the automatic parallel policy search method based on network-level simulation described in this embodiment includes the following steps:

[0041] Step S100: Obtain the neural network computation graph, and split the initial network model based on the neural network computation graph to obtain multiple node sub-network models, forming a small cluster network model;

[0042] Step S200: Obtain the time cost of each node sub-network model;

[0043] Step S300: Simulate the time cost of the large cluster network model based on the time cost of each node sub-network model;

[0044] Step S400: Based on the time cost of the simulated large cluster network model, perform a parallel strategy search and output the optimal parallel strategy.

[0045] like Figure 2 As shown, in this embodiment, after the user defines the neural network, the solution acquires it and constructs it into a neural network computation graph, including operator information and computational dependency information. After constructing the computation graph, the invention begins the simulation and verification of the network-level parallel strategy. First, the model is split and trained on a small-scale distributed cluster to collect network-level time cost information. The collected network-level time cost information can be used to simulate the time cost on a large cluster. Next, the invention performs automatic strategy search, using a grid search method to search for parallel strategies and simulate the time cost on a large cluster. Finally, the optimal strategy obtained from the search is selected, and the neural network training framework is called for training.

[0046] The overall process of the network-level parallel strategy simulation and verification scheme is as follows: Figure 3 As shown, the execution flow is divided into the following three processes:

[0047] A) Model Splitting: Given an initial model, this approach equally splits the model into subnetworks that are computed on each node. Model splitting occurs when model parallelism and pipeline parallelism coexist, therefore this approach requires model splitting.

[0048] B) Time Cost Sampling: To ensure accurate simulation, the computation and communication time of each node needs to be tested and sampled. This scheme uses a network-level sampling granularity that includes both computation and communication (network-level granularity is higher than neural network layer-level granularity). Time cost sampling is performed on small clusters with a limited number of nodes, with each node computing its corresponding sub-network model. In practice, the models trained on small clusters are relatively small.

[0049] C) Time-axis modeling: Based on certain parallel computing rules that conform to reality, and combined with the computational data obtained above, the training process of the large model on a large-scale cluster is derived, thereby completing the process of simulating the parallel scheme. Training is performed on a large cluster with a large number of nodes to simulate the time cost data of the parallel strategy of the large model.

[0050] Therefore, in this embodiment, the core of the entire simulation verification scheme lies in three parts: model splitting, time cost sampling, and timeline simulation. The flow of these three parts determines the accuracy with real-world conditions and whether it is possible to simulate large model training scenarios on small clusters. Therefore, the following sections of this embodiment will describe the flow of these three parts.

[0051] The steps S100 to S400 of the automatic parallel policy search method based on network-level simulation in this embodiment will be described in detail below.

[0052] Step S100: Obtain the neural network computation graph, and split the initial network model based on the neural network computation graph to obtain multiple node sub-network models, forming a small cluster network model.

[0053] Model decomposition is a process of breaking down a large model into subnets that each node can execute, to accommodate both model parallelism and pipelined parallelism in simulation and verification. The following are two parallel model decomposition methods: model parallelism and pipelined parallelism.

[0054] In this embodiment, the splitting of the initial network model based on the neural network computation graph includes a model splitting method in a model parallel mode, which includes: splitting the weights of each neural network layer of the initial network model into multiple parts, assigning each part to a device to perform calculations on the same input, and summarizing the outputs into a complete output through communication after the calculations are completed.

[0055] In other words, the principle of model parallelism is to split the weights of each neural network layer into multiple parts, assign each part to a device to compute on the same input, and then aggregate the outputs into a complete output through communication after the computation is complete. Therefore, the corresponding splitting method is to equally divide the parameters within the layer.

[0056] In this embodiment, the splitting of the initial network model based on the neural network computation graph includes a pipelined parallel mode model splitting method, which includes: splitting the initial network model into several cascaded node sub-network models, with each device computing one node sub-network model, which contains one or more computation layers, and the devices completing the whole network computation in a relay manner.

[0057] In other words, the principle of pipeline parallelism is to break down the entire network into several cascaded sub-models. Each device computes a sub-model, which contains one or more computation layers. The devices complete the network computation in a relay manner. Therefore, the corresponding decomposition method is to divide the entire network into several sub-networks with similar workloads based on the amount of workload.

[0058] Specifically, in this embodiment, the step of splitting the initial network model based on the neural network computation graph includes:

[0059] 1) Determine the parallel strategy, and determine the degree of parallelism of the model parallel mode and the pipeline parallel mode under the parallel strategy;

[0060] 2) Based on the model parallel mode, the operators in the initial network model are partitioned within layers to obtain operators adapted to model parallelism;

[0061] 3) Based on the pipeline parallel mode and each of the operators, the initial network model is divided into several cascaded node sub-network models.

[0062] like Figure 4 The diagram illustrates the process of model partitioning and sub-model generation. The main process is as follows: Given a model description, an instance parallelism mode is selected that includes data parallelism, model parallelism, and pipeline parallelism (e.g., in a small cluster of 8 nodes, data parallelism is 2, model parallelism is 2, and pipeline parallelism is 2). Based on the model parallelism method, the operators in the model are partitioned within layers to obtain operators suitable for model parallelism. Then, based on the pipeline parallelism, a pipeline partitioning strategy obtained from the distributed framework (the default is a uniform partitioning strategy) is used to construct the partitioned sub-network containing model parallelism operators, thus obtaining the actual sub-model running on each node.

[0063] Step S200: Obtain the time cost of each node sub-network model.

[0064] By sampling time overhead, the actual computation and communication overhead in the sub-model can be obtained.

[0065] In this embodiment, the time overhead includes: computation time overhead and communication time overhead; the computation time overhead includes one or more of forward computation time, backward computation time, and parameter update time; the communication time overhead includes at least computational value transmission time, computational value summarization time, and model gradient summarization time.

[0066] 1) Calculate the time cost:

[0067] 1-1) Forward computation time: The forward computation time of each neural network layer.

[0068] 1-2) Backward computation time: Backward computation time for each neural network layer.

[0069] 1-3) Parameter update time: The time required to update parameters based on gradients.

[0070] 2) Communication time overhead:

[0071] 2-1) Computational value transfer time: In pipeline parallelism, assuming that two pipeline stages are on different nodes, the communication overhead of transferring the calculated value from the previous pipeline stage to the next pipeline stage as input.

[0072] 2-2) Computational summarization time: In model parallelism, the communication overhead of summarizing partial computational values ​​into complete computational values.

[0073] 2-3) Model gradient summarization time: In data parallelism, the communication overhead of summarizing gradients between different nodes after backpropagation is completed.

[0074] 2-4) Other communication overhead: Depending on the task of different models, there may be additional communication overhead (e.g., communication overhead for additional numerical synchronization calculations).

[0075] The aforementioned time can be obtained through a single complete training process (subgraph training). Alternatively, a more general time complexity can be obtained by averaging multiple runs.

[0076] Step S300: Simulate the time cost of the large cluster network model based on the time cost of each node sub-network model.

[0077] In this embodiment, it further includes: simulating model parallelism and pipeline parallelism in the forward and backward phases, and simulating data parallelism in the parameter update phase.

[0078] By sampling the time overhead, a timeline can be established for subsequent analysis. To build a timeline that conforms to the specifications for large-scale cluster and large model training, and to ensure that the parallel mode conforms to industry computing workflows, this scheme defines the following two pipelined parallel algorithms as options: forward-first and backward-first.

[0079] Forward-first approach: First, complete the forward computation of all data, then perform the backward computation of all data. For example... Figure 5 The forward-first pipeline algorithm divides the data into four micro-batches, numbered 0-3; the pipeline size is 4, and the computation nodes are numbered P0-P3. All computation nodes complete the forward computation of all data before waiting for the subsequent nodes to complete the reverse computation.

[0080] Reverse-first approach: Define a warm-up phase where devices at earlier stages in the pipeline perform forward calculations for several stages first. After stabilization, perform one reverse and one forward (1F1B) calculation. For example... Figure 6The reverse-first pipelined algorithm shown divides the data into four micro-batches, numbered 0-3; the pipeline size is 4, and the computation nodes are numbered P0-P3. Node P0, at pipeline stage 0, completes the computation of all four micro-batches. As the pipeline stages increase, the number of forward computations performed by each node during the warm-up stage decreases. Node P3, at the end of the pipeline stages, completes the computation of only one micro-batch before entering the 1F1B stage. After all forward computations at this node are completed, the reverse computation of the remaining micro-batches is performed.

[0081] like Figure 7 As shown, for a complete iteration of model training (including forward, backward, and parameter update), this scheme first simulates model parallelism and pipeline parallelism in the forward and backward stages, and simulates data parallelism in the parameter update stage. The simulation is performed on each computation node. First, the forward stage (for forward-first algorithms) or warm-up stage (for backward-first algorithms) is established based on a forward-first or backward-first algorithm, incorporating computation and activation value communication overhead using a resilient algorithm. In the parameter update stage, this scheme adds data parallel gradient aggregation overhead, gradient update time, and other time overhead.

[0082] The timeline modeling flowchart for pipeline parallelism is as follows: Figure 8 As shown, it elaborates on the modeling details of forward computation, backward computation, and parameter updates according to the type of pipelined algorithm. Simulation of pipelined parallelism requires the time for transmitting computational values. It is worth noting that in pipelined parallelism, not all computation nodes need to transmit parameters. The decision of "whether to transmit activation values / gradients" depends on the following conditions: whether the pipeline stage is the first or last stage, and whether the target computation node is on the same machine. Regardless of whether a forward-first or backward-first algorithm is used, this flowchart strictly follows the prescribed rules and the algorithm itself to add all computations and communications to the timeline.

[0083] The parallel simulation of a single micro-batch forward and backward computation will utilize the time spent on both forward and backward computation and the time spent summarizing the calculated values. Furthermore, the collected data will be scaled according to the parallel scale of the simulation model to capture realistic data.

[0084] Since data parallelism is independent of training itself, it only adds a parameter all-reduce phase before gradient updates. For data parallelism, the computational logic performed by the devices that need to perform all-reduce is exactly the same in this embodiment. Therefore, the simulation of data parallelism only needs to insert the parameter summarization time.

[0085] Step S400: Based on the time cost of the simulated large cluster network model, perform a parallel strategy search and output the optimal parallel strategy.

[0086] In this embodiment, the parallel strategy is searched using a grid search method, and the search content is the pipeline parallelism, model parallelism, and data parallelism determined by the cluster size.

[0087] In this embodiment, the automatic parallel strategy is searched using a grid search method, and the search criteria are the pipeline parallelism, model parallelism, and data parallelism, which are determined by the cluster size. Because the search space is small, grid search can achieve high efficiency.

[0088] Figure 9 The search scheme of the automatic parallel strategy is demonstrated. Specifically, such as... Figure 9 As shown, in this embodiment, the parallel strategy search based on the time cost of the simulated large cluster network model, and the output of the optimal parallel strategy, includes:

[0089] 1) Input the cluster size, model partitioning, and time cost data for the large cluster network model;

[0090] 2) Simulate the time cost of parallel strategies in sequence, and determine the parallel strategy with the lowest time cost as the optimal parallel strategy.

[0091] In this embodiment, the interaction between the user and the software layer includes:

[0092] a) The user specifies the model scheme that the system needs to execute;

[0093] b) The user inputs the size of the training cluster;

[0094] c) The user confirms the parallel strategy obtained from the search.

[0095] The system operation logic in this embodiment includes:

[0096] a) Based on the model defined by the user, the model is split and the time overhead is collected on small clusters;

[0097] b) Based on the user's determination of the training cluster size and time overhead, search for the optimal cluster parallel training strategy.

[0098] This embodiment utilizes a parallel strategy search scheme based on network-level simulation, which offers high simulation accuracy, low cost, and yields better results from the searched parallel strategy scheme.

[0099] Example 2

[0100] like Figure 10As shown, this embodiment provides an electronic device 101, which includes a processor 1001 and a memory 1002. The memory 1002 is used to store computer programs. The processor 1001 is used to execute the computer programs stored in the memory 1002 to enable the electronic device 101 to perform the steps of the automatic parallel strategy search method based on network-level simulation as described in Embodiment 1.

[0101] Processor 1001 is used for executing the automatic parallel policy search program and training the model. Memory 1002 is used for storing the automatic parallel policy search program, the neural network model, and related parameters.

[0102] Furthermore, in this embodiment, the electronic device 101 also includes input / output devices for data transmission with the processor via a bus. The input / output devices are used for interaction with the user.

[0103] Since the specific implementation process of the automatic parallel policy search method based on network-level simulation has been described in detail in Example 1, it will not be repeated here.

[0104] Processor 1001 is a Central Processing Unit (CPU). Memory 1002 is connected to processor 1001 via a system bus and communicates with it. Memory 1002 stores computer programs, and processor 1001 runs the computer programs to execute the network-level simulation-based automatic parallel strategy search method. Memory 1002 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0105] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program thereon. When executed by the processor 1001, the computer program implements the steps of the automatic parallel policy search method based on network-level simulation described in Embodiment 1. Embodiment 1 has already provided a detailed description of the automatic parallel policy search method based on network-level simulation, and will not be repeated here.

[0106] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0107] In summary, this invention not only supports simulations in both model-parallel and pipeline-parallel scenarios, covering all existing parallel training scenarios, but also improves the accuracy and speed of simulation through its whole-network split performance evaluation scheme. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0108] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for automatic parallel strategy search based on network-level simulation, characterized in that: The method includes: Obtain the neural network computation graph, and based on the neural network computation graph, split the initial network model to obtain multiple node sub-network models, forming a small cluster network model; Obtain the time cost of each node sub-network model; the time cost includes: computation time cost and communication time cost; the computation time cost includes one or more of forward computation time, backward computation time, and parameter update time; the communication time cost includes at least computational value transmission time, computational value summarization time, and model gradient summarization time. Simulate the time cost of a large cluster network model based on the time cost of each node sub-network model; Parallel strategy search is performed based on the time cost of a simulated large cluster network model, and the optimal parallel strategy is output. It also includes: simulating model parallelism and pipeline parallelism in the forward and backward phases, and simulating data parallelism in the parameter update phase; The process of splitting the initial network model based on the neural network computation graph includes: Determine the parallel strategy, and determine the degree of parallelism of the model parallel mode and the pipeline parallel mode under the parallel strategy; Based on the model parallel mode, the operators in the initial network model are partitioned within layers to obtain operators adapted to model parallelism; Based on the pipeline parallel mode and each of the operators, the initial network model is divided into several cascaded node sub-network models; The parallel strategy is searched using a grid search method, and the search results are for pipeline parallelism, model parallelism, and data parallelism, which are determined by the cluster size. The parallel strategy search based on the time cost of the simulated large cluster network model and the output of the optimal parallel strategy include: inputting the cluster size, model partitioning and time cost data of the large cluster network model; simulating the time cost of parallel strategies in turn, and determining the parallel strategy with the lowest time cost as the optimal parallel strategy.

2. The automatic parallel policy search method based on network-level simulation according to claim 1, characterized in that: The splitting of the initial network model based on the neural network computation graph includes a model splitting method in a model parallel mode, including: The weights of each neural network layer in the initial network model are split into multiple parts, and each part is assigned to a device to calculate the same input. After the calculation is completed, the outputs are aggregated into a complete output through communication.

3. The automatic parallel policy search method based on network-level simulation according to claim 1 or 2, characterized in that: The splitting of the initial network model based on the neural network computation graph includes a pipelined parallel model splitting method, including: The initial network model is split into several cascaded node sub-network models. Each device computes a node sub-network model, which contains one or more computing layers. The devices complete the network computation in a relay manner.

4. A computer storage medium storing program instructions, characterized in that: When the program instructions are executed, they implement the steps of the automatic parallel strategy search method based on network-level simulation as described in any one of claims 1 to 3.

5. An electronic device, characterized in that: It includes a memory for storing a computer program; and a processor for running the computer program to implement the steps of the automatic parallel policy search method based on network-level simulation as described in any one of claims 1 to 3.

Images