Method, device, medium and program product for automatic tuning of distributed parallel strategy

Abstract

The application provides an automatic tuning method, device, medium and program product of a distributed parallel strategy, which comprises the following steps: constructing a search space according to model information and cluster information through a searcher; performing a pruning operation based on a video memory model and historical tuning results through a pruner, and calculating a pruning rate; if the pruning rate is greater than or equal to a set pruning rate threshold, generating an executable configuration script through a generator; performing performance estimation on the executable configuration script through an estimator, and outputting estimated performance information; recording the performance information through a recorder, and sending the performance information to the pruner to perform a traversal operation; after the traversal operation of the search space is completed, comparing all executable configuration scripts according to the results of the traversal operation, obtaining an optimal distributed training configuration script, and automatically executing and starting large-scale distributed training. The technical scheme of the application realizes rapid acquisition of an efficient large model training configuration, thereby saving training costs.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to an automatic optimization method, device, medium and program product for distributed parallel strategies. Background Technology

[0002] In recent years, with the advent of large-scale models, single-GPU memory has become completely insufficient to meet the training requirements, making distributed parallel training the only solution for large-scale model training. As distributed parallel training has received increasing attention from industry and academia, numerous parallel strategies and optimization techniques have emerged, continuously improving model training performance.

[0003] The main parallel and optimization strategies employed in related technologies include data parallelism, tensor parallelism, pipeline parallelism, and context parallelism. Since parallel and optimization strategies can be combined, the performance differences between different distributed parallel training configurations are huge. Therefore, it is crucial to quickly obtain efficient training configurations to save training costs.

[0004] The main drawbacks of the related technologies are:

[0005] First, the expert-configured training strategy lacks universality. When the type of machine used for model training changes, the original configuration training strategy is usually not the optimal strategy, or even cannot be used. When the number of machines used for model training changes, the original configuration training strategy will also face the above problems.

[0006] Second, expert experience is not globally optimal. It is difficult to achieve optimal performance by manually designing a limited number of experiments based on expert experience to obtain expert configuration training strategies.

[0007] Third, since parallel strategies and optimization strategies can be combined, the resulting training strategy configuration search space is huge. Using brute force search results in a long time to determine the optimal strategy, and requires complete training resources, making it highly dependent. Summary of the Invention

[0008] The purpose of this application is to provide an automatic tuning method, device, medium, and program product for distributed parallel strategies, which aims to quickly obtain efficient large model training configurations, thereby saving training costs.

[0009] According to a first aspect of this application, an automatic tuning method for a distributed parallel strategy is provided, comprising:

[0010] The search space is constructed using a search engine based on model and cluster information;

[0011] The pruning operation is performed using a pruning tool based on the memory model and historical optimization results, and the pruning rate is calculated.

[0012] If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script will be generated by the generator.

[0013] The executable configuration script is input into the estimator, which performs a performance prediction on the executable configuration script and outputs the predicted performance information.

[0014] The performance information output by the predictor is recorded by the recorder, and the recorded performance information is sent to the pruner to perform a traversal operation.

[0015] After the traversal operation of the search space is completed, the executor compares all runnable configuration scripts based on the results of the traversal operation to obtain the optimal distributed training configuration script.

[0016] The optimal distributed training configuration script is executed automatically to start large-scale distributed training.

[0017] In an optional implementation, the executable configuration script is input into a predictor, which performs a performance prediction on the executable configuration script and outputs the predicted performance information, including:

[0018] A process is launched using a single-process simulator to simulate the execution process of the process in distributed training.

[0019] Calculate the overhead of a single card during execution;

[0020] Based on the overhead of the single card, estimate the performance information of the executable configuration script and output the estimated performance information.

[0021] In an optional implementation, a process is started using a single-process simulator to simulate the execution process of the process in distributed training, including:

[0022] The single-process simulator starts a process through a custom communication backend;

[0023] Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated.

[0024] During the parallel scheduling of the distributed training pipeline, the pipeline is analyzed using the single-process simulator to obtain the analysis results.

[0025] Based on the analysis results, it was confirmed that the process in question requires different training tasks compared to other processes.

[0026] Based on the different training tasks that the process and other processes need to perform, determine the critical path that determines the training time of the process and other processes;

[0027] The simulation process described above is invoked for the training tasks on the critical path.

[0028] In an optional implementation, the method further includes:

[0029] If the pruning rate is less than the set pruning rate threshold, then continue to retrieve new configuration scripts from the search space;

[0030] The new configuration script is input into the pruner, and the pruning operation continues.

[0031] In an optional implementation, pruning is performed using a pruner based on the memory model and historical tuning results, and the pruning rate is calculated, including:

[0032] A memory model is established based on the activation tensor and the sum of parameters;

[0033] If the video memory of the configuration script obtained by the video memory model is greater than the video memory of a single hardware card, then the configuration script will be pruned.

[0034] In an optional implementation, the method further includes:

[0035] If the historical video memory of a historical configuration script is less than that of the current configuration script, and the video memory of the current configuration script is greater than that of a single hardware card, then the current configuration script will be pruned directly.

[0036] If the historical configuration scripts in the historical data are different from the current configuration scripts only in the recalculation dimension, and the configuration scripts with fewer recalculations are already runnable, then the configuration scripts with more recalculations will be pruned.

[0037] According to a second aspect of this application, an automatic tuning device for a distributed parallel strategy is provided, comprising:

[0038] The building unit is configured to construct the search space based on model information and cluster information using a searcher;

[0039] The pruning unit is configured to perform pruning operations based on the memory model and historical tuning results through a pruner, and to calculate the pruning rate;

[0040] The generation unit is configured to generate an executable configuration script by the generator if the pruning rate is greater than or equal to a set pruning rate threshold.

[0041] The performance prediction unit is configured to input the executable configuration script into the predictor, perform performance prediction on the executable configuration script through the predictor, and output the predicted performance information.

[0042] The traversal unit is configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform the traversal operation;

[0043] The comparison unit is configured to, after the traversal operation of the search space is completed, compare all runnable configuration scripts with the result of the traversal operation through the executor to obtain the optimal distributed training configuration script.

[0044] The execution unit is configured to automatically execute the optimal distributed training configuration script to initiate large-scale distributed training.

[0045] Compared with related technologies, the technical solution of this application has the following advantages:

[0046] This application employs an automated approach. A searcher constructs a search space based on model and cluster information, performing a search across the entire high-dimensional vector space to obtain the optimal parallel model configuration. A pruning agent performs pruning operations based on the GPU-in-memory model and historical tuning results, thereby improving pruning efficiency and resolving the time-consuming search caused by the large search space. The solution uses a single-process simulator as a predictor to estimate the performance of the executable configuration script. This allows for performance prediction of the entire cluster on a single GPU, saving training resources and time required for automatic tuning. Furthermore, it enables concurrent searching across the entire high-dimensional vector space, addressing the issue of requiring complete training resources for the search process. This achieves performance improvements and saves training costs under large-scale model and cluster training.

[0047] This application addresses the lack of universality in expert experience, making it applicable to any cluster size and model size. Its automated tuning mechanism takes cluster and model information as input and automatically executes the tuning process based on this input, independent of expert experience, thus solving the problem of suboptimal expert experience and achieving the best parallel training configuration. This application constructs a complete search space and performs a complete traversal of this search space using efficient pruning methods. Therefore, the configuration script obtained by this application is the optimal distributed training configuration script, solving the problem of long search time and making automatic tuning highly efficient. The single-process simulator proposed in this solution can perform complete training resource tuning using only a single GPU, greatly saving search resources. By further dividing the search space and using multiple single GPUs for concurrent searching, search efficiency is further improved.

[0048] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures and processes shown in the description and the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an automatic optimization method for a distributed parallel strategy provided in an embodiment of this disclosure.

[0051] Figure 2 This is a flowchart illustrating another automatic tuning method for a distributed parallel strategy provided in this embodiment of the disclosure;

[0052] Figure 3 This is a schematic diagram of a single-process simulator provided in an embodiment of this disclosure;

[0053] Figure 4 This is a schematic diagram of the structure of an automatic tuning device for a distributed parallel strategy provided in an embodiment of this disclosure. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] See Figure 1 The flowchart provided in this application illustrates an automatic tuning method for a distributed parallel strategy, comprising the following steps:

[0056] S101. Construct the search space based on model information and cluster information using a search engine.

[0057] In an optional implementation, the search space configuration parameters for the constructed search space are as follows:

[0058] a) Data Parallel Size: The value of data parallelism is an integer, which does not exceed the total number of cards in a single hardware card, and the value of data parallelism is divisible by the global batch size.

[0059] b) Tensor Parallel Size: The value of tensor parallelism is an integer. The value of tensor parallelism does not exceed the total number of hardware cards in a single machine. The value of tensor parallelism is divisible by the relevant parameters of the memory model. For example, the value of tensor parallelism is divisible by the number of hidden layers in the memory model.

[0060] c) Pipeline Parallel Size: The value of pipeline parallelism is an integer. The value of pipeline parallelism does not exceed the total number of cards in a single hardware card, and the value of pipeline parallelism is divisible by the number of hidden layers in the memory model.

[0061] d) Expert Parallel Size: The value of the expert parallel size is an integer, and the value of the expert parallel size does not exceed the number of experts.

[0062] e) Context Parallel Size: The context parallel size is an integer, and the value of the context parallel size does not exceed the total number of cards in a single hardware card, and the value of the context parallel size is divisible by the sequence length.

[0063] f) Distributed Optimizer: The value of the distributed optimizer is a boolean value. The distributed optimizer is used to optimize the parallelism of data. The distributed optimizer is effective when the parallelism of data is greater than 1.

[0064] g) Sequence Parallelism: The value of sequence parallelism is a Boolean value. Sequence parallelism is used to optimize tensor parallelism. Sequence parallelism is effective when tensor parallelism is greater than 1.

[0065] h) Recalculation method: The recalculation method can be ["block", "uniform"], where "block" means only the preceding layers are recalculated, and "uniform" means that the hidden layers of the memory model are grouped before recalculation.

[0066] For example, the transformer model is composed of stacked identical layers. The transformer model has 28 layers. The recompile method is set to "block" and the number of recompile layers is set to 5. This means that the first 5 layers of the transformer model are recomputed, and the last 23 layers of the transformer model do not need to be recomputed.

[0067] i) Recomputation granularity: The recomputation granularity can be ["full", "selective"], where "full" means recomputing the entire transformer layer, and "selective" means recomputing only the attention part of the memory model.

[0068] j) Number of recomputed layers: The number of recomputed layers is an integer, and its value does not exceed the total number of hidden layers in the memory model, nor does it exceed the total number of layers allocated under pipeline parallelism.

[0069] This application addresses the lack of universality in expert experience, making it applicable to any cluster size and any model size. The automated tuning mechanism of this application takes cluster information and model information as input and can automatically execute the automatic tuning process based on this input, without relying on expert experience. This solves the problem of non-optimal expert experience and can obtain the best parallel training configuration.

[0070] S102. Perform pruning operations based on the memory model and historical optimization results using the pruning tool, and calculate the pruning rate.

[0071] like Figure 2 The diagram illustrates a flowchart of another automatic optimization method for a distributed parallel strategy provided in this embodiment. This embodiment performs pruning operations based on a memory model and historical tuning results using a pruner. If the pruning rate is greater than or equal to a set pruning rate threshold, an executable configuration script is generated by a generator, and the pruner continues to the next step. If the pruning rate is less than the set pruning rate threshold, a new configuration script is retrieved from the search space. A memory model is established based on the activation tensor and the sum of parameters. If the memory size of the current configuration script obtained by the memory model is greater than the memory size of a single hardware card, the current configuration script is pruned.

[0072] Optionally, if historical configuration scripts exist in the historical data, a comparison operation is performed. The comparison principle is as follows: if the historical video memory of the historical configuration script in the historical data is less than that of the current configuration script, and the video memory of the current configuration script is greater than the video memory of a single hardware card (i.e., OOM memory overflow), then the current configuration script can be pruned directly. In terms of recalculation, if the historical configuration scripts in the historical data are identical to the current configuration script except for the recalculation dimension, and the configurations with fewer recalculations are already runnable, then the configurations with more recalculations are pruned.

[0073] S103. If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script is generated by the generator.

[0074] This application constructs a complete search space and performs a complete traversal of the search space through an efficient pruning method. Therefore, the configuration script obtained by this application is the optimal distributed training configuration script, which solves the problem of long search time in the search space and makes automatic tuning highly efficient.

[0075] It should be noted that the pruning rate threshold of this embodiment can be set according to the actual needs of the user, and this embodiment does not limit it.

[0076] S104. Input the executable configuration script into the estimator, perform performance estimation on the executable configuration script through the estimator, and output the estimated performance information.

[0077] Based on the performance prediction of executable configuration scripts by an estimator, this disclosure proposes a single-process-based simulator. The specific operation steps are as follows: the single-process simulator is used to completely and non-intrusively simulate the execution process of the single process in distributed training by starting only one process, thereby accurately predicting the performance of executable configuration scripts with the overhead of a single card. This performance prediction is scalable and its cost is not limited by the cluster and model size.

[0078] like Figure 3 As shown, Figure 3 This is a schematic diagram of a single-process simulator provided in an embodiment of this disclosure. The single-process simulator is implemented through a custom communication backend. In distributed training, the interaction between each process and other processes only occurs during communication. Therefore, replacing the original communication backend (such as NCCL and GLOO) allows for the complete simulation of a single process without changing the framework training process.

[0079] It should be noted that in pipeline parallelism, the tasks to be executed by different processes may differ. The single-process simulator analyzes the pipeline to find the critical path that determines the training time, and calls the simulation process mentioned above for the training tasks on the critical path. The non-intrusiveness and versatility of the single-process simulator make it easy to adapt to different frameworks and versions.

[0080] S105. Record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform a traversal operation.

[0081] The traversal operation refers to repeatedly executing steps S102, S103, S104, and S105 until the search space has been completely traversed.

[0082] S106. After the traversal operation of the search space is completed, the executor compares all runnable configuration scripts based on the results of the traversal operation to obtain the optimal distributed training configuration script.

[0083] S107. Automatically execute the optimal distributed training configuration script to start large-scale distributed training.

[0084] This disclosure provides an automatic tuning method for a distributed parallel strategy, aiming to quickly obtain efficient large-scale model training configurations, thereby saving training costs. Specifically, this disclosure adopts an automated approach, using a searcher to construct a search space based on model and cluster information, searching within the full high-dimensional vector space to obtain the optimal parallel model configuration. A pruning tool performs pruning operations based on the GPU memory model and historical tuning results, thereby improving pruning efficiency and solving the problem of time-consuming searches caused by a large search space. The solution uses a single-process simulator as a predictor to predict the performance of executable configuration scripts, enabling performance prediction of a complete cluster on a single GPU, saving training resources and time required for automatic tuning. Furthermore, it allows concurrent searching within the full high-dimensional vector space, solving the problem of needing to utilize complete training resources during the search process. This achieves performance improvement and saves training costs under large-scale model and cluster training.

[0085] In a preferred embodiment, the executable configuration script is input to a predictor, which performs a performance prediction on the executable configuration script and outputs the predicted performance information, including:

[0086] A process is launched using a single-process simulator to simulate the execution process of the process in distributed training.

[0087] Calculate the overhead of a single card during execution;

[0088] Based on the overhead of the single card, estimate the performance information of the executable configuration script and output the estimated performance information.

[0089] In a preferred embodiment, a process is started using a single-process simulator to simulate the execution process of the process in distributed training, including:

[0090] The single-process simulator starts a process through a custom communication backend;

[0091] Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated.

[0092] During the parallel scheduling of the distributed training pipeline, the pipeline is analyzed using the single-process simulator to obtain the analysis results.

[0093] Based on the analysis results, it was confirmed that the process in question requires different training tasks compared to other processes.

[0094] Based on the different training tasks that the process and other processes need to perform, determine the critical path that determines the training time of the process and other processes;

[0095] The simulation process described above is invoked for the training tasks on the critical path.

[0096] In a preferred embodiment, the method further includes:

[0097] If the pruning rate is less than the set pruning rate threshold, then continue to retrieve new configuration scripts from the search space;

[0098] The new configuration script is input into the pruner, and the pruning operation continues.

[0099] In a preferred embodiment, a pruning operation is performed using a pruner based on the memory model and historical tuning results, and the pruning rate is calculated, including:

[0100] A memory model is established based on the activation tensor and the sum of parameters;

[0101] If the video memory of the configuration script obtained by the video memory model is greater than the video memory of a single hardware card, then the configuration script will be pruned.

[0102] In a preferred embodiment, the method further includes:

[0103] If the historical video memory of a historical configuration script is less than that of the current configuration script, and the video memory of the current configuration script is greater than that of a single hardware card, then the current configuration script will be pruned directly.

[0104] If the historical configuration scripts in the historical data are different from the current configuration scripts only in the recalculation dimension, and the configuration scripts with fewer recalculations are already runnable, then the configuration scripts with more recalculations will be pruned.

[0105] Optionally, recomputation refers to releasing the memory of the activation tensor after the forward computation is completed, and then recomputing it when the reverse computation requires the activation tensor. It is a technique that trades computation for memory, thus increasing runtime. Recomputation can be performed at different granularities; for example, it can recompile the entire transformer layer or only the core attention layer portion of the transformer layer.

[0106] In addition, the number of layers to be recalculated can be selected; for example, all layers can be recalculated, or a specific number of layers can be specified. Therefore, there are two specific pruning rules for recalculation:

[0107] 1. If a configuration script with high computational complexity cannot run due to exceeding the memory capacity of a single graphics card, and other computational complexity dimensions remain unchanged, then the configuration script with low computational complexity will also be unable to run. In this case, the configuration script with high computational complexity will be pruned.

[0108] 2. If configuration scripts with fewer recalculations can run, while configuration scripts with more recalculations can certainly run, their performance will be inferior to those with fewer recalculations, even if other recalculation dimensions remain unchanged. Therefore, configuration scripts with more recalculations can be pruned directly.

[0109] like Figure 4 As shown, in a second aspect, this application provides an automatic tuning device for a distributed parallel strategy, comprising:

[0110] Building unit 41 is configured to construct the search space based on model information and cluster information using a searcher;

[0111] The pruning unit 42 is configured to perform pruning operations based on the memory model and historical tuning results through a pruner, and to calculate the pruning rate.

[0112] The generation unit 43 is configured to generate an executable configuration script by a generator if the pruning rate is greater than or equal to a set pruning rate threshold.

[0113] The performance prediction unit 44 is configured to input the executable configuration script into the predictor, perform performance prediction on the executable configuration script through the predictor, and output the predicted performance information.

[0114] Traversal unit 45 is configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform traversal operations;

[0115] The comparison unit 46 is configured to, after the traversal operation of the search space is completed, compare all runnable configuration scripts with the result of the traversal operation through the executor to obtain the optimal distributed training configuration script.

[0116] Execution unit 47 is configured to automatically execute the optimal distributed training configuration script to initiate large-scale distributed training.

[0117] The above-described device can be implemented using the automatic optimization method of the distributed parallel strategy provided in the first aspect of the embodiment. For specific implementation details, please refer to the description in the first aspect of the embodiment, which will not be repeated here.

[0118] It is understood that the model structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0119] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automatic tuning method for a distributed parallel strategy, characterized in that, include: The search space is constructed using a search engine based on model and cluster information; The pruning operation is performed using a pruning tool based on the memory model and historical optimization results, and the pruning rate is calculated. If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script will be generated by the generator. The executable configuration script is input into the estimator, which performs a performance prediction on the executable configuration script and outputs the predicted performance information. The performance information output by the predictor is recorded by the recorder, and the recorded performance information is sent to the pruner to perform a traversal operation. After the traversal operation of the search space is completed, the executor compares all runnable configuration scripts based on the results of the traversal operation to obtain the optimal distributed training configuration script. The optimal distributed training configuration script is executed automatically to start large-scale distributed training. The executable configuration script is input into the estimator, which performs a performance prediction on the executable configuration script and outputs the predicted performance information, including: A process is launched using a single-process simulator to simulate the execution process of the process in distributed training. Calculate the overhead of a single card during execution; Based on the overhead of the single card, estimate the performance information of the executable configuration script and output the estimated performance information.

2. The automatic tuning method for distributed parallel strategies according to claim 1, characterized in that, A process is launched using a single-process simulator to simulate the execution process of the process in distributed training, including: The single-process simulator starts a process through a custom communication backend; Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated. During the parallel scheduling of the distributed training pipeline, the pipeline is analyzed using the single-process simulator to obtain the analysis results. Based on the analysis results, it was confirmed that the process in question requires different training tasks compared to other processes. Based on the different training tasks that the process and other processes need to perform, determine the critical path that determines the training time of the process and other processes; The simulation process described above is invoked for the training tasks on the critical path.

3. The automatic tuning method for distributed parallel strategies according to claim 1, characterized in that, The method further includes: If the pruning rate is less than the set pruning rate threshold, then continue to retrieve new configuration scripts from the search space; The new configuration script is input into the pruner, and the pruning operation continues.

4. The automatic tuning method for distributed parallel strategies according to claim 1, characterized in that, The pruning process is performed using a pruning tool based on the memory model and historical tuning results, and the pruning rate is calculated, including: A memory model is established based on the activation tensor and the sum of parameters; If the video memory of the configuration script obtained by the video memory model is greater than the video memory of a single hardware card, then the configuration script will be pruned.

5. The automatic tuning method for distributed parallel strategies according to claim 4, characterized in that, The method further includes: If the historical video memory of a historical configuration script in the historical data is less than that of the current configuration script, and the video memory of the current configuration script is greater than that of a single hardware card, then the current configuration script will be pruned directly. If the historical configuration scripts in the historical data are different from the current configuration scripts only in the recalculation dimension, and the configuration scripts with fewer recalculations are already runnable, then the configuration scripts with more recalculations will be pruned.

6. An automatic tuning device for a distributed parallel strategy, characterized in that, include: The building unit is configured to construct the search space based on model information and cluster information using a searcher; The pruning unit is configured to perform pruning operations based on the memory model and historical tuning results through a pruner, and to calculate the pruning rate; The generation unit is configured to generate an executable configuration script by the generator if the pruning rate is greater than or equal to a set pruning rate threshold. The performance prediction unit is configured to input the executable configuration script into the predictor, perform performance prediction on the executable configuration script through the predictor, and output the predicted performance information. The traversal unit is configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform the traversal operation; The comparison unit is configured to, after the traversal operation of the search space is completed, compare all runnable configuration scripts with the result of the traversal operation through the executor to obtain the optimal distributed training configuration script. The execution unit is configured to automatically execute the optimal distributed training configuration script to initiate large-scale distributed training; The executable configuration script is input into the estimator, which performs a performance prediction on the executable configuration script and outputs the predicted performance information, including: A process is launched using a single-process simulator to simulate the execution process of the process in distributed training. Calculate the overhead of a single card during execution; Based on the overhead of the single card, estimate the performance information of the executable configuration script and output the estimated performance information.

7. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the automatic tuning method of the distributed parallel strategy according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the automatic tuning method of the distributed parallel strategy as described in any one of claims 1 to 5.

9. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the automatic tuning method for the distributed parallel strategy as described in any one of claims 1 to 5.

Images