Automatic tuning method and device for distributed parallel strategy, medium and program product

Abstract

The invention provides an automatic tuning method and device for a distributed parallel strategy, a medium and a program product. The method comprises the steps that a search space is constructed through a searcher according to model information and cluster information; performing pruning operation based on the video memory model and the historical tuning result through a pruning device, and calculating a pruning rate; if the pruning rate is greater than or equal to a set pruning rate threshold value, generating an executable configuration script through a generator; performing performance pre-estimation on the executable configuration script through a pre-estimator, and outputting pre-estimated performance information; recording performance information through a recorder, and sending the performance information to a pruning device to execute traversal operation; and after the traversal operation of the search space is completed, comparing all the runnable configuration scripts according to the result of the traversal operation to obtain an optimal distributed training configuration script, automatically executing the optimal distributed training configuration script, and starting large-scale distributed training. According to the technical scheme provided by the invention, efficient large model training configuration can be quickly obtained, so that the training cost is saved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, medium, and program product for automatically tuning a distributed parallel strategy. Background Art

[0002] In recent years, in the era of large models, single graphics card memory has completely failed to meet the training requirements of models. Distributed parallel training has become the only solution for large model training. As distributed parallel training has attracted increasing attention from industry and academia, a variety of parallelization and optimization strategies have emerged to continuously improve model training performance.

[0003] The main parallel strategies and optimization strategies adopted by related technologies include data parallelism, tensor parallelism, pipeline parallelism, context parallelism, etc. Since parallel strategies and optimization strategies can be combined with each other, the performance differences between different distributed parallel training configurations are huge. How to quickly obtain efficient training configurations is crucial to saving training costs.

[0004] The main disadvantages of the related technologies are:

[0005] First, expert configuration training strategies lack universality. When the type of model training machines changes, the original configuration training strategy is usually not the optimal strategy, or even unusable. When the number of model training machines changes, the original configuration training strategy will also face the above problems.

[0006] Second, expert experience is not globally optimal. The expert configuration training strategy obtained by manually designing a limited number of experiments based on expert experience is difficult to achieve optimal performance.

[0007] Third, since parallel strategies and optimization strategies can be combined with each other, the search space for training strategy configuration is huge. The brute force search method takes a long time to determine the optimal strategy, and requires complete training resources, which is highly dependent. Summary of the Invention

[0008] The purpose of this application is to provide a method, device, medium and program product for automatic tuning of a distributed parallel strategy, aiming to quickly obtain efficient large-model training configuration, thereby saving training costs.

[0009] According to a first aspect of the present application, a method for automatically tuning a distributed parallel strategy is provided, comprising:

[0010] The search space is constructed based on the model information and cluster information through the searcher;

[0011] The pruner performs pruning based on the memory model and historical tuning results, and calculates the pruning rate.

[0012] If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script is generated through the generator;

[0013] Inputting the executable configuration script into an estimator, performing performance estimation on the executable configuration script by the estimator, and outputting estimated performance information;

[0014] Recording the performance information output by the estimator through a recorder, and sending the recorded performance information to the pruner to perform a traversal operation;

[0015] After the traversal operation of the search space is completed, the executor compares all executable configuration scripts according to the results of the traversal operation to obtain the optimal distributed training configuration script;

[0016] Automatically execute the optimal distributed training configuration script to start large-scale distributed training.

[0017] In an optional embodiment, the executable configuration script is input into an estimator, and the estimator performs performance estimation on the executable configuration script and outputs estimated performance information, including:

[0018] Launch a process through a single-process simulator to simulate the execution of the process in distributed training;

[0019] Calculate the cost of a single card during execution;

[0020] Performance information of the executable configuration script is estimated based on the overhead of the single card, and the estimated performance information is output.

[0021] In an optional embodiment, starting a process through a single process simulator to simulate the execution of the process in distributed training includes:

[0022] The single-process simulator starts a process through a custom communication backend;

[0023] Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated;

[0024] During the pipeline parallel scheduling process of distributed training, the pipeline is analyzed by the single-process simulator to obtain analysis results;

[0025] According to the analysis results, different training tasks that need to be performed by the process and other processes are determined;

[0026] Determining a critical path that determines the training time of the process and other processes based on different training tasks that need to be performed by the process and other processes respectively;

[0027] The above simulation process is called for the training tasks on the critical path.

[0028] In an optional embodiment, the method further comprises:

[0029] If the pruning rate is less than the set pruning rate threshold, continue to extract a new configuration script from the search space;

[0030] The new configuration script is input into the pruner to continue to perform the pruning operation.

[0031] In an optional embodiment, a pruning operation is performed by a pruner based on a video memory model and historical tuning results, and a pruning rate is calculated, including:

[0032] Build a memory model based on the activation tensor and the sum of parameters;

[0033] If the video memory of the current configuration script obtained by the video memory model is larger than the video memory of the hardware single card, the current configuration script is pruned.

[0034] In an optional embodiment, the method further comprises:

[0035] If the historical video memory of the historical configuration script in the historical data is smaller than the video memory of the current configuration script, and the video memory of the current configuration script is larger than the video memory of the hardware single card, the current configuration script will be pruned directly;

[0036] If the historical configuration scripts in the historical data differ from the current configuration script only in the recalculation dimension, and the configuration scripts with less recalculation are already runnable, the configuration script with more recalculation will be pruned.

[0037] According to a second aspect of the present application, there is provided an automatic tuning device for a distributed parallel strategy, comprising:

[0038] A construction unit is configured to construct a search space according to the model information and the cluster information through a searcher;

[0039] The pruning unit is configured to perform pruning operations based on the memory model and historical tuning results through the pruner and calculate the pruning rate;

[0040] a generating unit configured to generate an executable configuration script through a generator if the pruning rate is greater than or equal to a set pruning rate threshold;

[0041] a performance estimation unit configured to input the executable configuration script into an estimator, perform performance estimation on the executable configuration script through the estimator, and output estimated performance information;

[0042] a traversal unit configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform a traversal operation;

[0043] A comparison unit is configured to compare all executable configuration scripts according to the result of the traversal operation after the traversal operation in the search space is completed, so as to obtain an optimal distributed training configuration script;

[0044] The execution unit is configured to automatically execute the optimal distributed training configuration script to start large-scale distributed training.

[0045] Compared with related technologies, the technical solution of this application has the following advantages:

[0046] This application adopts an automated approach, using a searcher to construct a search space based on model information and cluster information, and to search within the full high-dimensional vector space to obtain the optimal model parallel configuration; a pruning operation is performed by a pruner based on the video memory model and historical tuning results, thereby improving the efficiency of the pruning operation and solving the time-consuming search caused by the large search space. The solution of this application uses a single-process simulator as an estimator to perform performance estimation on executable configuration scripts. It can estimate the performance on a complete cluster on a single card, saving the training resources and training time required for automatic tuning, and can perform concurrent searches within the full high-dimensional vector space, solving the problem that the search process needs to utilize complete training resources, achieving performance improvements under large-model and large-cluster training, and saving training costs.

[0047] This application solves the problem of lack of universality of expert experience and is applicable to any cluster scale and any model size. The automatic tuning mechanism of this application takes cluster information and model information as input, and can automatically execute the automatic tuning process based on the input. It does not rely on expert experience, solves the problem of non-optimal expert experience, and can obtain the best parallel training configuration; this application constructs a complete search space, and uses an efficient pruning method to completely traverse the search space. Therefore, the configuration script obtained by this application is the optimal distributed training configuration script, which solves the problem of long search time in the search space and makes the automatic tuning highly efficient; the single-process simulator proposed in this solution can use single-card resources to perform tuning on the complete training resources, which greatly saves search resources. By further dividing the search space and using multiple single cards for concurrent search, the search efficiency is further improved.

[0048] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be achieved and obtained through the structures and processes indicated in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. It is obvious that the drawings described below are certain embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flow chart of an automatic tuning method for a distributed parallel strategy provided by an embodiment of the present disclosure;

[0051] Figure 2 This is a flow chart of another automatic tuning method for a distributed parallel strategy provided by an embodiment of the present disclosure;

[0052] Figure 3 is a schematic diagram of a single-process simulator provided by an embodiment of the present disclosure;

[0053] Figure 4 It is a structural diagram of an automatic tuning device for a distributed parallel strategy provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] See also Figure 1 The present application provides a method for automatically tuning a distributed parallel strategy, including the following steps:

[0056] S101. Construct a search space based on model information and cluster information through a searcher.

[0057] In an optional embodiment, the search space configuration parameters of the constructed search space are as follows:

[0058] a) Data Parallel Size: The value of data parallelism is an integer, which does not exceed the total number of hardware cards and is divisible by the global batch size.

[0059] b) Tensor Parallel Size: The value of tensor parallelism is an integer. The value of tensor parallelism does not exceed the total number of hardware cards in a whole machine, and the value of tensor parallelism satisfies the requirement of being divisible by the relevant parameters of the video memory model. For example, the value of tensor parallelism is divisible by the number of hidden layers in the video memory model.

[0060] c) Pipeline Parallel Size: The value of pipeline parallelism is an integer. The value of pipeline parallelism does not exceed the total number of hardware single cards, and the value of pipeline parallelism satisfies the requirement of being divisible by the number of hidden layers of the video memory model.

[0061] d) Expert Parallel Size: The value of expert parallelism is an integer, and the value of expert parallelism does not exceed the number of experts.

[0062] e) Context parallelism (TensorParallel Size): The value of context parallelism is an integer, and the value of context parallelism does not exceed the total number of hardware cards, and the value of context parallelism satisfies the requirement of being divisible by the sequence length.

[0063] f) Distributed optimizer: The value of the distributed optimizer is a Boolean value. The distributed optimizer is used to optimize data parallelism. When the data parallelism is greater than 1, the distributed optimizer takes effect.

[0064] g) Sequence parallelism: The value of sequence parallelism is Boolean. Sequence parallelism is used to optimize tensor parallelism. When the tensor parallelism is greater than 1, sequence parallelism takes effect.

[0065] h) Recalculation method: The value of the recalculation method is ["block", "uniform"], where "block" means only recalculating the front layers, and "uniform" means grouping the hidden layers of the memory model before recalculating.

[0066] For example, a transformer model is composed of stacked identical layers. The number of layers in the transformer model is 28. The recalculation method is set to "block" and the number of recalculation layers is set to 5. This means that the first 5 layers of the transformer model are recalculated, and the remaining 23 layers of the transformer model do not need to be recalculated.

[0067] i) Recomputation granularity: The recomputation granularity value is ["full", "selective"], where "full" means recomputing the entire transformer layer, and "selective" means recomputing only the attention part of the memory model.

[0068] j) Number of recalculated layers: The value of the recalculated layer number is an integer, which does not exceed the total number of hidden layers of the memory model and does not exceed the total number of layers allocated under pipeline parallelism.

[0069] This application solves the problem of lack of universality of expert experience and is applicable to any cluster scale and any model size. The automatic tuning mechanism of this application takes cluster information and model information as input, and can automatically execute the automatic tuning process based on the input. It does not rely on expert experience, solves the problem of non-optimal expert experience, and can obtain the optimal parallel training configuration.

[0070] S102: Perform pruning operations based on the video memory model and historical optimization results through a pruner, and calculate the pruning rate.

[0071] like Figure 2 As shown, it is a flow chart of another automatic tuning method of a distributed parallel strategy provided by an embodiment of the present disclosure. The embodiment of the present disclosure performs pruning operations based on a video memory model and historical adjustment results through a pruner. If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script is generated by a generator, and the next step is continued through the pruner. If the pruning rate is less than the set pruning rate threshold, a new configuration script is continuously retrieved from the search space. A video memory model is established based on the activation tensor and the sum of parameters. If the video memory of the current configuration script obtained by the video memory model is greater than the video memory of a single hardware card, the current configuration script is pruned.

[0072] Optionally, if there are historical configuration scripts in the historical data, a comparison operation is performed. The comparison principle is: if the historical video memory of the historical configuration script in the historical data is smaller than the video memory of the current configuration script, and the video memory of the current configuration script is larger than the video memory of the hardware single card, that is, OOM memory overflow, the current configuration script can be directly pruned; in terms of recalculation, if the historical configuration script in the historical data is the same as the current configuration script except for the recalculation dimension, and the configuration with less recalculation is already runnable, the configuration with more recalculation will be pruned.

[0073] S103: If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script is generated by the generator.

[0074] This application constructs a complete search space and completely traverses the search space through an efficient pruning method. Therefore, the configuration script obtained by this application is the optimal distributed training configuration script, which solves the problem of long search time in the search space and makes automatic tuning highly efficient.

[0075] It should be noted that the pruning rate threshold in the embodiment of the present disclosure can be set according to the actual needs of the user, and the embodiment of the present disclosure does not limit this.

[0076] S104: Input the executable configuration script into an estimator, perform performance estimation on the executable configuration script through the estimator, and output estimated performance information.

[0077] The performance of the executable configuration script is estimated by an estimator. Based on this, the embodiment of the present disclosure proposes a single-process-based simulator. The specific operation steps are as follows: the single-process simulator is used to completely and non-invasively simulate the execution process of the single process in distributed training by starting only one process, thereby accurately estimating the performance of the executable configuration script with the overhead of a single card. This performance estimation is scalable and its cost is not limited by the cluster and model scale.

[0078] like Figure 3 As shown, Figure 3 This figure shows a schematic diagram of a single-process simulator provided in an embodiment of the present disclosure. The single-process simulator is implemented through a custom communication backend. In distributed training, the interaction between each process and other processes only occurs during communication. Therefore, replacing the original communication backend (such as NCCL and GLOO) can allow for a complete simulation of a single process without changing the framework training process.

[0079] It should be noted that in pipeline parallelism, the tasks that different processes need to perform may be different. The single-process simulator will analyze the pipeline to find the critical path that determines the training time and call the simulation process mentioned above for the training tasks on the critical path. The non-invasiveness and versatility of the single-process simulator make it easily adaptable to different frameworks and versions.

[0080] S105. Record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform a traversal operation.

[0081] The traversal operation refers to repeatedly executing step S102, step S103, step S104 and step S105 until the search space is completely traversed.

[0082] S106. After the traversal operation in the search space is completed, the executor compares all executable configuration scripts according to the results of the traversal operation to obtain the optimal distributed training configuration script.

[0083] S107: Automatically execute the optimal distributed training configuration script to start large-scale distributed training.

[0084] The embodiment of the present disclosure provides an automatic tuning method for a distributed parallel strategy, which aims to quickly obtain an efficient large-model training configuration, thereby saving training costs. Specifically, the embodiment of the present disclosure adopts an automated approach, using a searcher to construct a search space based on model information and cluster information, and searches in the full high-dimensional vector space to obtain the optimal model parallel configuration; a pruning operation is performed by a pruner based on a video memory model and historical tuning results, thereby improving the efficiency of the pruning operation and solving the time-consuming search caused by the large search space. The solution of the present application uses a single-process simulator as an estimator to perform performance estimation on executable configuration scripts, and can estimate the performance on a complete cluster on a single card, saving the training resources and training time required for automatic tuning, and can perform concurrent searches in the full high-dimensional vector space, solving the problem that the search process needs to utilize complete training resources, achieving performance improvement in large-model and large-cluster training, and saving training costs.

[0085] In a preferred embodiment, the executable configuration script is input into an estimator, and the estimator performs performance estimation on the executable configuration script and outputs estimated performance information, including:

[0086] Launch a process through a single-process simulator to simulate the execution of the process in distributed training;

[0087] Calculate the cost of a single card during execution;

[0088] Performance information of the executable configuration script is estimated based on the overhead of the single card, and the estimated performance information is output.

[0089] In a preferred embodiment, starting a process through a single process simulator and simulating the execution of the process in distributed training includes:

[0090] The single-process simulator starts a process through a custom communication backend;

[0091] Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated;

[0092] During the pipeline parallel scheduling process of distributed training, the pipeline is analyzed by the single-process simulator to obtain analysis results;

[0093] According to the analysis results, different training tasks that need to be performed by the process and other processes are determined;

[0094] Determining a critical path that determines the training time of the process and other processes based on different training tasks that need to be performed by the process and other processes respectively;

[0095] The above simulation process is called for the training tasks on the critical path.

[0096] In a preferred embodiment, the method further comprises:

[0097] If the pruning rate is less than the set pruning rate threshold, continue to extract a new configuration script from the search space;

[0098] The new configuration script is input into the pruner to continue to perform the pruning operation.

[0099] In a preferred embodiment, a pruning operation is performed by a pruner based on a video memory model and historical tuning results, and a pruning rate is calculated, including:

[0100] Build a memory model based on the activation tensor and the sum of parameters;

[0101] If the video memory of the current configuration script obtained by the video memory model is larger than the video memory of the hardware single card, the current configuration script is pruned.

[0102] In a preferred embodiment, the method further comprises:

[0103] If the historical video memory of the historical configuration script in the historical data is smaller than the video memory of the current configuration script, and the video memory of the current configuration script is larger than the video memory of the hardware single card, the current configuration script will be pruned directly;

[0104] If the historical configuration scripts in the historical data differ from the current configuration script only in the recalculation dimension, and the configuration scripts with less recalculation are already runnable, the configuration scripts with more recalculation are pruned.

[0105] Optionally, recomputation involves releasing the activation tensor memory after the forward computation is complete, and recalculating it when the activation tensor is needed for the backward computation. This technique trades computation for memory, thus increasing runtime. Recomputation can be performed at different granularities, for example, recomputing the entire transformer layer or just the core attention layer within the transformer layer.

[0106] In addition, you can also choose the number of layers to recalculate, for example, you can choose to recalculate all layers, or you can specify the number of layers to recalculate. Therefore, there are two specific pruning rules for recalculation:

[0107] 1. If the configuration script with more recalculation cannot run due to the hardware card memory being exceeded, and the configuration script with less recalculation cannot run while other recalculation dimensions remain unchanged, the configuration script with more recalculation will be pruned.

[0108] 2. If the configuration script with less recalculation can be run, when other recalculation dimensions remain unchanged, the configuration script with more recalculation can definitely be run, but the performance is not as good as the configuration script with less recalculation. Therefore, the configuration script with more recalculation can be directly pruned.

[0109] like Figure 4 As shown, in a second aspect, the present application provides an automatic tuning device for a distributed parallel strategy, comprising:

[0110] A construction unit 41 is configured to construct a search space according to the model information and the cluster information through a searcher;

[0111] The pruning unit 42 is configured to perform a pruning operation based on a memory model and historical tuning results through a pruner, and calculate a pruning rate;

[0112] The generating unit 43 is configured to generate an executable configuration script through a generator if the pruning rate is greater than or equal to a set pruning rate threshold;

[0113] a performance estimating unit 44 configured to input the executable configuration script into an estimator, perform performance estimation on the executable configuration script through the estimator, and output estimated performance information;

[0114] a traversal unit 45 configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform a traversal operation;

[0115] The comparison unit 46 is configured to compare all executable configuration scripts according to the result of the traversal operation after the traversal operation of the search space is completed, so as to obtain the optimal distributed training configuration script;

[0116] The execution unit 47 is configured to automatically execute the optimal distributed training configuration script to start large-scale distributed training.

[0117] The above-mentioned device can be implemented by the automatic tuning method of the distributed parallel strategy provided by the embodiment of the first aspect above. The specific implementation method can be found in the description of the embodiment of the first aspect and will not be repeated here.

[0118] It is understood that the model structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art may also readily conceive of combinations and adjustments to the structural features of the above embodiments as needed, and should not limit the concepts of this application to the specific details of the above examples.

[0119] Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed parallel strategy automatic tuning method, characterized in that: include: The search space is constructed based on the model information and cluster information through the searcher; The pruner performs pruning based on the memory model and historical tuning results, and calculates the pruning rate. If the pruning rate is greater than or equal to the set pruning rate threshold, an executable configuration script is generated through the generator; Inputting the executable configuration script into an estimator, performing performance estimation on the executable configuration script by the estimator, and outputting estimated performance information; Recording performance information output by the estimator through a recorder, and sending the recorded performance information to the pruner to perform a traversal operation; After the traversal operation of the search space is completed, the executor compares all executable configuration scripts according to the results of the traversal operation to obtain the optimal distributed training configuration script; Automatically execute the optimal distributed training configuration script to start large-scale distributed training.

2. The automatic tuning method for distributed parallel strategy according to claim 1, characterized in that: Inputting the executable configuration script into an estimator, performing performance estimation on the executable configuration script by the estimator, and outputting estimated performance information, including: Launch a process through a single-process simulator to simulate the execution of the process in distributed training; Calculate the cost of a single card during execution; Performance information of the executable configuration script is estimated based on the overhead of the single card, and the estimated performance information is output.

3. The automatic tuning method for distributed parallel strategy according to claim 2, characterized in that: Starting a process through a single-process simulator to simulate the execution process of the process in distributed training, including: The single-process simulator starts a process through a custom communication backend; Without changing the framework training process, the interaction between the process and other processes in distributed training is fully simulated; During the pipeline parallel scheduling process of distributed training, the pipeline is analyzed by the single-process simulator to obtain analysis results; According to the analysis results, different training tasks that need to be performed by the process and other processes are determined; Determining a critical path that determines the training time of the process and other processes based on different training tasks that need to be performed by the process and other processes respectively; The above simulation process is called for the training tasks on the critical path.

4. The automatic tuning method for distributed parallel strategy according to claim 1, characterized in that: The method further comprises: If the pruning rate is less than the set pruning rate threshold, continue to extract a new configuration script from the search space; The new configuration script is input into the pruner to continue to perform pruning operations.

5. The automatic tuning method of distributed parallel strategy according to claim 1, characterized in that: The pruner performs pruning based on the memory model and historical tuning results, and calculates the pruning rate, including: Build a memory model based on the activation tensor and the sum of parameters; If the video memory of the current configuration script obtained by the video memory model is larger than the video memory of the hardware single card, the current configuration script is pruned.

6. The automatic tuning method for distributed parallel strategy according to claim 5, characterized in that: The method further comprises: If the historical video memory of the historical configuration script in the historical data is smaller than the video memory of the current configuration script, and the video memory of the current configuration script is larger than the video memory of the hardware single card, the current configuration script will be pruned directly; If the historical configuration scripts in the historical data differ from the current configuration script only in the recalculation dimension, and the configuration scripts with less recalculation are already runnable, the configuration script with more recalculation will be pruned.

7. An automatic tuning device for a distributed parallel strategy, characterized in that: include: A construction unit is configured to construct a search space according to the model information and the cluster information through a searcher; The pruning unit is configured to perform pruning operations based on the memory model and historical tuning results through the pruner and calculate the pruning rate; a generating unit configured to generate an executable configuration script through a generator if the pruning rate is greater than or equal to a set pruning rate threshold; a performance estimation unit configured to input the executable configuration script into an estimator, perform performance estimation on the executable configuration script through the estimator, and output estimated performance information; a traversal unit configured to record the performance information output by the estimator through a recorder, and send the recorded performance information to the pruner to perform a traversal operation; A comparison unit is configured to compare all executable configuration scripts according to the result of the traversal operation after the traversal operation in the search space is completed, so as to obtain an optimal distributed training configuration script; The execution unit is configured to automatically execute the optimal distributed training configuration script to start large-scale distributed training.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the automatic tuning method for the distributed parallel strategy according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the automatic tuning method for a distributed parallel strategy according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the automatic tuning method of the distributed parallel strategy described in any one of claims 1 to 6 are implemented.

Images