A model distribution parallelization training process automatic optimization method, system and device

Abstract

The present application relates to the technical field of distributed model training, and provides a model distribution parallel training process automatic optimization method, system and device. The method comprises the following steps: defining a task plane according to the communication and computing power of a neural network model, dividing the task plane to obtain a divided task plane; constructing a training time consumption model according to the divided task plane to obtain total time consumption; using a tournament algorithm to design a genetic algorithm to solve the training time consumption model according to the total time consumption to obtain a training optimization result; and embedding the training optimization result into the neural network model for optimization and updating. The present application models the training process in detail under the scene of a mesh pipeline and a heterogeneous network and computing power equipment. According to the established training process-training time consumption model, a meta-heuristic algorithm is applied to search for an optimal solution to realize automatic optimization of the distributed parallel training process.

Description

An automatic optimization method, system, and device for distributed parallel training process of models. Technical Field

[0001] This invention relates to the field of distributed model training technology, and provides an automatic optimization method, system and device for distributed parallel training processes. Background Technology

[0002] Most existing methods for automatically optimizing distributed parallel training processes for models are designed for specific model and hardware architectures, employing multiple parallel strategies to create distributed training and inference architectures. These architectures typically perform well within limited scenarios and are deployed in production applications. However, the design and resource configuration of these architectures usually require specialized engineering experience and are time-consuming. Given the increasing diversity of models and deep learning infrastructure, designing dedicated distributed training and inference architectures for each scenario becomes unacceptable in terms of cost and time. Therefore, designing an automatic parallelism architecture based on arbitrary model architectures and hardware resources is crucial for reducing upfront workload for specific scenarios, lowering the training threshold for large models with varying architectures, and accelerating the development of model inference services. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a method, system, and apparatus for automatically optimizing a distributed parallel training process, which realizes the automatic optimization of the distributed parallel training process by applying a metaheuristic algorithm to search for a better solution based on an established training process-training time model.

[0004] This invention provides an automatic optimization method for distributed parallel training processes of models, including:

[0005] S1: Define the task plane based on the communication and computing power of the neural network model, divide the task plane, and obtain the divided task plane;

[0006] S2: Construct a training time model based on the divided task plane to obtain the total time;

[0007] S3: Based on the total time consumption, use the tournament algorithm to design a genetic algorithm to solve the training time consumption model and obtain the training optimization results;

[0008] S4: Embed the training optimization results into the neural network model for optimization and updating.

[0009] According to the present invention, an automatic optimization method for distributed parallel training process of a model is provided, wherein step S1 includes:

[0010] S11: Define the task plane, which includes a global data parallelism layer, a mesh pipeline parallelism layer, and a local data parallelism layer;

[0011] S12: Use the process group initialization algorithm to divide the task plane to obtain the divided task plane.

[0012] According to the present invention, an automatic optimization method for distributed parallel training process of a model is provided, wherein the process group initialization algorithm includes the following steps:

[0013] S121: Input the training plan and total number of layers of the neural network model, and set the current split point to 0;

[0014] S122: Initialize a process group;

[0015] S123: Set the next device as the current device;

[0016] S124: Determine whether the current split point is greater than or equal to the current device's starting layer number and less than the current device's ending layer number. If yes, proceed to step S125; otherwise, proceed to step S126.

[0017] S125: Add the current device to the current process group;

[0018] S126: Determine whether all devices have been traversed. If yes, proceed to step S127; otherwise, proceed to step S123.

[0019] S127: Set the current process group's layer range to: [current split point, minimum end layer value within the current process group], and add the current process group to the set of all process groups;

[0020] S128: Determine if the current process group's end level is equal to the total level minus 1. If yes, proceed to step S129; otherwise, proceed to step S122.

[0021] S129: Output the set of all process groups, which is the partitioned task plane.

[0022] According to the present invention, an automatic optimization method for distributed parallel training process of a model is provided, step S2 includes:

[0023] S21: Calculate the total pipeline time based on the divided task plane. ;

[0024] S22: Calculate the All Reduce time based on the divided task plane. ;

[0025] S23: Calculate total time consumption :

[0026] .

[0027] According to the present invention, an automatic optimization method for distributed parallel training process of a model is provided, step S21 includes:

[0028] S211: Calculate the forward computation time based on the divided task plane. and reverse calculation time :

[0029]

[0030]

[0031] in, For device ID, For the total number of devices, For the current device The corresponding pipeline parallelism level in the model is the split point between the previous pipeline parallelism level and the previous pipeline parallelism level, where i is the device. In the global data parallelism index, j represents the device. In the pipeline parallel hierarchy, k is the device number. Numbering at the local data parallelism level For the model number Forward computation of a layer This represents the layer number of the neural network. For equipment computing power For equipment Decision variables, For equipment micro-batch size, For the model number The reverse computation cost of the layer;

[0032] S212: Calculate the forward communication duration based on the divided task plane. and reverse communication duration :

[0033]

[0034]

[0035] in, To find the maximum value function, This represents the current stage number of the pipeline. For equipment and equipment Data transfer volume between them , for The device's number at the local data parallelism level. For equipment Decision variables, The dividing point The amount of data to be transmitted Indicates equipment and equipment The actual bandwidth between Indicates equipment and equipment The actual bandwidth between;

[0036] S213: Calculate total pipeline time :

[0037]

[0038]

[0039] in, For equipment Global data parallel processing time For equipment The time consumed in the pipeline stage. The time taken for each stage For equipment It accounts for the total number of pipelines. For equipment Total number of occupancy stages For equipment The total number of task planes after partitioning.

[0040] According to the automatic optimization method for model distributed parallel training process provided by the present invention, the process of converting the total time into chromosomes is as follows: the process of converting the total time into chromosomes is as follows: taking the smallest square in the divided task plane as the unit, the information in the corresponding position point of the device is converted into genetic material content.

[0041] The information within the location point corresponding to the device includes: device ID, total time consumed by the device, location point of the device, and micro-batch size of the device.

[0042] According to the automatic optimization method for distributed parallel training process of the model provided by the present invention, step S3 includes:

[0043] S31: Set the chromosome as the initial solution population and set the number of cycles to 0;

[0044] S32: Use the tournament algorithm to select the initial population of special solutions;

[0045] S33: Use the roulette wheel algorithm to perform random crossover or mutation on the initial special solution population;

[0046] S34: Calculate the fitness of the initial special solution based on the initial special solution population after random crossover or mutation, generate a new population, and increment the cycle count by 1;

[0047] S35: Determine whether the loop count has reached the termination condition. If not, return to step S32.

[0048] If the goal is to output the individual with the highest fitness in the new population.

[0049] According to the automatic optimization method for distributed parallel training process of the model provided by the present invention, step S4 includes:

[0050] In each device, the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level to which the process group initialization algorithm belongs are selected. For the parameter gradients of all layers within the corresponding layer range of the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level of the device, AllReduce collection communication and parameter update are completed in the corresponding process group according to the layer number order.

[0051] This invention also provides an automatic optimization system for distributed parallel training processes, comprising:

[0052] Task plane partitioning module: Defines the task plane based on the communication and computing power of the neural network model, partitions the task plane, and obtains the partitioned task plane;

[0053] Time consumption calculation module: Constructs a training time consumption model based on the divided task plane to obtain the total time consumption;

[0054] Optimization module: Based on the time consumption calculation module, a genetic algorithm is designed using the tournament algorithm to optimize the training time, and the optimization results of the training process are obtained;

[0055] Model training module: Trains the neural network model based on the training optimization results.

[0056] The present invention also provides an electronic device, including a memory, a processor, a graphics computing unit (GPU), and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an automatic optimization method for a model distributed parallel training process as described above.

[0057] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0058] This invention provides a method, system, and apparatus for automatically optimizing a distributed parallel training process. It involves detailed modeling of the training process in scenarios involving mesh pipelines and heterogeneous networks and computing power. Based on the established training process-training time model, a metaheuristic algorithm is applied to search for a better solution, thereby achieving automatic optimization of the distributed parallel training process.

[0059] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 is a flowchart of an automatic optimization method for a distributed parallel training process of a model provided by the present invention.

[0062] Figure 2 is a task plan view of an embodiment of the present invention.

[0063] Figure 3 is a task plan diagram after division according to an embodiment of the present invention.

[0064] Figure 4 is a flowchart of the process group initialization algorithm according to an embodiment of the present invention.

[0065] Figure 5 is a block diagram of an automatic optimization system for distributed parallel training process of a model provided by the present invention.

[0066] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention.

[0067] Figure label:

[0068] 101. Task plane partitioning module; 102. Time consumption calculation module; 103. Training module; 104. Self-optimization module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus; 850. Graphics computing unit; 860. Computing node; 870. Inter-device network. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0070] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0071] The present invention will now be described with reference to Figures 1 to 6.

[0072] Example

[0073] As shown in Figure 1, Figure 1 is a flowchart of an automatic optimization method for distributed parallel training of a model provided by the present invention, including:

[0074] S1: Define the task plane based on the communication and computing power of the neural network model, divide the task plane, and obtain the divided task plane;

[0075] S2: Construct a training time model based on the divided task plane to obtain the total time;

[0076] S3: Based on the total time consumption, use the tournament algorithm to design a genetic algorithm to solve the training time consumption model and obtain the training optimization results;

[0077] S4: Embed the training optimization results into the neural network model for optimization and updating.

[0078] As shown in Figure 2, this embodiment of the invention uses four Nvidia RTX3090 GPUs (blue squares), four Nvidia Tesla V100S GPUs (green squares), and four Nvidia Tesla P100 GPUs (yellow squares). Figure 2 defines a task plane, which can be divided into three levels: global data parallelism, mesh pipeline parallelism, and local data parallelism. The horizontal axis represents the number of model layers, and the vertical axis represents the micro-batch size. Each rectangle represents a device. The size of the rectangle represents the range of model layers trained by the device and the micro-batch size; the position of the rectangle in the figure represents its relationship with other devices in pipeline and data parallel transmission.

[0079] Specifically, step S1 includes:

[0080] S11: Define the task plane, which includes a global data parallelism layer, a mesh pipeline parallelism layer, and a local data parallelism layer;

[0081] S12: Use the process group initialization algorithm to divide the task plane to obtain the divided task plane.

[0082] Specifically, as shown in Figure 4, the process group initialization algorithm includes the following steps:

[0083] S121: Input the training plan and total number of layers of the neural network model, and set the current split point to 0;

[0084] S122: Initialize a process group;

[0085] S123: Set the next device as the current device;

[0086] S124: Determine whether the current split point is greater than or equal to the current device's starting layer number and less than the current device's ending layer number. If yes, proceed to step S125; otherwise, proceed to step S126.

[0087] S125: Add the current device to the current process group;

[0088] S126: Determine whether all devices have been traversed. If yes, proceed to step S127; otherwise, proceed to step S123.

[0089] S127: Set the current process group's layer range to [current split point, minimum end layer value within the current process group], and add the current process group to the set of all process groups;

[0090] S128: Determine if the current process group's end level is equal to the total level minus 1. If yes, proceed to step S129; otherwise, proceed to step S122.

[0091] S129: Output the set of all process groups, which is the partitioned task plane.

[0092] As shown in Figure 3, Figure 3 is a task plan diagram after division according to an embodiment of the present invention. The part selected by the red dashed line in Figure 3 is the divided task plan.

[0093] Specifically, step S2 includes:

[0094] S21: Calculate the total pipeline time based on the divided task plane. ;

[0095] S22: Calculate the All Reduce time based on the divided task plane. The AllReduce time at the maximum communication bandwidth is the total AllReduce time of the entire system.

[0096] S23: Calculate total time consumption :

[0097] .

[0098] Specifically, step S21 includes:

[0099] S211: Calculate the forward computation time based on the divided task plane. and reverse calculation time :

[0100]

[0101]

[0102] in, For device ID, For the total number of devices, For the current device The corresponding pipeline parallelism level in the model is the split point between the previous pipeline parallelism level and the previous pipeline parallelism level, where i is the device. In the global data parallelism index, j represents the device. In the pipeline parallel hierarchy, k is the device number. Numbering at the local data parallelism level For the model number Forward computation of a layer This represents the layer number of the neural network. For equipment computing power For equipment Decision variables, For equipment micro-batch size, For the model number The reverse computation cost of the layer;

[0103] S212: Calculate the forward communication duration based on the divided task plane. and reverse communication duration :

[0104]

[0105]

[0106] in, This represents the current stage number of the pipeline. For equipment and equipment Data transfer volume between them , for The device's number at the local data parallelism level. For equipment Decision variables, The dividing point The amount of data to be transmitted Indicates equipment and equipment The actual bandwidth between Indicates equipment and equipment The actual bandwidth between;

[0107] The actual bandwidth is calculated as follows:

[0108]

[0109]

[0110]

[0111]

[0112] in, For equipment and equipment Physical bandwidth between For link The number of links sharing bandwidth. This represents the total number of task planes after the overall division. For equipment It accounts for the total number of pipelines. This represents the current stage number of the pipeline. The transmission occupancy variable is a 0-1 variable representing the link. Whether or not and Used for data transmission between; This is a shared link variable, a 0-1 variable, representing the link. With the second link Whether to share bandwidth For group ordinal numbers, The total number of groups, This is a shared variable for the link, a 0-1 variable representing the link. Are you in the group? Internal sharing; This is a shared variable for the second link, a 0-1 variable, representing the second link. Are you in the group? Internal sharing.

[0113] S213: Calculate total pipeline time :

[0114]

[0115]

[0116] in, For equipment Global data parallel processing time For equipment The pipeline phase takes time. The time taken for each stage For equipment It accounts for the total number of pipelines. For equipment Total number of occupancy stages For equipment The total number of task planes after partitioning.

[0117]

[0118]

[0119] in, The reverse communication duration for the next pipeline parallel layer.

[0120] Specifically, the process of converting the total time consumption into chromosomes is as follows: taking the smallest square in the divided task plane as the unit, the genetic material content is based on the information within the corresponding location point of the device;

[0121] The information within the location point corresponding to the device includes: device ID, total time consumed by the device, location point of the device, and micro-batch size of the device.

[0122] Specifically, step S3 includes:

[0123] S31: Set the chromosome as the initial solution population and set the number of cycles to 0;

[0124] S32: Use the tournament algorithm to select the initial population of special solutions;

[0125] S33: Use the roulette wheel algorithm to perform random crossover or mutation on the initial special solution population;

[0126] S34: Calculate the fitness of the initial special solution based on the initial special solution population after random crossover or mutation, generate a new population, and increment the cycle count by 1;

[0127] S35: Determine whether the loop count has reached the termination condition. If not, return to step S32.

[0128] If the goal is to output the individual with the highest fitness in the new population.

[0129] The process of converting the total time into chromosomes is as follows: taking the smallest square in the divided task plane as the unit, the genetic material content is based on the information in the corresponding location point of the device.

[0130] The information within the location point corresponding to the device includes: device ID, total time consumed by the device, location point of the device, and micro-batch size of the device.

[0131] Specifically, as shown in Table 1, Table 1 illustrates the differences between the method proposed in this invention and existing methods:

[0132] Table 1. Differences between the method proposed in this invention and existing methods.

[0133]

[0134] This invention constructs a training framework for scenarios using mesh pipelines and heterogeneous networks and computing devices, expands the search space for training process optimization, and solves the problem of automatic optimization of parallel training processes under heterogeneous devices.

[0135] Specifically, step S4 includes:

[0136] In each device, the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level to which the process group initialization algorithm belongs are selected. For the parameter gradients of all layers within the corresponding layer range of the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level of the device, AllReduce collection communication and parameter update are completed in the corresponding process group according to the layer number order.

[0137] As shown in Figure 5, Figure 5 is a block diagram of an automatic optimization system for distributed parallel training processes of a model, including:

[0138] Task plane partitioning module 101: Defines the task plane based on the communication and computing power of the neural network model, partitions the task plane, and obtains the partitioned task plane;

[0139] Time consumption calculation module 102: Constructs a training time consumption model based on the divided task plane to obtain the total time consumption;

[0140] Optimization Module 103: Based on the time consumption calculation module, a genetic algorithm is designed using the tournament algorithm to optimize the training time and obtain the training process optimization results;

[0141] Model training module 104: Trains the neural network model based on the training optimization results.

[0142] Figure 6 illustrates a schematic diagram of the physical structure of a computing node cluster. As shown in Figure 6, the cluster includes several computing nodes 860. Each computing node may include: a processor 810, multiple graphics processing units (GPUs) 850, a communication interface 820, a memory 830, a communication bus 840, and an inter-device network 870. The processor 810, GPUs 850, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The computing nodes 860 can communicate with each other via the communication interface 820 and the inter-device network 870. The processor 810 can call logical instructions in the memory 830 and control the GPUs 850 to execute corresponding computational operations to perform an automatic optimization method for a distributed parallel training process. This method includes:

[0143] S1: Define the task plane based on the communication and computing power of the neural network model, divide the task plane, and obtain the divided task plane;

[0144] S2: Construct a training time model based on the divided task plane to obtain the total time;

[0145] S3: Based on the total time consumption, use the tournament algorithm to design a genetic algorithm to solve the training time consumption model and obtain the training optimization results;

[0146] S4: Embed the training optimization results into the neural network model for optimization and updating.

[0147] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a group of computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0151] It should be noted that the embodiments of this disclosure can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0152] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0153] While this disclosure has been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the specific embodiments disclosed. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An automatic optimization method for distributed parallel training processes of models, characterized in that, include: S1: Define the task plane based on the communication and computing power of the neural network model, divide the task plane, and obtain the divided task plane; S2 Step S2 includes: S21: Calculate the total pipeline time based on the divided task planes; The training time model is constructed according to the divided task planes. Step S21 includes: S211: Calculate the forward computation time based on the divided task plane. and reverse calculation time ： in, For device ID, The total number of devices. For the current device The corresponding pipeline parallelism level in the model is the split point between the previous pipeline parallelism level and the previous pipeline parallelism level, where i is the device. In the global data parallelism index, j represents the device. In the pipeline parallel hierarchy, k is the device number. Numbering at the local data parallelism level For the model number Forward computation of a layer This represents the layer number of the neural network. For equipment computing power For equipment Decision variables, For equipment micro-batch size, For the model number S212: Calculate the forward communication duration based on the partitioned task plane. and reverse communication duration ： in, To find the maximum value function, This represents the current stage number of the pipeline. For equipment and equipment Data transfer volume between them ， for The device's number at the local data parallelism level. For equipment Decision variables, The dividing point The amount of data to be transmitted Indicates device and equipment The actual bandwidth between Indicates device and equipment The actual bandwidth between; S213: Calculate the total pipeline time. ： in, For equipment Global data parallel processing time For equipment The time consumed in each stage of the production line The time taken for each stage For equipment Total number of production lines For equipment Total number of occupancy stages For equipment The total number of task planes after partitioning. S22: Calculate the total reduction time based on the partitioned task plane, representing the reverse communication time of the next pipeline parallel layer; S23: Calculate the total time taken ： S3: Based on the total time consumption, use the tournament algorithm to design a genetic algorithm to solve the training time consumption model and obtain the training optimization results; S4: Embed the training optimization results into the neural network model for optimization and update.

2. The automatic optimization method for distributed parallel training process of a model according to claim 1, characterized in that, Step S1 includes: S11: Defining a task plane, the task plane including a global data parallelism level, a mesh pipeline parallelism level, and a local data parallelism level; S12: Using a process group initialization algorithm to divide the task plane to obtain the divided task plane.

3. The automatic optimization method for distributed parallel training process of a model according to claim 2, characterized in that, The process group initialization algorithm includes the following steps: S121: Input the training plan and total number of layers of the neural network model, and set the current split point to 0; S122: Initialize a process group; S123: Set the next device as the current device; S124: Determine whether the current split point is greater than or equal to the starting layer number of the current device and less than the ending layer number of the current device. If yes, proceed to step S125; otherwise, proceed to step S126; S125: Add the current device to the current process group; S126: Determine whether all devices have been traversed. If yes, proceed to step S127; otherwise, proceed to step S123; S127: Set the current process group layer range to [current split point, minimum ending layer number within the current process group], and add the current process group to the set of all process groups; S128: Determine whether the ending layer number of the current process group is equal to the total number of layers minus 1. If yes, proceed to step S129; otherwise, proceed to step S122; S129: Output the set of all process groups, where the set of all process groups is the partitioned task plane.

4. The automatic optimization method for distributed parallel training process of a model according to claim 1, characterized in that, The total time consumption is converted into chromosomes by the following process: taking the smallest square in the divided task plane as the unit, the information in the corresponding location point of the device is converted into genetic material content. The information within the location point corresponding to the device includes: device ID, total time consumed by the device, location point of the device, and micro-batch size of the device.

5. The automatic optimization method for distributed parallel training process of a model according to claim 4, characterized in that, Step S3 The process includes: S31: Setting the chromosome as the initial solution population and setting the loop count to 0; S32: Using the tournament algorithm to select the initial special solution population; S33: Using the roulette wheel algorithm to perform random crossover or mutation on the initial special solution population; S34: Calculating the fitness of the initial special solution based on the initial special solution population after random crossover or mutation, generating a new population, and incrementing the loop count by 1; S35: Determining whether the loop count has reached the termination condition. If not, returning to step S32; if it has, outputting the individual with the highest fitness in the new population.

6. The automatic optimization method for distributed parallel training process of a model according to claim 2, characterized in that, Step S4 includes: selecting the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level to which the process group initialization algorithm belongs in each device; for the parameter gradients of all layers within the corresponding layer number range of the global data parallelism level, mesh pipeline parallelism level, and local data parallelism level of the device, completing the full reduction set communication and parameter update in the corresponding process group according to the layer number order.

7. An automatic optimization system for distributed parallel training processes of a model, used to execute the automatic optimization method for distributed parallel training processes of a model as described in any one of claims 1 to 6, characterized in that, include: Task plane partitioning module: Defines the task plane based on the communication and computing power of the neural network model, and partitions the task plane to obtain the partitioned task plane; Time consumption calculation module: Constructs a training time consumption model based on the partitioned task plane to obtain the total time consumption; Optimization module: Based on the total time consumption, designs a genetic algorithm using the tournament algorithm to solve the training time consumption model and obtain the training optimization result; Model training module: Embeds the training optimization result into the neural network model for optimization and update.

8. An electronic device comprising a memory, a processor, a graphics computing unit, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the automatic optimization method for the model distributed parallel training process as described in any one of claims 1 to 6.

Images