Large model parallel training strategy generation method oriented to domestic supercomputing system

Abstract

According to the domestic supercomputing system-oriented large model parallel training strategy generation method, a parallel training strategy comprises at least one of a parallel training method, a batch size and a video memory optimization method, the parallel training strategy forms a search space, and the method comprises the following steps: acquiring parameters required for generating the parallel training strategy; based on the parameters, a plurality of decision trees are constructed, the decision trees are used for decomposing a search space of the parallel training strategy, and each decision tree is used for representing a sub-search space of the parallel strategy; based on the decision tree and the parameters, the model training method is evaluated through a performance evaluation method, and the calculation performance and the communication performance of the model under different parallel strategies are determined; constructing an integer programming model based on the calculation performance and the communication performance; and determining an optimal solution of the integer programming model, and taking a parallel training strategy corresponding to the optimal solution as a determined parallel training strategy. According to the method, the most convenient and rapid large model parallel training strategy can be generated for the domestic supercomputing system.

Description

Technical Field

[0001] The present invention relates to the technical field of large model training strategy generation, and in particular to a large model parallel training strategy generation method for a domestic supercomputer system. Background Art

[0002] Computer hardware performance continues to improve, deep learning algorithms are rapidly optimized, and the development of large models is changing with each passing day. A series of AI applications based on large models have been launched one after another. In November 2022, a company launched an AI conversational chatbot ChatGPT. Its outstanding natural language generation capabilities have attracted widespread attention from all over the world. It has exceeded 100 million users in 2 months, and a wave of large models has been set off at home and abroad. Representatives of large models include ChatGPT and subsequent GPT-4, Llama, PaLM, Spark Cognitive Large Model, Wenxin Yiyan Large Model, Pangu Large Model, Zidong Taichu Large Model, etc. The training of large models requires a lot of computing power and is economically expensive. Due to the technological blockade of Western countries and the ban on the sale of high-end chips, the computing power challenge has become more arduous. Although domestic chip manufacturers have developed a series of high-performance processors, such as Haiguang domestic accelerators, Huawei Ascend NPU and other AI processors, as well as Huawei Kunpeng, the new generation of Shenwei multi-core and other general-purpose processors, the current domestic large model ecosystem is still imperfect. How to make large models efficiently trained in domestic supercomputing environments remains to be solved and improved. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product including a computer program for generating a large-model parallel training strategy for a domestic supercomputer system, which can generate the most convenient and fast large-model parallel training strategy for a domestic supercomputer system.

[0004] In the first aspect, an embodiment of the present application provides a method for generating a large model parallel training strategy for a domestic supercomputer system, wherein the parallel training strategy includes at least one of a parallel training method, a batch size BatchSize, and a video memory optimization method, and the parallel training strategy constitutes a search space. The method includes: obtaining parameters required to generate a parallel training strategy; based on the parameters, constructing multiple decision trees, and the multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to characterize a sub-search space of the parallel strategy; based on the decision trees and parameters, the model training method is evaluated by a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies; based on the computing performance and communication performance, an integer programming model is constructed; the optimal solution of the integer programming model is determined, and the parallel training strategy corresponding to the optimal solution is used as the determined parallel training strategy.

[0005] In some possible implementations, the construction rules of the decision tree include: the height of the decision tree is the number of available parallel training methods; any parallel training method cannot be repeatedly applied to different levels of the decision tree; the degree of non-leaf nodes is increased from 2 to n Select from, n≥0.

[0006] In some possible implementations, the model training method is evaluated by a performance evaluation method based on a decision tree and parameters, including: a parallel training strategy search space decomposition step based on a decision tree, and using a decision tree to determine whether the parallel training strategy complies with experience and video memory constraints and BatchSize restrictions to reduce the size of the search space.

[0007] In some possible implementations, the parameters include the hidden layer dimension of the model, the number of attention heads, the sequence length, the number of model layers, and the set Batchsize interval.

[0008] In some possible implementations, the model training method is evaluated by a performance evaluation method, including: based on the GPT-3 model architecture, by combining a static formula with linear regression.

[0009] In some possible implementations, determining the computational performance of the model under different parallel strategies includes: obtaining the computational overhead of different parallel training strategies by simulating actual training conditions of the model.

[0010] In some possible implementations, the communication performance of the model under different parallel strategies is determined, including: based on the hardware structure of domestic supercomputers, determining the communication rate within and between nodes, and calculating the communication overhead in model training.

[0011] In some possible implementations, the integer programming model includes a video memory optimization method, and the video memory optimization method includes recalculation, zero redundancy, and memory transfer.

[0012] In some possible implementations, the objective function of the integer programming model is

[0013] arg min∑ T [O ck T ck (s i )+T cal (s i )+O z3 T z3 (s i )+O off T tran (s i )+T comm (s i )]

[0014] In the formula, s i represents the parallel strategy i, T represents the computing time, O ck Characterizes whether the parallel training strategy i uses recomputation and has O ck ∈{0,1},T ck Characterizes the additional computational overhead of the parallel training strategy when using recalculation, T cal Characterizes the forward and reverse computation time of parallel training strategy i when the video memory optimization method is not used, including the execution time of video memory intensive operators and memory access intensive operators, O z3 Characterizes whether the parallel training strategy i uses the zero-redundancy optimization technology of level three and has O z3 ∈{0,1},T z3 The time that represents the additional communication overhead when using the level 3 zero redundancy optimization technique is increased. off Characterizes whether parallel training strategy i uses memory transfer and has O off ∈{0,1},T tran Characterizes the additional time of memory transfer overhead, T comm represents the communication time of parallel strategy i when the video memory optimization method is not used. Among them, since the additional computational overhead of recalculation is consistent with the forward calculation time during model training when all model layers are recalculated, T ck It can be approximately equal to the forward calculation time of the model.

[0015] In the second aspect, an embodiment of the present application provides a device for generating a large-model parallel training strategy for a domestic supercomputer system, including: an acquisition module for acquiring parameters required for generating a parallel training strategy; a processing module for constructing multiple decision trees based on the parameters, and the multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to characterize a sub-search space of the parallel strategy; the processing module is also used to evaluate the model training method through a performance evaluation method based on the decision trees and parameters, and determine the computing performance and communication performance of the model under different parallel strategies; the processing module is also used to construct an integer programming model based on the computing performance and communication performance; the processing module is also used to determine the optimal solution of the integer programming model, and determine the parallel training strategy based on the optimal solution.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method described in any one of the first aspects.

[0017] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method described in any one of the first aspects is executed.

[0018] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program. When the computer program product runs on a processor, the processor executes the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a method for generating a large model parallel training strategy for a domestic supercomputer system provided in an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of an inter-GPU granularity decision tree provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of a decision tree of inter-GPU-intra-GPU granularity provided in an embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of a large-model parallel training strategy generation device for a domestic supercomputer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0026] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0027] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0028] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0029] To facilitate the understanding of the embodiments of the present application, the following will further explain and illustrate specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0030] First, the technical terms involved in the embodiments of the present application are introduced:

[0031] 1. Batch size (BatchSize) refers to the number of samples processed simultaneously in each iteration in machine learning and deep learning. Choosing a suitable BatchSize can help the model learn faster and avoid instability in the training process.

[0032] Next, the technical solution provided by the embodiments of the present application is introduced.

[0033] The current distributed parallel training strategies for large models mainly include data parallelism, tensor parallelism, pipeline parallelism, and hybrid expert parallelism. Among them, pipeline parallelism (PP) distributes different stages of the model to different devices to form a processing pipeline. It is a type of model parallelism (MP). Its core idea is to split the model. Data parallelism divides the data into multiple batches and processes them in parallel on multiple devices. Model parallelism distributes different parts of the model to different devices. Both pipeline linear parallelism and tensor parallelism divide the model itself, with the aim of training larger models using limited single-card video memory. Simply put, pipeline parallelism divides the model horizontally, that is, divides the model by layer; tensor parallelism divides the model vertically. There are two main methods for generating parallel training strategies. One is to use the experience of parallel system experts to manually set and optimize parallel training strategies, and the other is to use tools or frameworks to generate parallel training strategies. In the embodiment of the present application, the large model parallel training strategy generation method for the domestic supercomputer system mainly includes three parts, the first is a large model parallel training performance evaluation method for the domestic supercomputer, the second is a parallel training strategy search space decomposition method based on a decision tree, and the third is a parallel strategy generation algorithm based on integer programming. In the embodiment of the present application, the domestic supercomputer is described by taking the "Dongfang" supercomputer as an example, but the "Dongfang" supercomputer does not constitute a limitation to the present application.

[0034] For the performance evaluation method of large-model parallel training for domestic supercomputers, it includes a fast evaluation method for video memory overhead, a computing performance evaluation method, and a communication performance evaluation method. The fast evaluation method for video memory overhead is based on the GPT-3 model architecture, and combines static formulas with linear regression to achieve a more accurate evaluation of the video memory overhead of model training. The computing performance evaluation method obtains the computing overhead of different parallel training strategies under a single micro-batch by simulating the actual training situation of the model, and combines the formula obtained by analysis to obtain the computing overhead of the parallel training strategy under different BatchSizes. The communication performance evaluation method is based on the hardware structure of the "Dongfang" supercomputer, determines the communication rate within and between nodes, and combines the calculation amount formula to derive the communication overhead in model training.

[0035] For the parallel training strategy search space decomposition method based on decision tree, the method is based on decision tree and decomposes the search space in the search space including data parallel dimension, model parallel dimension, pipeline parallel dimension, BatchSize, whether to use recalculation, whether to use zero redundancy method and the level of corresponding zero redundancy method, and whether to use memory exchange (also known as memory transfer) method. According to whether the operation can be completed in a single GPU, the parallel training method is divided into inter-GPU granularity and intra-GPU granularity. Since data parallelism, pipeline parallelism and tensor parallelism cannot be completed in a single GPU, data parallelism, pipeline parallelism and tensor parallelism are used as parallel training methods of inter-GPU granularity. Since recalculation method, zero redundancy method and memory exchange method can all be operated inside the GPU, recalculation method, zero redundancy method and memory exchange method are used as parallel training methods of intra-GPU granularity. According to the defined decision tree construction method, inter-GPU granularity decision trees and inter-GPU-intra-GPU granularity decision trees are constructed, and the decision trees are used to determine whether the parallel training strategy meets the experience and video memory constraints and BatchSize restrictions, thereby reducing the size of the search space.

[0036] For the parallel strategy generation algorithm based on integer programming, the parallel strategy generation problem is transformed into an integer programming problem that minimizes the model training time overhead under a given global batch size, and the solver is used to solve it, thereby generating a high-throughput parallel training strategy. The performance evaluation method proposed in this method is applied to evaluate the generated parallel training strategy in the decomposed parallel training strategy search space, thereby obtaining the optimal parallel training strategy including the video memory optimization method under the current hardware environment and model parameter configuration.

[0037] For example, Figure 1 The flowchart of a method for generating a large model parallel training strategy for a domestic supercomputer system provided by an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps:

[0038] S11: Obtain the parameters required to generate a parallel training strategy.

[0039] In this embodiment, the computing device can read a pre-stored configuration file to obtain some parameters required for the generation strategy. The parameters may include the hidden layer dimension of the model, the number of attention heads, the sequence length, the number of model layers, etc., and may also include the BatchSize interval set by the user.

[0040] S12: construct multiple decision trees based on the acquired parameters, and the multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy.

[0041] In this embodiment, a decision tree is constructed based on the acquired parameters. The construction of the decision tree needs to follow certain construction rules, including: the height of the decision tree is the number of available parallel training methods, each decision tree represents a sub-search space; any parallel training method cannot be repeatedly applied to different levels of the decision tree; the degree of non-leaf nodes should be from {1,2,4,8,…,2 n}, n ≥ 0. In the context of parallel training strategies, the search space contains all possible parallelization methods and configurations. This may include different data parallelism, model parallelism, pipeline parallelism strategies, and different combinations of parameters such as BatchSize, whether to use recomputation, zero redundancy optimization, etc. Since the search space may be very large, directly searching for the optimal strategy may be very time-consuming or impractical. Therefore, the search space can be divided into smaller subspaces, each of which contains a group of strategies with similar characteristics or satisfying specific conditions, called sub-search spaces.

[0042] Based on this construction rule, a GPU inter-granularity decision tree without a video memory optimization method is first constructed. For example, Figure 2 Schematic diagram of a GPU inter-granularity decision tree provided in an embodiment of the present application. Figure 2 As shown in the figure, the yellow hollow circles represent pipeline parallelism, the green hollow circles represent data parallelism, and the blue hollow circles represent tensor parallelism. Taking four GPUs as an example, Figure 2 ① in the figure indicates the case where the number of pipeline parallel devices is 1, which corresponds to data parallelism of four devices, data parallelism of two devices and tensor parallelism of two devices, or tensor parallelism of four devices. Figure 2 ② in the figure indicates that the number of pipeline parallel devices is 2, which corresponds to data parallelism of two devices or tensor parallelism of two devices. Figure 2 ③ in the figure indicates that the number of pipeline parallel devices is 4, and all devices are pipelined in parallel. Figure 2 In the example, four GPUs are used, but the number of GPUs does not constitute a limitation to this application. It is understood that the number of GPUs is not fixed.

[0043] Secondly, based on the inter-GPU granularity decision tree, a decision tree for inter-GPU and intra-GPU granularity is constructed, and the video memory optimization method is used to determine whether the parallel training strategy can reach the BatchSize limit set by the user within the video memory limit. Video memory optimization methods may include methods such as recalculation optimization, zero redundancy optimization, and memory transfer optimization. Among them, recalculation is usually used to reduce video memory usage by recalculating certain intermediate results instead of storing them in the video memory. Zero redundancy optimization is used to reduce the amount of data transmitted between multiple devices. Memory transfer refers to moving data between different storage levels (such as video memory and main memory) to optimize performance and reduce video memory usage. Exemplarily, Figure 3 Schematic diagram of a decision tree of inter-GPU and intra-GPU granularity provided by an embodiment of the present application. Figure 3 As shown, the decision tree includes both between GPUs and within GPUs, such as Figure 2 The decision tree shown can further include recalculation, zero redundancy, and memory transfer on this basis to construct a further decision tree. Figure 3 ① in the figure indicates that the number of pipeline parallel devices is 1, which corresponds to data parallelism of four devices, data parallelism of two devices and tensor parallelism of two devices, or tensor parallelism of four devices. In the case of data parallelism of four devices, recalculation, zero redundancy, and memory transfer can be performed in each data parallelism. In the case of data parallelism of two devices and tensor parallelism of two devices, recalculation, zero redundancy, and memory transfer can be performed in each tensor parallelism. In the case of tensor parallelism of four devices, recalculation and memory transfer can be performed in each tensor parallelism. Figure 3 ② in the figure indicates that the number of pipeline parallel devices is 2, which corresponds to data parallelism of two devices or tensor parallelism of two devices. In the case of data parallelism of two devices, each data parallelism can include recalculation, zero redundancy, and memory transfer. In the case of tensor parallelism of two devices, each tensor parallelism can include recalculation and memory transfer. Figure 3 The number of pipeline parallel devices in shows the case where there are 4 pipeline parallel devices, and all devices are pipeline parallel. When all devices are pipeline parallel, pipeline parallelism includes recalculation and memory transfer. Pipeline parallelism does not include zero redundancy.

[0044] S13: Based on the constructed decision tree and the obtained parameters, the model training method is evaluated through a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies.

[0045] In this embodiment, based on the parallel training strategy search space decomposition step of the decision tree, the decision tree is used to determine whether the parallel training strategy meets the experience and video memory constraints and BatchSize restrictions to reduce the size of the search space. The constructed decision tree is used to screen the parallel training strategy. Each decision tree represents a sub-search space and defines a series of decision rules. The parallel strategy screened out by the decision tree is used to simulate the model training, and the computational overhead of different parallel training strategies is obtained by simulating the actual training of the model. . The simulation process does not require the actual operation of the model training, but estimates the possible performance of the model training under a given strategy. At this time, the parallel strategy needs to meet the video memory restrictions. For the parallel strategy that meets the video memory restrictions, the model training under the parallel strategy is simulated, and the computational time overhead in each iteration is recorded. The performance is characterized by the time overhead, and the higher the performance, the shorter the time required. In order to obtain reliable evaluation results, multiple iterative simulations are required and the average iteration time is calculated. This can smooth out the anomalies or fluctuations that may exist in individual iterations and provide a more accurate performance estimate. At the same time, it is also necessary to evaluate the model training time overhead under different BatchSizes according to the BatchSize interval set by the user. In this way, the optimal BatchSize can be determined to balance video memory usage and computational efficiency. After completing multiple iterations, the computational cost of model training under the parallel strategy is determined based on the average forward propagation computation time of each iteration in the simulated training and the average backward propagation computation time. The model training time overhead under different pipeline parallelism is calculated based on the computational overhead formula and the computational overhead of a single micro-batch training obtained by simulation. Micro-batches correspond one-to-one to pipeline parallelism, and a single micro-batch is a part of pipeline parallelism, such as Figure 2 P1 or P2 shown, etc. When there is no solution that meets the video memory limit and the BatchSize range specified by the user, the process ends.

[0046] S14: Construct an integer programming model based on computing performance and communication performance.

[0047] In this embodiment, an objective function is constructed by the calculation time data simulated in step S13, including the average calculation time of forward propagation and back propagation under different micro-batch sizes. This function can reflect the total cost of model training time. The smaller the total cost, the faster the model training speed, that is, the better the model parallel training strategy. Therefore, the problem of model parallel training strategy is transformed into the problem of solving the optimal solution of the objective function. Define decision variables in integer programming, which represent different parallel training strategy options. For example, whether to use a specific parallel technology, the size of BatchSize, whether to apply recalculation, etc. At the same time, according to hardware limitations (such as video memory limitations), model parameters and user-set conditions, establish constraints. These constraints can ensure that the solution found is feasible and meets the given restrictions. Optimization techniques such as recalculation, zero redundancy and memory transfer need to be considered in the model at the same time.

[0048] In some possible embodiments, an integer programming model is constructed by combining the computing performance of the model under different parallel strategies with the communication performance obtained by reading the configuration parameters, and its objective function is:

[0049] arg min∑ T [O ck T ck (s i )+T cal (s i )+O z3 T z3 (s i )+O off T tran (s i )+T comm (s i )]

[0050] In the formula, s i represents the parallel strategy i, T represents the computing time, O ck Characterizes whether the parallel training strategy i uses recomputation and has O ck ∈{0,1},T ck Characterizes the additional computational overhead of the parallel training strategy when using recalculation, T cal Characterizes the forward and reverse computation time of parallel training strategy i when the video memory optimization method is not used, including the execution time of video memory intensive operators and memory access intensive operators, O z3 Characterizes whether the parallel training strategy i uses the zero-redundancy optimization technology of level three and has O z3 ∈{0,1}, Tz3 represents the time added by the additional communication overhead when using the level 3 zero redundancy optimization technique, O off Characterizes whether parallel training strategy i uses memory transfer and has Ooff ∈{0,1},T tran Characterizes the additional time of memory transfer overhead, T comm represents the communication time of parallel strategy i when the video memory optimization method is not used. Among them, since the additional computational overhead of recalculation is consistent with the forward calculation time during model training when all model layers are recalculated, T ck It can be approximately equal to the forward calculation time of the model.

[0051] S15: Determine an optimal solution of the integer programming model, and determine a parallel training strategy based on the optimal solution.

[0052] In this embodiment, after the integer programming model is constructed, the integer programming solver can be used to find the optimal solution of the objective function under different fixed BatchSizes. The parallel strategy corresponding to the optimal solution is used as the parallel training strategy for the large model to be generated.

[0053] The above is a large model parallel training strategy generation method for domestic supercomputer systems provided in the embodiment of the present application, and a performance evaluation method is used to quickly and accurately evaluate the video memory overhead, computing performance and communication overhead under different parallel training strategies. Then, a decision tree-based search space decomposition method is adopted to divide the complex parallel training strategy space into subspaces, effectively reducing the number of strategies that need to be explored. Then, using the simulated calculation time data, an integer programming model is constructed, which aims to minimize the model training time under given hardware and model parameters. Finally, the optimal solution is found through the solver, and an efficient parallel training strategy is determined to achieve rapid training of large models on specific supercomputer hardware. This method combines theoretical analysis, simulation evaluation and mathematical optimization to ensure the high performance and feasibility of the parallel training strategy, and can generate the most convenient and fast large model parallel training strategy for domestic supercomputer systems, breaking through the technical blockade.

[0054] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0055] Based on the method in the above embodiment, the embodiment of the present application also provides a large model parallel training strategy generation device for domestic supercomputer systems.

[0056] For example, Figure 4 A large model parallel training strategy generation device for domestic supercomputer systems is shown. Figure 4 As shown, the large model parallel training strategy generation device 400 for the domestic supercomputer system may include: an acquisition module 401 and a processing module 402.

[0057] Wherein, the acquisition module 401 is used to obtain the parameters required for generating the parallel training strategy;

[0058] A processing module 402 is used to construct a plurality of decision trees based on the parameters, wherein the plurality of decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy;

[0059] The processing module 402 is further used to evaluate the model training method through a performance evaluation method based on the decision tree and the parameters, and determine the computing performance and communication performance of the model under different parallel strategies;

[0060] The processing module 402 is further used to construct an integer programming model based on the computing performance and the communication performance;

[0061] The processing module 402 is further used to determine the optimal solution of the integer programming model and determine the parallel training strategy based on the optimal solution.

[0062] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0063] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0064] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0065] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0066] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0067] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0068] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A method for generating large model parallel training strategies for domestic supercomputer systems, characterized in that: The parallel training strategy includes at least one of a parallel training method, a batch size BatchSize, and a video memory optimization method. The parallel training strategy constitutes a search space. The method includes: Get the parameters required to generate a parallel training strategy; Based on the parameters, construct a plurality of decision trees, wherein the plurality of decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy; Based on the decision tree and the parameters, the model training method is evaluated by a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies; Based on the computing performance and the communication performance, construct an integer programming model; An optimal solution of the integer programming model is determined, and a parallel training strategy corresponding to the optimal solution is used as the determined parallel training strategy.

2. The method according to claim 1, characterized in that The construction rules of the decision tree include: The height of the decision tree is the number of available parallel training methods; Any parallel training method cannot be repeatedly applied to different levels of the decision tree; The degree of non-leaf nodes ranges from 2 n Select from, n≥0.

3. The method according to claim 1, characterized in that The step of evaluating the model training method based on the decision tree and the parameters by a performance evaluation method includes: The parallel training strategy search space decomposition step is based on the decision tree. The decision tree is used to determine whether the parallel training strategy conforms to the experience and memory constraints and BatchSize restrictions to reduce the size of the search space.

4. The method according to claim 1, characterized in that: The parameters include the hidden layer dimension of the model, the number of attention heads, the sequence length, the number of model layers, and the set Batchsize interval.

5. The method according to claim 1, characterized in that The model training method is evaluated by a performance evaluation method, including: Based on the GPT-3 model architecture, it is carried out by combining static formula with linear regression.

6. The method according to claim 1, characterized in that Determining the computing performance of the model under different parallel strategies includes: obtaining the computing overhead of different parallel training strategies by simulating the actual training situation of the model.

7. The method according to claim 1, characterized in that Determine the communication performance of the model under different parallel strategies, including: based on the hardware structure of domestic supercomputers, determine the communication rate within and between nodes, and calculate the communication overhead in model training.

8. The method according to claim 1, characterized in that The integer programming model includes the video memory optimization method, which includes recalculation, zero redundancy and memory transfer.

9. The method according to claim 1, characterized in that: The objective function of the integer programming model is argmin∑ T [O ck T ck (s i )+T cal (s i )+O z3 T z3 (s i )+O off T tran (s i )+T comm (s i )] In the formula, s i represents the parallel strategy i, T represents the computing time, O ck Characterizes whether the parallel training strategy i uses recomputation and has O ck ∈{0,1}, Tck represents the additional time of the parallel training strategy when using the extra computational overhead of heavy calculation, T cal Characterizes the forward and reverse computation time of parallel training strategy i when the video memory optimization method is not used, including the execution time of video memory intensive operators and memory access intensive operators, O z3 Characterizes whether the parallel training strategy i uses the zero-redundancy optimization technology of level three and has O z3 ∈{0,1}, Tz3 represents the time added by the additional communication overhead when using the level 3 zero redundancy optimization technique, O off Characterizes whether parallel training strategy i uses memory transfer and has O off ∈{0,1}, Ttran represents the time added by the additional overhead of memory transfer, and Tcomm represents the communication time of parallel strategy i when the video memory optimization method is not used. Among them, since the additional computational overhead of recalculation is consistent with the forward calculation time during model training when all model layers are recalculated, T ck It can be approximately equal to the forward calculation time of the model.

10. A large model parallel training strategy generation device for domestic supercomputer systems, characterized in that: The device comprises: The acquisition module is used to obtain the parameters required to generate a parallel training strategy; A processing module, configured to construct a plurality of decision trees based on the parameters, wherein the plurality of decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy; The processing module is further used to evaluate the model training method through a performance evaluation method based on the decision tree and the parameters, and determine the computing performance and communication performance of the model under different parallel strategies; The processing module is further used to construct an integer programming model based on the computing performance and the communication performance; The processing module is also used to determine the optimal solution of the integer programming model and determine the parallel training strategy based on the optimal solution.

Images