A method for generating large model parallel training strategies for domestic supercomputing systems

Abstract

A method for generating a large-model parallel training strategy for domestic supercomputer systems. The parallel training strategy includes at least one of a parallel training method, a batch size, and a memory optimization method. The parallel training strategy constitutes a search space. The method includes: obtaining parameters required to generate the parallel training strategy; constructing multiple decision trees based on the parameters, each of which is used to decompose the search space of the parallel training strategy, with each decision tree representing a sub-search space of the parallel strategy; evaluating the model training strategy using a performance evaluation method based on the decision trees and parameters to determine the model's computational and communication performance under different parallel strategies; constructing an integer programming model based on the computational and communication performance; determining the optimal solution of the integer programming model, and using the parallel training strategy corresponding to the optimal solution as the determined parallel training strategy. This method can generate the most convenient and efficient large-model parallel training strategy for domestic supercomputer systems.

Description

Technical Field

[0001] The present invention relates to the technical field of large-model training strategy generation, and in particular to a large-model parallel training strategy generation method for domestic supercomputing systems. Background Art

[0002] With the continuous improvement of computer hardware performance and the rapid optimization of deep learning algorithms, the development of large models is advancing at a rapid pace. A series of AI applications based on large models have emerged. In November 2022, a company launched ChatGPT, an AI conversational chatbot. Its outstanding natural language generation capabilities attracted widespread attention worldwide, surpassing 100 million users within two months, sparking a wave of large models both domestically and internationally. Representative large models include ChatGPT and its subsequent predecessors, GPT-4, Llama, PaLM, the Spark Cognitive Large Model, the Wenxin Yiyan Large Model, the Pangu Large Model, and the Zidong Taichu Large Model. Training large models requires enormous computing power and is economically prohibitive. Western technological blockades and the ban on the sale of high-end chips have exacerbated the computing power challenge. Although domestic chip manufacturers have developed a series of high-performance processors, such as the Haiguang domestic accelerator, AI processors such as the Huawei Ascend NPU, and general-purpose processors such as the Huawei Kunpeng and the new generation Sunway multi-core processor, the current domestic large model ecosystem remains incomplete. The challenge of efficiently training large models on domestic supercomputing environments remains to be solved and improved. Summary of the Invention

[0003] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, computing device, computer storage medium and product containing a computer program for generating a large-model parallel training strategy for a domestic supercomputing system, which can generate the most convenient and fast large-model parallel training strategy for a domestic supercomputing system.

[0004] In the first aspect, an embodiment of the present application provides a method for generating a large-model parallel training strategy for a domestic supercomputing system. The parallel training strategy includes at least one of a parallel training method, a batch size BatchSize, and a video memory optimization method. The parallel training strategy constitutes a search space. The method includes: obtaining parameters required to generate a parallel training strategy; based on the parameters, constructing multiple decision trees, and the multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy; based on the decision trees and parameters, the model training method is evaluated by a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies; based on the computing performance and communication performance, an integer programming model is constructed; the optimal solution of the integer programming model is determined, and the parallel training strategy corresponding to the optimal solution is used as the determined parallel training strategy.

[0005] In some possible implementations, the construction rules of the decision tree include: the height of the decision tree is the number of available parallel training methods; any parallel training method cannot be repeatedly applied to different levels of the decision tree; the degree of non-leaf nodes is from 2 to n Select from, n≥0.

[0006] In some possible implementations, the model training method is evaluated based on the decision tree and parameters through a performance evaluation method, including: a parallel training strategy search space decomposition step based on the decision tree, and using the decision tree to determine whether the parallel training strategy meets the experience and memory constraints and BatchSize restrictions to reduce the search space size.

[0007] In some possible implementations, the parameters include the model's hidden layer dimensions, the number of attention heads, the sequence length, the number of model layers, and the set Batchsize interval.

[0008] In some possible implementations, the model training method is evaluated through a performance evaluation method, including: based on the GPT-3 model architecture, by combining static formulas with linear regression.

[0009] In some possible implementations, determining the computational performance of the model under different parallel strategies includes: obtaining the computational overhead of different parallel training strategies by simulating actual model training conditions.

[0010] In some possible implementations, the communication performance of the model under different parallel strategies is determined, including: based on the hardware structure of domestic supercomputers, determining the communication rates within and between nodes, and calculating the communication overhead in model training.

[0011] In some possible implementations, the integer programming model includes a memory optimization method, and the memory optimization method includes recalculation, zero redundancy, and memory transfer.

[0012] In some possible implementations, the objective function of the integer programming model is

[0013] arg min∑ T [O ck T ck (s i )+T cal (s i )+O z3 T z3 (s i )+O off T tran (s i )+T comm (s i )]

[0014] Where s i represents the parallel strategy i, T represents the computing time, O ck Characterizes whether the parallel training strategy i uses recomputation and has O ck ∈{0,1},T ck Characterizes the additional computational overhead of the parallel training strategy when using heavy computation, T cal Characterizes the forward and reverse computation time of parallel training strategy i when the memory optimization method is not used, including the execution time of memory-intensive operators and memory-access-intensive operators, O z3 Characterizes whether the parallel training strategy i uses the zero-redundancy optimization technology of level three and has O z3 ∈{0,1},T z3 Characterizes the time added by the additional communication overhead when using the level three zero redundancy optimization technology, O off Characterizes whether the parallel training strategy i uses memory transfer and has O off ∈{0,1},T tran Characterizes the time added by the additional overhead of memory transfer, T comm Characterizes the communication time of parallel strategy i when the video memory optimization method is not used. Among them, since the additional computational overhead of recalculation is consistent with the forward computation time during model training when all model layers are recalculated, T ck It can be approximately equal to the forward calculation time of the model.

[0015] In the second aspect, an embodiment of the present application provides a device for generating a large-model parallel training strategy for a domestic supercomputer system, including: an acquisition module for acquiring parameters required for generating a parallel training strategy; a processing module for constructing multiple decision trees based on the parameters, and the multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy; the processing module is also used to evaluate the model training method based on the decision tree and the parameters through a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies; the processing module is also used to construct an integer programming model based on the computing performance and communication performance; the processing module is also used to determine the optimal solution of the integer programming model, and determine the parallel training strategy based on the optimal solution.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the method as described in any one of the first aspects.

[0017] In a fourth aspect, an embodiment of the present application provides a computing device comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method as described in any one of the first aspects is executed.

[0018] In a fifth aspect, an embodiment of the present application provides a product comprising a computer program, which, when the computer program product runs on a processor, enables the processor to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a method for generating a large-model parallel training strategy for a domestic supercomputer system provided in an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of an inter-GPU granularity decision tree provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a decision tree for inter-GPU and intra-GPU granularity provided by an embodiment of the present application;

[0023] Figure 4 This is a structural diagram of a large-model parallel training strategy generation device for domestic supercomputing systems provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0026] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0027] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0028] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0029] To facilitate understanding of the embodiments of the present application, further explanation will be given below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0030] First, the technical terms involved in the embodiments of this application are introduced:

[0031] 1. Batch size: In machine learning and deep learning, batch size determines the number of samples processed simultaneously in each iteration. Choosing an appropriate batch size can help the model learn faster while avoiding instability during training.

[0032] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0033] Current distributed parallel training strategies for large models primarily include data parallelism, tensor parallelism, pipeline parallelism, and hybrid expert parallelism. Pipeline parallelism (PP) distributes different stages of a model to different devices, forming a processing pipeline. This is a type of model parallelism (MP). Its core concept is to partition the model. Data parallelism divides data into batches and processes them in parallel across multiple devices. Model parallelism distributes different parts of the model to different devices. Both pipeline parallelism and tensor parallelism partition the model itself, aiming to train larger models using the limited graphics memory of a single graphics card. Simply put, pipeline parallelism partitions the model horizontally, i.e., by layer; tensor parallelism partitions the model vertically. There are two main approaches to generating parallel training strategies: one leverages the experience of parallel system experts to manually set and optimize the parallel training strategy; the other utilizes tools or frameworks to generate the strategy. In the embodiment of the present application, the large-model parallel training strategy generation method for a domestic supercomputer system mainly includes three parts: the first is a large-model parallel training performance evaluation method for a domestic supercomputer, the second is a parallel training strategy search space decomposition method based on a decision tree, and the third is a parallel strategy generation algorithm based on integer programming. In the embodiment of the present application, the domestic supercomputer is described using the "Dongfang" supercomputer as an example, but the "Dongfang" supercomputer does not constitute a limitation of this application.

[0034] Performance evaluation methods for large-model parallel training on domestic supercomputers include a rapid evaluation method for video memory overhead, a computational performance evaluation method, and a communication performance evaluation method. The rapid evaluation method for video memory overhead is based on the GPT-3 model architecture and combines static formulas with linear regression to achieve a relatively accurate evaluation of the video memory overhead of model training. The computational performance evaluation method simulates actual model training to obtain the computational overhead of different parallel training strategies under a single micro-batch. Combined with the formula derived from the analysis, the computational overhead of the parallel training strategy under different batch sizes is then obtained. The communication performance evaluation method, based on the hardware structure of the "Dongfang" supercomputer, determines the communication rates within and between nodes and, combined with the computational cost formula, derives the communication overhead during model training.

[0035] The decision tree-based parallel training strategy search space decomposition method uses a decision tree to decompose the search space into a set of parameters, including the data parallelism dimension, model parallelism dimension, pipeline parallelism dimension, batch size, whether to use recomputation, whether to use zero-redundancy methods and their corresponding levels, and whether to use memory swapping (also known as memory transfer). Parallel training methods are categorized into inter-GPU and intra-GPU granularity based on whether they can be performed on a single GPU. Since data parallelism, pipeline parallelism, and tensor parallelism cannot be performed on a single GPU, they are considered as inter-GPU parallel training methods. Since recomputation, zero-redundancy methods, and memory swapping can all be performed within a GPU, they are considered as intra-GPU parallel training methods. Following the defined decision tree construction method, inter-GPU and inter-GPU-intra-GPU decision trees are constructed. The decision trees are used to determine whether the parallel training strategy complies with empirical and memory constraints, as well as the batch size limit, thereby reducing the search space.

[0036] For the integer programming-based parallel strategy generation algorithm, the parallel strategy generation problem is transformed into an integer programming problem that minimizes the model training time overhead under a given global batch size and solved using a solver to generate a high-throughput parallel training strategy. The performance evaluation method proposed in this method is applied to evaluate the generated parallel training strategies within the decomposed parallel training strategy search space, thereby determining the optimal parallel training strategy, including memory optimization, for the current hardware environment and model parameter configuration.

[0037] For example, Figure 1 The flowchart of a method for generating a large model parallel training strategy for a domestic supercomputer system provided by an embodiment of the present application is shown. Figure 1 As shown, the method may include the following steps:

[0038] S11: Obtain the parameters required to generate a parallel training strategy.

[0039] In this embodiment, the computing device can read a pre-stored configuration file to obtain some parameters required for the generation strategy. These parameters may include the model's hidden layer dimensions, the number of attention heads, the sequence length, the number of model layers, and the user-defined BatchSize range.

[0040] S12: Construct multiple decision trees based on the obtained parameters. The multiple decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel strategy.

[0041] In this embodiment, a decision tree is constructed based on the acquired parameters. The construction of the decision tree needs to follow certain construction rules, including: the height of the decision tree is the number of available parallel training methods, each decision tree represents a sub-search space; any parallel training method cannot be repeatedly applied to different levels of the decision tree; the degree of non-leaf nodes should be from {1, 2, 4, 8, ..., 2 n}, n ≥ 0. In the context of parallel training strategies, the search space contains all possible parallelization methods and configurations. This may include different data parallelism, model parallelism, pipeline parallelism strategies, as well as different combinations of parameters such as batch size, whether to use recomputation, zero-redundancy optimization, etc. Since the search space can be very large, directly searching for the optimal strategy may be very time-consuming or impractical. Therefore, the search space can be divided into smaller subspaces, each of which contains a group of strategies with similar characteristics or meeting certain conditions, called sub-search spaces.

[0042] Based on this construction rule, we first construct an inter-GPU granularity decision tree without the memory optimization method. For example, Figure 2 Schematic diagram of an inter-GPU granularity decision tree provided in an embodiment of the present application. Figure 2 As shown, the yellow hollow circles represent pipeline parallelism, the green hollow circles represent data parallelism, and the blue hollow circles represent tensor parallelism. Taking four GPUs as an example, Figure 2 ① in the figure indicates that the number of pipeline parallel devices is 1, which corresponds to data parallelism of four devices, or data parallelism of two devices and tensor parallelism of two devices, or tensor parallelism of four devices. Figure 2 ② in the figure indicates that the number of pipeline parallel devices is 2, which corresponds to data parallelism of two devices or tensor parallelism of two devices. Figure 2 ③ in the figure indicates that the number of parallel pipeline devices is 4, and all devices are pipelined in parallel. Figure 2 In the example, four GPUs are used, but the number of GPUs does not constitute a limitation to this application. It is understood that the number of GPUs is not fixed.

[0043] Secondly, based on the inter-GPU granularity decision tree, a decision tree for inter-GPU and intra-GPU granularity is constructed, and the video memory optimization method is used to determine whether the parallel training strategy can reach the BatchSize limit set by the user within the video memory limit. Video memory optimization methods may include methods such as recalculation optimization, zero redundancy optimization, and memory transfer optimization. Among them, recalculation is usually used to reduce video memory usage by recalculating certain intermediate results instead of storing them in video memory. Zero redundancy optimization is used to reduce the amount of data transferred between multiple devices. Memory transfer refers to moving data between different storage levels (such as video memory and main memory) to optimize performance and reduce video memory usage. For example, Figure 3 This is a schematic diagram of a decision tree for inter-GPU and intra-GPU granularity provided by an embodiment of the present application. Figure 3 As shown, the decision tree includes both between GPUs and within GPUs, such as Figure 2 The decision tree shown can further include recalculation, zero redundancy, and memory transfer to construct a further decision tree. Figure 3 ① indicates the case where the number of pipeline parallel devices is 1, corresponding to the possibility of data parallelism on four devices, data parallelism on two devices and tensor parallelism on two devices, or tensor parallelism on four devices. In the case of data parallelism on four devices, recomputation, zero redundancy, and memory transfer can be performed in each data parallel operation. In the case of data parallelism on two devices and tensor parallelism on two devices, recomputation, zero redundancy, and memory transfer can be performed in each tensor parallel operation. In the case of tensor parallelism on four devices, recomputation and memory transfer can be performed in each tensor parallel operation. Figure 3 The ② in the figure indicates that the number of pipeline parallel devices is two, corresponding to data parallelism on two devices or tensor parallelism on two devices. In the case of data parallelism on two devices, each data parallel operation can include recomputation, zero redundancy, and memory transfer. In the case of tensor parallelism on two devices, each tensor parallel operation can include recomputation and memory transfer. Figure 3 The number in the figure represents the case where there are four pipeline parallel devices, and all devices are pipelined in parallel. When all devices are pipelined in parallel, pipeline parallelism includes recalculation and memory transfer. Pipeline parallelism does not include zero redundancy.

[0044] S13: Based on the constructed decision tree and the obtained parameters, the model training method is evaluated through the performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies.

[0045] In this embodiment, a parallel training strategy search space decomposition step is performed based on a decision tree. The decision tree is used to determine whether the parallel training strategy complies with empirical and memory constraints, as well as the batch size limit, thereby reducing the search space size. The constructed decision tree is used to screen parallel training strategies. Each decision tree represents a sub-search space and defines a series of decision rules. Model training simulations are performed using the parallel strategies screened by the decision tree. By simulating actual model training, the computational costs of different parallel training strategies are determined. This simulation does not require actual model training, but rather estimates the potential performance of model training under a given strategy. The parallel strategy must comply with the memory limit. For parallel strategies that comply with the memory limit, model training is simulated under that strategy, and the computational time cost of each iteration is recorded. Performance is characterized by time cost; higher performance corresponds to shorter time required. To obtain reliable evaluation results, multiple iterations are simulated and the average iteration time is calculated. This smooths out any anomalies or fluctuations that may exist in individual iterations, providing a more accurate performance estimate. Furthermore, the model training time cost for different batch sizes is evaluated based on the batch size range set by the user. In this way, the optimal BatchSize can be determined to balance the use of video memory and computational efficiency. After completing multiple iterations, the computational cost of model training under the parallel strategy is determined based on the average forward propagation computation time of each iteration in the simulated training and the average backward propagation computation time. The model training time overhead under different pipeline parallelism is calculated based on the computational overhead formula and the computational overhead of a single micro-batch training obtained by simulation. Micro-batches correspond one-to-one to pipeline parallelism, and a single micro-batch is a part of the pipeline parallelism, such as Figure 2 P1 or P2 shown, etc. When there is no solution that meets the video memory limit and the user-specified BatchSize range, the process ends.

[0046] S14: Construct an integer programming model based on computing performance and communication performance.

[0047] In this embodiment, an objective function is constructed using the computational time data simulated in step S13, including the average computational time for forward propagation and backward propagation under different micro-batch sizes. This function can reflect the total cost of model training time. The smaller the total cost, the faster the model training speed, that is, the better the model parallel training strategy. Therefore, the problem of model parallel training strategy is transformed into the problem of solving the optimal solution of the objective function. Decision variables in integer programming are defined, which represent different parallel training strategy options. For example, whether to use a specific parallel technology, the size of BatchSize, whether to apply recalculation, etc. At the same time, constraints are established based on hardware limitations (such as video memory limitations), model parameters, and user-set conditions. These constraints can ensure that the solution found is feasible and meets given constraints. Optimization techniques such as recalculation, zero redundancy, and memory transfer also need to be considered in the model.

[0048] In some possible embodiments, an integer programming model is constructed by combining the computational performance of the model under different parallel strategies with the communication performance obtained by reading the configuration parameters, and its objective function is:

[0049] arg min∑ T [O ck T ck (s i )+T cal (s i )+O z3 T z3 (s i )+O off T tran (s i )+T comm (s i )]

[0050] Where s i represents the parallel strategy i, T represents the computing time, O ck Characterizes whether the parallel training strategy i uses recomputation and has O ck ∈{0,1},T ck Characterizes the additional computational overhead of the parallel training strategy when using heavy computation, T cal Characterizes the forward and reverse computation time of parallel training strategy i when the memory optimization method is not used, including the execution time of memory-intensive operators and memory-access-intensive operators, O z3 Characterizes whether the parallel training strategy i uses the zero-redundancy optimization technology of level three and has O z3 ∈{0,1}, Tz3 represents the time added by the additional communication overhead when using the level three zero redundancy optimization technology, O off Characterizes whether the parallel training strategy i uses memory transfer and has Ooff ∈{0,1},T tran Characterizes the time added by the additional overhead of memory transfer, T comm Characterizes the communication time of parallel strategy i when the video memory optimization method is not used. Among them, since the additional computational overhead of recalculation is consistent with the forward computation time during model training when all model layers are recalculated, T ck It can be approximately equal to the forward calculation time of the model.

[0051] S15: Determine an optimal solution of the integer programming model, and determine a parallel training strategy based on the optimal solution.

[0052] In this embodiment, after the integer programming model is constructed, the integer programming solver can be used to find the optimal solution of the objective function under different fixed batch sizes. The parallel strategy corresponding to the optimal solution is used as the parallel training strategy for the large model to be generated.

[0053] The above is a method for generating a large-model parallel training strategy for a domestic supercomputer system provided in an embodiment of the present application. The performance evaluation method is used to quickly and accurately evaluate the memory overhead, computing performance, and communication overhead under different parallel training strategies. Next, a search space decomposition method based on a decision tree is adopted to divide the complex parallel training strategy space into subspaces, effectively reducing the number of strategies that need to be explored. Then, an integer programming model is constructed using simulated computing time data, which aims to minimize the model training time under given hardware and model parameters. Finally, the optimal solution is found through the solver, and an efficient parallel training strategy is determined to achieve rapid training of large models on specific supercomputer hardware. This method combines theoretical analysis, simulation evaluation, and mathematical optimization to ensure the high performance and feasibility of the parallel training strategy. It can generate the most convenient and fast large-model parallel training strategy for domestic supercomputer systems and break through technical blockades.

[0054] It is understandable that the size of the sequence number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. All or part of any features of any embodiment of the present application can be freely and arbitrarily combined without contradiction. The combined technical solution is also within the scope of the present application.

[0055] Based on the method in the above embodiment, the embodiment of the present application also provides a large-model parallel training strategy generation device for domestic supercomputing systems.

[0056] For example, Figure 4 A large-scale model parallel training strategy generation device for domestic supercomputer systems is shown. Figure 4 As shown, the large-model parallel training strategy generation device 400 for domestic supercomputer systems may include: an acquisition module 401 and a processing module 402.

[0057] Wherein, the acquisition module 401 is used to obtain the parameters required to generate the parallel training strategy;

[0058] A processing module 402 is configured to construct a plurality of decision trees based on the parameters, wherein the plurality of decision trees are configured to decompose the search space of the parallel training strategy, and each decision tree is configured to represent a sub-search space of the parallel strategy;

[0059] The processing module 402 is further configured to evaluate the model training method based on the decision tree and the parameters using a performance evaluation method to determine the computing performance and communication performance of the model under different parallel strategies;

[0060] The processing module 402 is further configured to construct an integer programming model based on the computing performance and the communication performance;

[0061] The processing module 402 is further configured to determine an optimal solution to the integer programming model and determine a parallel training strategy based on the optimal solution.

[0062] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0063] Based on the methods in the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the methods in the above embodiments.

[0064] Based on the methods in the above embodiments, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the methods in the above embodiments.

[0065] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0066] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0067] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0068] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

Claims

1. A method for generating large-model parallel training strategies for domestic supercomputer systems, characterized in that: The parallel training strategy includes at least one of a parallel training method, a batch size BatchSize, and a video memory optimization method. The parallel training strategy constitutes a search space. The method includes: Get the parameters required to generate a parallel training strategy; Based on the parameters, constructing a plurality of decision trees, wherein the plurality of decision trees are used to decompose the search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel training strategy; Based on the decision tree and the parameters, the model training method is evaluated by a performance evaluation method to determine the computing performance and communication performance of the model under different parallel training strategies; Based on the computing performance and the communication performance, an integer programming model is constructed; the integer programming model includes a video memory optimization method, the video memory optimization method includes recalculation, zero redundancy and memory transfer, and the objective function of the integer programming model is Where, Characterize the parallel training strategy i, Characterize the calculation time, Characterizes whether the parallel training strategy i uses recomputation and has , Characterizes the additional computational overhead of parallel training strategies when using heavy computation, Characterize the forward and reverse computation time of parallel training strategy i when the memory optimization method is not used, including the execution time of memory-intensive operators and memory-access-intensive operators, Characterizes whether the parallel training strategy i uses the level 3 zero-redundancy optimization technology and has , Characterizes the time added by the additional communication overhead when using the level 3 zero-redundancy optimization technique, Characterizes whether the parallel training strategy i uses memory transfer and has , Characterizes the time added by the additional overhead of memory transfer, Characterizes the communication time of parallel training strategy i when the video memory optimization method is not used; An optimal solution of the integer programming model is determined, and a parallel training strategy corresponding to the optimal solution is used as the determined parallel training strategy.

2. The method according to claim 1, characterized in that The construction rules of the decision tree include: The height of the decision tree is the number of available parallel training methods; Any parallel training method cannot be applied repeatedly to different levels of the decision tree; The degree of non-leaf nodes is Select from, n≥0.

3. The method according to claim 1, characterized in that The step of evaluating the model training method based on the decision tree and the parameters by a performance evaluation method includes: The parallel training strategy search space decomposition step is based on the decision tree. The decision tree is used to determine whether the parallel training strategy meets the experience and memory constraints and BatchSize restrictions, so as to reduce the search space size.

4. The method according to claim 1, wherein The parameters include the model's hidden layer dimensions, the number of attention heads, the sequence length, the number of model layers, and the set Batchsize interval.

5. The method according to claim 1, wherein The model training method is evaluated by the performance evaluation method, including: Based on the GPT-3 model architecture, it is performed by combining static formulas with linear regression.

6. The method according to claim 1, characterized in that Determining the computational performance of the model under different parallel training strategies includes: obtaining the computational overhead of different parallel training strategies by simulating actual training conditions of the model.

7. The method according to claim 1, characterized in that Determine the communication performance of the model under different parallel training strategies, including: based on the hardware structure of domestic supercomputers, determine the communication rate within and between nodes, and calculate the communication overhead during model training.

8. A large-model parallel training strategy generation device for domestic supercomputer systems, characterized in that: The device comprises: The acquisition module is used to obtain the parameters required to generate the parallel training strategy; A processing module, configured to construct a plurality of decision trees based on the parameters, wherein the plurality of decision trees are used to decompose a search space of the parallel training strategy, and each decision tree is used to represent a sub-search space of the parallel training strategy; The processing module is further configured to evaluate the model training method based on the decision tree and the parameters using a performance evaluation method to determine the computing performance and communication performance of the model under different parallel training strategies; The processing module is further used to construct an integer programming model based on the computing performance and the communication performance; the integer programming model includes a video memory optimization method, the video memory optimization method includes recalculation, zero redundancy and memory transfer, and the objective function of the integer programming model is Where, Characterize the parallel strategy i, Characterize the calculation time, Characterizes whether the parallel training strategy i uses recomputation and has , Characterizes the additional computational overhead of parallel training strategies when using heavy computation, Characterize the forward and reverse computation time of parallel training strategy i when the memory optimization method is not used, including the execution time of memory-intensive operators and memory-access-intensive operators, Characterizes whether the parallel training strategy i uses the level 3 zero-redundancy optimization technology and has , Characterizes the time added by the additional communication overhead when using the level 3 zero-redundancy optimization technique, Characterizes whether the parallel training strategy i uses memory transfer and has , Characterizes the time added by the additional overhead of memory transfer, Characterizes the communication time of parallel strategy i when the video memory optimization method is not used; The processing module is further configured to determine an optimal solution to the integer programming model and determine a parallel training strategy based on the optimal solution.

Images