A method and device for allocating computing power for large model training

Abstract

The present disclosure relates to a computing power allocation method and device for large model training, the method comprising: obtaining a to-be-trained large model and a training data set, and obtaining current GPU resources; constructing a cluster computing power resource allocation model; wherein the cluster computing power resource allocation model comprises a constraint condition and a target function; obtaining the input of the cluster computing power resource allocation model based on the to-be-trained large model, the training data set and the current GPU resources, and outputting a decision variable and an optimal target value through the cluster computing power resource allocation model; wherein the decision variable comprises tensor parallelism, pipeline parallelism, data parallelism, batch quantity and pipeline micro-batch size, and the optimal target value comprises minimizing the training time of the to-be-trained large model. In this way, computing power resources can be reasonably allocated from the current GPU resources for training the to-be-trained large model, so that the training time can be minimized while meeting the training of the to-be-trained large model through the training data set, thereby improving the efficiency of large model training.

Description

Technical Field

[0001] This disclosure relates to the field of large model training technology, and in particular to a method and apparatus for allocating computing power for large model training. Background Technology

[0002] With the explosive development of large-scale model application technologies, intelligent computing power has become an important driving force for social development. GPUs (Graphics Processing Units), as a typical representative of intelligent computing chips, provide an important infrastructure for the computing power required for training and inference of artificial intelligence models.

[0003] Currently, due to the continuous development of large-scale model technology, the complexity of model structures and the number of parameters are constantly increasing, facing the problem that the computing power of a single GPU chip or a single server node cannot meet the training and inference tasks of large models. Especially during the pre-training of large models, large-scale GPU cluster distributed technology is usually used to allocate computing power to complete model training and reduce the training time of large models.

[0004] When allocating computing power to GPU clusters for large model training, related technologies mainly rely on experimental methods and experience with training parameters, lacking attention to networking schemes and the details of interconnection technologies between GPUs. This results in a lack of quantitative evaluation of training time and training performance bottlenecks, leading to low training efficiency of large models on GPU clusters. Summary of the Invention

[0005] This disclosure provides a method and apparatus for allocating computing power for training large models.

[0006] According to a first aspect of this disclosure, a method for allocating computing power for training large models is provided, the method comprising:

[0007] Obtain the large model to be trained and the training dataset, and obtain the current GPU resources;

[0008] A cluster computing power resource allocation model is constructed; wherein, the cluster computing power resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints;

[0009] The cluster computing power resource allocation model is obtained based on the large model to be trained, the training dataset, and the current GPU resources, and the decision variables and optimal target values ​​are output through the cluster computing power resource allocation model. The decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch size, and pipeline micro-batch size. The optimal target value includes minimizing the training time of the large model to be trained.

[0010] According to a second aspect of this disclosure, a computing power allocation device for large model training is provided, the device comprising:

[0011] The data acquisition module is used to acquire the large model to be trained and the training dataset, as well as the current GPU resources;

[0012] The model building module is used to build a cluster computing resource allocation model; wherein, the cluster computing resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints.

[0013] The computing resource allocation module is used to obtain the input of the cluster computing resource allocation model based on the large model to be trained, the training dataset, and the current GPU resources, and output decision variables and optimal target values ​​through the cluster computing resource allocation model; wherein, the decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch processing quantity, and pipeline micro-batch size, and the optimal target value includes minimizing the training time of the large model to be trained.

[0014] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0016] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the methods described above.

[0017] The computing power allocation method and apparatus for large model training provided in this disclosure acquire the large model to be trained and the training dataset, and obtain the current GPU resources; construct a cluster computing power resource allocation model; wherein the cluster computing power resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch size, and total training time constraints; the input of the cluster computing power resource allocation model is obtained based on the large model to be trained, the training dataset, and the current GPU resources, and the decision variables and optimal objective value are output through the cluster computing power resource allocation model; wherein the decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch size, and pipeline micro-batch size, and the optimal objective value includes minimizing the training time of the large model to be trained. This allows for the rational allocation of computing power resources from the current GPU resources to train the large model to be trained, enabling the large model to be trained to be trained using the training dataset while minimizing the training time and improving the efficiency of large model training. Attached Figure Description

[0018] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A flowchart illustrating a computational power allocation method for training large models, provided as an exemplary embodiment of this disclosure;

[0020] Figure 2 A schematic diagram of a cluster computing power resource allocation model provided as an exemplary embodiment of this disclosure;

[0021] Figure 3 A schematic diagram of the performance calculation process for key indicators of a GPU cluster provided as an exemplary embodiment of this disclosure;

[0022] Figure 4 A schematic diagram illustrating the calculation process of key performance indicators for large model training provided in this embodiment of the disclosure;

[0023] Figure 5 A schematic block diagram of the functional modules of a computing power allocation device for large model training provided as an exemplary embodiment of this disclosure;

[0024] Figure 6 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure;

[0025] Figure 7 A block diagram of a computer system provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0034] With the explosive development of large-scale model application technology, intelligent computing power has become an indispensable key element for the development of countries, industries, and enterprises. As a new productive force, it injects new momentum into the digital transformation of various industries and is an important driving force for high-quality economic and social development.

[0035] GPUs, as crucial intelligent computing chips, provide essential infrastructure for the computing power required for training and inference of artificial intelligence models. However, due to the diverse range of GPU chip manufacturers and technical standards, and the fact that GPU chip technology is still rapidly evolving, intelligent computing centers face numerous critical challenges when building and deploying large-scale GPU computing clusters.

[0036] Currently, large-scale model technology continues to develop, with model structure complexity and parameter volume constantly increasing. This presents a challenge as the computing power of a single GPU chip or server node cannot meet the training and inference requirements of large models. Especially during large model pre-training, large-scale GPU clusters with distributed computing power are typically used to allocate computing power and complete model training, reducing training time. Leading companies have begun building GPU clusters with over 10,000 GPUs to construct large models with over 100 billion parameters. However, improving the utilization rate of GPU cluster computing power is a complex issue. It is influenced by key technologies such as cluster parameter networking, communication algorithms, single GPU performance, single-server GPU interconnection, and multi-server GPU interconnection, as well as by various factors including the large model structure, model training hyperparameter design, and distributed training methods. Current GPU cluster computing power allocation methods for large model training primarily rely on experimental approaches and empirical training parameters. For experts training large models, less attention is paid to networking schemes and the details of interconnection technology between GPUs. The main focus is on adjusting the hyperparameters of large model training. Multiple sets of comparative experiments are used to pre-train small batches of training data, collect data such as GPU utilization and training time from multiple sets of experiments, select the optimal hyperparameters, and carry out subsequent large model training tasks.

[0037] When building GPU cluster computing power in intelligent computing centers, the network configuration recommendations from chip manufacturers are crucial. Therefore, the network parameters from different manufacturers are somewhat dependent on the GPU chip, and there is a lack of technology for heterogeneous chip networking. While intelligent computing center cluster construction technologies based on relevant chips are relatively mature and closed, domestic solutions for cluster computing power currently face several technical challenges and require further improvement and optimization.

[0038] Therefore, the relevant technologies mainly have the following technical problems:

[0039] (1) Existing large-scale model training mainly uses pre-batch training data with different training hyperparameters to collect key performance data of large-scale model training in comparative experimental groups. However, quantitative analysis methods and tools are lacking. In limited comparative experimental groups, the optimal parameter combination can only be selected within a limited group. It may be that the selected training scheme reaches a local optimum, but it is not necessarily the optimal training parameter combination scheme.

[0040] (2) Currently, the design and optimization of distributed training schemes for large models are mainly based on experience, and there is a lack of quantitative evaluation of training time and training performance bottlenecks. In particular, there is a lack of comprehensive evaluation methods that combine networking, GPU interconnection, key performance parameters of GPU chips, and communication algorithms. Model training experience lacks a deep understanding of technologies such as networking and GPU interconnection, and these technologies, in turn, lack guidance for model training.

[0041] (3) For different GPU clusters, especially for domestically produced chip clusters that rely on large model training, there is a lack of quantitative evaluation methods for GPU chip performance and GPU cluster performance. When conducting distributed training for different types of large models, different distributed training parameters may encounter limiting factors such as communication walls, memory walls, and high computing power, which may prevent the GPU cluster from improving the training speed of large models. There is a lack of a predictive evaluation method that can accurately assess the performance of GPU clusters and analyze the reasons for the limitations of GPU cluster performance.

[0042] Therefore, based on the refined modeling and calculation of key performance indicators of GPU clusters, this embodiment of the present disclosure constructs a GPU intelligent computing cluster computing power allocation method, which can automatically recommend distributed training schemes for large models, balance GPU computing power performance, GPU memory, on-server GPU interconnect performance and cross-server GPU interconnect performance, achieve the goal of minimizing training time under limited GPU cluster resources, reduce the training time of large models, automatically optimize distributed training schemes for large models, and further provide suggestions for improving GPU cluster model training efficiency by combining the calculation results of key performance indicators of GPU clusters and the optimal distributed training scheme for large models.

[0043] Furthermore, by evaluating key performance indicators of computing GPU chips, GPU card interconnection within servers, and GPU card interconnection across servers, the embodiments can predict the actual computing power achievable by GPU chips and GPU clusters for large model training without the need for pre-training small batches of models. The intermediate results of the calculation of achievable computing power can be used to analyze the main reasons for the limitations of GPUs or GPU clusters, and can effectively evaluate the key performance indicators of GPU chips and GPU clusters that can be optimized.

[0044] This implementation combines distributed training of large models with a refined assessment of the computing power and GPU memory required for training large models, based on GPU performance metrics, intra-server GPU-to-GPU interconnection bandwidth, and cross-server GPU-to-GPU interconnection bandwidth. It estimates the minimum number of GPU cluster resources needed for model training and predicts the training time for large models with limited training cycles and training data volumes. Particularly for performance testing of domestically produced GPU chips and GPU clusters, the GPU cluster key performance metric calculation model provides a more objective and systematic evaluation of GPU cluster performance.

[0045] Specifically, this disclosure first provides a method for allocating computing power for training large models, such as... Figure 1 As shown, the method may include the following steps:

[0046] In step S110, the large model to be trained and the training dataset are obtained, and the current GPU resources are obtained.

[0047] In this embodiment, the large model to be trained refers to a large model that needs to be trained, and the large model to be trained is trained using a training dataset. The current GPU resources can be GPU resources in a GPU cluster, used to allocate resources from the current GPU resources for training the large model to be trained.

[0048] In step S120, a cluster computing power resource allocation model is constructed.

[0049] The cluster computing resource allocation model includes constraints and an objective function. The constraints include: GPU cluster resource constraints, individual GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints. These constraints and the objective function will be described in detail in subsequent embodiments.

[0050] In step S130, the input of the cluster computing power resource allocation model is obtained based on the large model to be trained, the training dataset, and the current GPU resources, and the decision variables and the optimal target value are output through the cluster computing power resource allocation model.

[0051] The decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch size, and pipeline micro-batch size. The optimal objective value is to minimize the training time of the large model to be trained.

[0052] The embodiment constructs a cluster computing power resource allocation model by obtaining the data input values ​​corresponding to the cluster computing power resource allocation model through the large model to be trained, the training dataset, and the current GPU resources. By minimizing the training time of the large model to be trained, the decision variables of the large model to be trained are obtained. If the currently available GPU resources can be used to train the large model to be trained, computing power resources can be reasonably allocated from the current GPU resources to train the large model to be trained. This allows the large model to be trained to be trained to be trained through the training dataset while minimizing the training time and improving the efficiency of large model training.

[0053] For details, see Figure 2 As shown, Figure 2 This is a schematic diagram of the cluster computing power resource allocation model structure provided in an embodiment of this disclosure. In this embodiment, the cluster computing power resource allocation model includes: a GPU intelligent computing cluster key performance indicator module 10, a large model training key performance indicator module 20, and a GPU cluster computing power allocation module 30 for large model training.

[0054] The GPU intelligent computing cluster key performance indicator module 10 is used to calculate key performance indicators in the GPU intelligent computing cluster.

[0055] Among them, the GPU intelligent computing cluster key performance indicator module 10 includes: GPU chip performance indicator calculation module 11, server-inter-GPU card interconnection module 12, parameter plane networking and cross-GPU card interconnection module 13, and communication algorithm key performance indicator calculation module 14.

[0056] The Large Model Training Key Performance Indicators Module 20 is used to calculate the key indicators of the large model to be trained in distributed training.

[0057] Among them, the key performance indicator module 20 for large model training includes: key performance indicator calculation module 21 for large model training, key performance indicator calculation module 22 for GPU cluster distributed training, and key performance indicator calculation module 23 for GPU cluster distributed training communication.

[0058] The GPU cluster computing power allocation module 30 for large model training is used to allocate computing power resources to the large model to be trained by optimizing the objective function under constraints.

[0059] In this embodiment, based on the above modules, a GPU cluster computing power resource allocation model for large model training is constructed. Under the constraints of GPU cluster resources, the large model distributed training scheme is optimized, the large model training time is reduced, and a computing power allocation scheme for large model distributed training is formed.

[0060] The embodiments provided in this disclosure can automatically estimate key GPU performance indicators, server-wide GPU inter-card interconnect performance indicators, cross-server GPU inter-card interconnect performance indicators, and communication algorithm performance indicators. Based on the selected large model, it automatically assesses the memory usage, computational load, and computational intensity required for large model training. It supports the calculation of average GPU memory usage, average computational load, computational intensity, communication mode, communication volume, number of communication communications, and latency in distributed training modes for large models, thereby estimating the large model training computation time. Based on the above-mentioned key indicator estimation, it can implement a computing power allocation method for large model training, allocating GPU cluster resource nodes to large model training tasks, improving GPU resource utilization, reducing training time, and thus improving the usability of GPU clusters for large model training.

[0061] Based on the above embodiments, the key performance indicators in the above GPU intelligent computing cluster include: GPU graphics card parameter performance indicators, GPU card interconnection topology indicators within the server, and GPU card interconnection indicators across servers.

[0062] In this embodiment, the GPU graphics card performance parameters include: data read time, computation time, computation intensity, and available GPU computing power; wherein, the data read time is determined based on the ratio between the amount of data read and written by the GPU and the chip bandwidth, the computation time is determined based on the ratio between the number of floating-point operations of the GPU and the peak computing power of the chip, the computation intensity is determined based on the ratio between the total number of floating-point operations and the amount of memory accesses, and the available GPU computing power is determined based on the peak computing power, computation intensity, and chip bandwidth of the chip.

[0063] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the performance calculation process for key metrics of a GPU cluster provided in this embodiment. The performance calculation process for key metrics of a GPU cluster mainly includes the following steps:

[0064] Step S310: Calculate GPU performance metrics.

[0065] In this embodiment, the main factors affecting GPU performance are peak computing power g. flops Chip bandwidth g bd And the amount of GPU memory (m). Large model training and inference require a certain amount of GPU memory. Due to the limitation of GPU memory size, a single GPU may not be able to complete the training of large models. It is necessary to use distributed training technologies such as data parallelism, model parallelism or tensor parallelism of intelligent computing clusters to solve the problem of insufficient GPU memory while accelerating the training of large models.

[0066] The amount of memory on a single chip limits whether model training or inference can be completed on a single GPU chip, and the peak computing power (g) is limited to a certain extent. flops Chip bandwidth g bdThis directly impacts the chip's actual computing performance. GPUs possess asynchronous data read / write and computation capabilities, with data read time being t. m (Equation 1-1), i.e., the amount of data read and written m vol With chip bandwidth g bd The ratio of .

[0067] t m =m vol / g bd (Equation 1-1)

[0068] The calculation time is expressed as t. c (Equation 1-2) is used to calculate the floating-point operand m. ops With chip computing power g flops The ratio of (number of floating-point operations per second).

[0069] t c =m ops / g flops (Equation 1-2)

[0070] The actual total time is t = max(t m , t c The computing power performance of a single GPU is affected by the chip's communication bandwidth and peak chip computing power.

[0071] Arithmetic intensity (I) is defined as the total number of floating-point operations (m) during program execution. ops Divide by the amount of memory accessed, m vol ,Right now:

[0072] I = m ops / m vol (Unit: FLOPs / BYTE) (Equation 1-3)

[0073] If the computational strength of the program I > g flops / g bd Then the program's calculation time is t = t c At this point, the program's execution time is limited by the chip's peak computing power; conversely, t = t m The runtime is limited by the chip's bandwidth. Therefore, in large model training tasks, the training time is related to the peak computing power and bandwidth of the GPU chip. The Roofline model can better explain the calculation method of the available computing power of the GPU, which is represented by g. util ,Right now:

[0074] g util =min(g flops , I×g bd (Equation 1-4)

[0075] Based on the correspondence between the available computing power and computing intensity of the chip generated by the Roofline model in this disclosure embodiment, it can be seen that the memory bandwidth g can be represented by the slope of a line segment. bd When the computational intensity I of the model training or inference task is less than the maximum computational intensity I max At that time, the available computing power of the GPU is g util Below peak computing power g flops When the calculated intensity I is greater than the maximum calculated intensity I max At that time, the available computing power of the GPU is g util =g flops .

[0076] Step S320: Interconnection between GPU cards within the server.

[0077] The above embodiments analyze key GPU performance indicators, including peak GPU computing power, memory bandwidth, memory usage, and model training and induction computation intensity. Since it is nearly impossible to train large models with all parameters using only a single GPU, distributed training with GPU clusters is often employed, including multi-GPU interconnects within servers and multi-GPU interconnects across servers to provide cluster computing power.

[0078] This step in the embodiment mainly focuses on GPU interconnect technology within servers. Commonly used GPU interconnect methods include PCIe, NVlink, and NVSwitch interconnects. Domestic chip manufacturers have also developed interconnect technologies based on the characteristics of their chips. Furthermore, there are various GPU interconnect topologies, with commonly used topologies including full interconnect and mesh cube interconnect.

[0079] In the embodiments provided in this disclosure, regardless of the GPU topology, the communication bandwidth between any two GPUs can be described using a communication bandwidth matrix. For n GPUs within a node, the communication bandwidth matrix is ​​represented as follows:

[0080]

[0081] Among them, b i,j This represents the communication bandwidth between GPU i and GPU j. If i = j, then b i,j =∞, meaning the communication bandwidth is infinite within the same GPU. Generally, b i,j =b j,i The data transfer latency between two GPUs, i and j, is estimated as the sum of the amount of data transferred and b. i,j The ratio between them.

[0082] Step S330: Network setup and interconnection between GPU cards across servers.

[0083] Cross-server GPU interconnection utilizes Restricted Memory Access (RDMA) technology to enable data transfer between GPUs on different servers, mitigating the latency associated with GPUs exchanging data via host memory. Two commonly used RDMA implementations are InfiniBand (IB) and RoCE. The communication bandwidth between cross-server GPUs depends on the network interface card (NIC) bandwidth, which is limited by the PCIe bandwidth of the GPU server (typically a 400Gbps NIC). Therefore, the actual interconnect bandwidth between GPUs is only 50GB / s. Consequently, cross-server GPU interconnection bandwidth is significantly lower than intra-server GPU interconnection bandwidth, resulting in longer data transfer latency.

[0084] During model training, the parameter plane networking scheme also affects the actual bandwidth of cross-server GPU interconnection. Existing GPU cluster parameter plane networking schemes often adopt a high-performance distributed Fat Tree networking architecture, where the bandwidth of upper and lower layers does not converge, forming a non-blocking network. GPU parameter plane networks can adopt a two-layer Leaf-Spine networking architecture, while large-scale GPU cluster parameter plane networks often adopt a three-layer Leaf-Spine-Core architecture.

[0085] Assuming the GPU parameter plane network uses switches of the same specifications, each switch has w ports. In the Leaf-Spine network architecture, each Leaf layer switch has a maximum of w ports. The Spine layer has a maximum of one uplink and one downlink port. Each Spine layer switch can connect to a maximum of w Leaf switches. This network structure can connect to a maximum of [number] Leaf switches. One GPU chip.

[0086] In a Leaf-Spine-Core three-layer network, the Core layer switch can connect to... Leaf-Spine networks, with a maximum of One Spine switch One Leaf switch, There are Core switches and Leaf switches. With one uplink and one downlink port, the three-layer architecture can connect up to [number] connections. Each GPU chip. The Fat Tree network structure also has drawbacks; the number of switch ports limits the number of GPUs that can be interconnected.

[0087] Given that cross-server GPU interconnect bandwidth is significantly less than intra-server GPU interconnect bandwidth, connecting GPUs within the same node to the same Leaf switch is meaningless. Distributed parallel training of large models is influenced by the parameter surface networking design; therefore, communication involving large amounts of data should be performed between GPU nodes within the server, which offers higher communication bandwidth. Similarly, a bandwidth matrix (Equation 1-5) can be used to describe the communication performance between GPUs.

[0088] Step S340: Calculation of key indicators for the communication algorithm.

[0089] In this embodiment, GPU interconnection can employ All-Reduce communication. The Ring algorithm and the double-binary tree algorithm can both be used to implement All-Reduce communication. The Ring All-Reduce algorithm is widely used, breaking down an All-Reduce communication into two steps: a Scatter-Reduce step and an All-Gather step. The number of GPUs is n, the amount of communication data processed is d, the inter-GPU communication bandwidth is bd, and the inter-GPU device latency is l. gpu The computation time per unit data volume is c. gpu In the Ring All-Reduce algorithm, the amount of communication data between two adjacent GPUs along the ring edge is equal to the total amount of communication data. The number of communication sessions between GPUs is 2×(n-1), and the proportion of total communication for each loop edge is...

[0090] In the embodiments provided in this disclosure, the key metrics for distributed training of the large model to be trained may include: key metrics for large model training, key metrics for distributed training, and distributed training communication metrics for GPU clusters.

[0091] Key metrics for large model training include: GPU memory usage, computational cost, and computational intensity of the large model. GPU memory usage is determined based on the memory usage of model parameters, gradients, optimizer parameters, and activation values. Computational cost is determined based on the forward pass computation of the large model to be trained and the number of model layers. Computational intensity of the large model is determined based on computational cost, the number of available GPUs, and the available computing power of the GPUs. Key metrics for distributed training include GPU metrics under data parallelism, pipelined parallelism, and tensor parallelism. Communication metrics for GPU cluster distributed training include communication metrics under data parallelism, pipelined parallelism, and tensor parallelism.

[0092] Specifically, such as Figure 4 As shown, Figure 4This is a schematic diagram illustrating the calculation process of key performance indicators (KPIs) for large model training provided in this embodiment. The calculation process mainly includes the following steps:

[0093] Step S410: Calculate key performance indicators for model training.

[0094] Key metrics for training large models include GPU memory usage and computational cost. GPU memory usage determines the lower limit of GPU computing resources required to run the model, while computational cost affects training time.

[0095] The memory overhead during large model training includes the memory usage of model parameters (mem). p Gradient memory usage (mem) g Optimizer parameters: memory usage (mem) o Activation value, video memory usage (mem) a Etc., the total GPU memory usage (mem) for training large models:

[0096] mem = mem p +mem g +mem o +mem a (Equation 2-1)

[0097] If the memory usage (mem) is insufficient for a single GPU, then a single GPU cannot meet the model training requirements. Memory usage is related to model parameters, gradients, optimizers, the number of activations, and data precision. Large model training typically employs mixed-precision training, where model parameters and gradients are represented using 16-bit floating-point numbers, while optimizer parameters are represented using 32-bit floating-point numbers. This example assumes mixed-precision training. The choice of optimizer also affects memory usage; this example assumes the Adam optimizer is used. In actual model training, methods such as activation recalculation and offloading can reduce memory usage, but these are trade-offs between time and space complexity. For example, training a large model with p parameters using all parameters, the memory usage for model parameters, gradients, and optimizers is approximately:

[0098] mem p +mem g +mem o =16p (Equation 2-2)

[0099] Among them, mem p =2p,mem g =2p,mem o=12p. Activation values ​​are intermediate results that need to be saved during the model's backpropagation process, and are related to the training batch size and model structure. Taking a large model with p parameters based on a transformer structure as an example, the model has l layers, h hidden layers, a sequence length of sq, and h multi-head numbers. a The training batch size is bs, and the number of model parameters is:

[0100] p = v × h + (12h) 2 +13h)×l (Equation 2-3) Its activation value accounts for approximately:

[0101] mem a = (34bs×sq×h×+5bs×sq) 2 ×h a )×l (Equation 2-4)

[0102] When a single GPU cannot meet the computational power requirements for training large model parameters, distributed training methods such as data parallelism, pipeline parallelism, and tensor parallelism are used to reduce the memory requirements of a single GPU. These methods will be specifically described in the following embodiments.

[0103] The computational intensity I can be estimated by the computational cost and memory access cost during model training. The computational cost m during model training... ops This represents the number of floating-point operations performed on a batch of data during large model training, measured in FLOPs. The computational cost of training a large model is the sum of the computational costs of forward and backward propagation; typically, the computational cost of backward propagation is approximately twice that of forward propagation.

[0104] Step S420: Calculation of key performance indicators for distributed training of the GPU cluster.

[0105] Large-scale model distributed training parallel techniques include data parallelism, pipelined parallelism, and tensor parallelism, which address the issue of insufficient GPU memory during large-scale model training, while simultaneously improving training speed and performance. Different distributed parallel methods have varying communication volumes and frequencies; the parametric surface network structure is designed based on these factors. Data parallelism addresses computational limitations by increasing parallel computing power, thereby improving training and induction speed.

[0106] Data parallelism splits batch data into different GPUs for computation. The number of model parameters and activation values ​​in GPU memory will not decrease. Training speed is improved by increasing computing power. This method itself does not reduce the GPU memory usage of model training. However, ZeRO technology reduces the GPU memory usage of a single chip by splitting the optimizer, gradient, and model parameters into multiple GPU chips.

[0107] In this embodiment, the data-parallel ZeRO method is not used as a method to reduce GPU memory concerns; instead, pipeline parallelism and tensor parallelism are primarily employed to address the issue of limited GPU chip memory size. Based on the characteristics of data parallelism, tensor parallelism, and model parallelism, the GPU memory usage and GPU computational load for different parallelism methods are calculated.

[0108] The number of GPU clusters is n, that is, n = n d ×n t ×n p We combine multiple parallel methods, including data parallelism, model parallelism, and tensor parallelism, to partition GPU clusters for training large models.

[0109] Step S430: Calculation of distributed training communication metrics for the GPU cluster.

[0110] Combining the characteristics of communication algorithms and distributed training methods, the communication volume and latency of various distributed parallel methods are estimated. Data parallelism divides the batch training data into multiple smaller batches and distributes them to different GPUs for forward propagation computation. During backpropagation, an All-Reduce communication is used to synchronize gradients and update weights. In this embodiment, the All-Reduce communication operation based on a ring algorithm can be used as an example.

[0111] Pipeline parallelism can solve the memory limitation problem by splitting the model along the layer structure across different GPUs, reducing memory requirements while increasing computing power. The number of pipeline parallels is n. p GPUs communicate directly using a P2P method, performing n operations during both forward and backward propagation. p -1 P2P communication, transmitting activation values ​​or partial derivative values ​​of different layers of the model; the amount of communication data for P2P forward and backward propagation is d. p The total communication data volume for one forward and backward propagation is 2d. p ×(n p -1), communication delay is approximately o p :

[0112]

[0113] Therefore, the communication latency of pipelined parallelism is directly proportional to the number of parallel operations and inversely proportional to the communication bandwidth.

[0114] Tensor parallelism can also solve the problem of GPU memory limitations by modifying the computation methods within model layers, thereby reducing the model's GPU memory usage. Taking a large model based on Transformer layers as an example, a Transformer layer performs two All-Reduce communications during both forward and backward propagation, passing activation values ​​or partial derivatives. Assume the number of tensor parallel operations is n. t Communication data volume d tAll-Reduce communication is performed, with a transmission delay of 0 for 4 All-Reduce communication sessions. t :

[0115]

[0116] The communication volume between the two GPUs is 8d. t .

[0117] The embodiments provided in this disclosure, when automating the allocation of computing power in GPU clusters for large model training, improve the training speed of large models under limited GPU cluster computing power resources by establishing a GPU cluster computing power allocation optimization model, and estimate the training time of large models based on different distributed parallel training methods.

[0118] The goal of the GPU cluster computing power allocation optimization model is to accelerate the training of large models based on the same amount of training data. The structural parameters of a large transformer-based model include the number of model layers (l), hidden layers (h), sequence length (sq), and number of multi-head arrays (h). a The batch size is bs. Large model training is performed using a distributed approach combining data parallelism, pipelined parallelism, and tensor parallelism on the intelligent computing cluster. The decision variables for the GPU cluster computing power allocation optimization model include the number of parallel operations (data parallelism, pipelined parallelism, and tensor parallelism), denoted as n respectively. d n p n t Batch size bs, n micro This represents the pipeline microbatch size. Assuming data parallelism, pipeline parallelism, tensor parallelism, and inter-GPU communication bandwidth, the values ​​are bd respectively. d ,bd p ,bd t .

[0119] The GPU cluster computing power allocation optimization model aims to minimize the training time t. train The objective function is defined as:

[0120] obj = mint train (Equation 2-7)

[0121] The training time for a large model is approximately the computation time t. comp and communication time t comm The sum of t. That is: t train =t comp +t comm .

[0122] The total amount of training data for a large model training session is N, and the training period is e. Estimate the computation time t for model training under data parallelization. comp :

[0123]

[0124] in, Indicates the time required for parallel data computation. Also indicates the size of the batch of data processed in parallel computation. Further calculations of pipeline parallel computing time

[0125]

[0126] In the simple assembly line, bubbles exist in parallel, with a bubble ratio of [missing value]. bs p This indicates the amount of data in a parallel micro-batch pipeline. n micro This represents the microbatch size in the pipeline. Further calculations are needed to determine the tensor parallel computation time.

[0127]

[0128] Obtain the communication time t, which includes data parallelism, pipelined parallelism, and tensor parallelism. comm The total time for parallel communication to acquire data is... Total time for pipelined parallel communication Total time for tensor parallel communication

[0129] By combining communication technology, networking technology, and GPU interconnect technology, a computing bandwidth matrix mat is constructed between GPUs. bd , representing the computational bandwidth between any two GPUs i and j:

[0130]

[0131] Among them, b i,j =inf, 1≤i,j≤n. Taking Leaf-Spine networking as an example, the computational bandwidth between two GPUs within a server node is higher than the computational bandwidth between two GPUs connected via a Leaf switch, and higher than the computational bandwidth between two GPUs interconnected via both Spine and Leaf switches. Generally, the bandwidth for tensor parallelism, pipelined parallelism, and data parallelism is determined using the following method:

[0132] 1) Tensor parallel bandwidth calculation:

[0133] Tensor Parallelism n t Less than or equal to the number of GPUs in a single server Right now bd t =maxb i，j GPU i and GPU j are on the same server, bd tMaximum interconnect bandwidth between GPUs within the server:

[0134] bd t =maxb i，j (Equation 2-11)

[0135] Among them, b i，j This represents the maximum interconnect bandwidth between GPU cards within the server.

[0136] 2) Calculation of pipeline parallel bandwidth:

[0137] If the number of parallel pipelines is n p With tensor parallel number n t The product is less than or equal to the maximum number of GPUs in the server. Right now bd t =bd p =maxb i，j GPU i and GPU j are interconnected within the server.

[0138] If the number of parallel pipelines is n p With tensor parallel number n t The product is greater than the maximum number of GPUs in the server. The maximum number of GPU servers connected to the Leaf layer switch is less than or equal to Maximum number of GPUs in the server The product, i.e. So bd p =maxb i，j In this context, GPU i and GPU j are interconnected by a layer switch.

[0139]

[0140] 3) Data parallel bandwidth calculation.

[0141] If the number of parallel data n d Number of parallel pipelines n p With tensor parallel number n t The product is less than or equal to the maximum number of GPUs in the server. Right now bd t =bd p =bd d =maxb i，j GPU i and GPU j are interconnected within the server.

[0142] If the number of parallel data n d Number of parallel pipelines n p With tensor parallel number n t The product is greater than the maximum number of GPUs in the server. The maximum number of GPU servers connected to the Leaf layer switch is less than or equal to Maximum number of GPUs in the server The product, i.e. So bd d =maxb i，j GPU i and GPU j are interconnected by a layer switch.

[0143] If the number of parallel data n d Number of parallel pipelines n p With tensor parallel number n t The product is greater than the maximum number of GPU servers connected to the Leaf layer switch. Maximum number of GPUs in the server The product is less than or equal to the maximum number of downlink switches that a Spine layer switch can connect to. Maximum number of GPU servers connected to the Leaf layer switch Maximum number of GPUs in the server The product, i.e. So bd d =maxb i，j GPU i and GPU j are interconnected through two layers of switches.

[0144]

[0145] Based on equations (2-20), (2-24), and (2-28), an optimization model for intelligent computing power allocation is established, with n as the decision variable. d n p n t ,bs,n micro Therefore, the objective function is:

[0146]

[0147] In this embodiment, the constraints of the cluster computing power resource allocation model include: GPU cluster resource constraints, individual GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints. Therefore, GPU cluster computing power allocation is affected by various constraints, the main ones being as follows:

[0148] 1) GPU cluster resource constraints: Ensure that the GPU resources used for training all models do not exceed the total number of GPUs, n.

[0149] n≥n d ×n p ×n t (Equation 2-15)

[0150] 2) Memory usage per GPU (mem) gpu Not exceeding the GPU memory size m, i.e., mem gpu ≤m:

[0151]

[0152] 3) Tensor parallelism constraints: Tensor parallelism efficiency is generally affected by communication volume and bandwidth. Since a server-side GPU interconnect method is used, the tensor parallelism number n... t It will not exceed the maximum number of interconnected GPUs within the server.

[0153]

[0154] 4) Pipeline parallelism constraint: If the number of pipeline parallelism n p With tensor parallel number n t The product will not exceed the maximum number of GPU servers connected to the Leaf layer switch. To ensure that the interconnect between parallel GPUs in the pipeline is directly connected to at most one layer of switch, that is:

[0155]

[0156] 5) Production line micro-batch size n micro Data parallelism n d Less than the batch size bs, that is:

[0157] n micro ≤bs (Equation 2-19)

[0158] n d ≤bs (Equation 2-20)

[0159] 6) Total training time shall not exceed t max :

[0160] t train ≤t max (Equation 2-21)

[0161] A nonlinear optimization model for GPU cluster computing power allocation can be constructed based on the above embodiments, and the decision variable n with the shortest training time can be calculated using the grid index simulation method. d n p n t ,bs,n micro Value, and the optimal target value t train If the allocation model has no solution, it means that the existing GPU cluster resources are insufficient to meet the requirements within a time limit of t. maxTraining a large model with limited training data and a finite timeframe is completed. Heuristic methods are employed to solve the decision variables and compute the objective value of the nonlinear optimization model for GPU cluster computing power allocation in ultra-large-scale cluster environments. Based on n... d n p n t The cluster computing power is allocated according to the method of first interconnecting GPUs within the server and then interconnecting across servers to achieve tensor parallelism, pipeline parallelism and data parallelism in sequence.

[0162] The embodiments provided in this disclosure, when training large models in intelligent computing products, can achieve automated allocation of GPU cluster computing power, optimize distributed training schemes, improve the training efficiency of large models, and optimize the utilization rate of GPU cluster computing power. Specifically, the following technical effects can be achieved:

[0163] (1) The embodiments of this disclosure combine the hardware performance of GPU intelligent computing clusters and the characteristics of large model distributed training. They can quantitatively analyze and calculate a variety of indicators to realize an automated GPU cluster computing power allocation tool for large model training. It does not require the use of small batch data for pre-training. It can quantitatively optimize the computing power allocation scheme for different GPU clusters and different large model training, and optimize training time and training efficiency.

[0164] (2) The embodiments provided in this disclosure analyze the main factors affecting GPU computing performance from the perspective of single GPU card performance, including chip peak computing power, memory capacity, and memory bandwidth. They also provide an in-depth analysis of the GPU card interconnection topology and performance within servers, as well as the parameter plane networking structure and cross-server GPU card interconnection performance, summarizing key core performance indicators for GPU chips, server-wide GPU interconnection, and cross-server GPU interconnection. Based on these indicators, a multi-angle, fine-grained performance calculation method and computational model are used to evaluate the key performance parameters of GPU clusters. Existing large-scale distributed training methods rarely consider networking and server-wide GPU card interconnection performance, or only use coarse-grained qualitative methods to evaluate the actual performance of GPU card interconnection during large-scale model training.

[0165] (3) The embodiments provided in this disclosure deeply analyze the key indicators of large model training and distributed training. By establishing a large model training memory usage evaluation model and a large model training computational load evaluation model, a large model training computational intensity model is established. The performance bottleneck of the GPU cluster can be analyzed through the results of the computational intensity model. For distributed training modes such as data parallelism, pipeline parallelism, and model parallelism, GPU average memory usage and GPU average computational load estimation models are established. These models can quantitatively estimate the usage and utilization of GPU computing power and memory resources. They can be used as the main computational model in the computing power allocation algorithm, and the estimated values ​​can also be used to supervise the actual operation of the GPU cluster during model training.

[0166] (4) The embodiments provided in this disclosure establish a calculation model for communication latency under different distributed training modes by analyzing the communication mode, communication volume and communication number under different distributed training modes. This model can quantitatively analyze the data communication latency of GPU card interconnection mode, and thus more precisely determine the training time required for large models.

[0167] (5) The embodiments provided in this disclosure propose an automated method for GPU cluster computing power allocation for large model training. This method can optimize the distributed training scheme for large models and improve training speed under limited GPU cluster resources. The goal of the automated method is to minimize training time, including computation time and communication time, and includes constraints such as GPU cluster resources, GPU memory, number of distributed parallel operations, and maximum total training time. The model is solved using grid search or heuristic methods to obtain the key parameters of the distributed training scheme. The cluster computing power is allocated according to the method of first inter-server GPU interconnection and then cross-server interconnection, thereby realizing tensor parallelism, pipeline parallelism and data parallelism in sequence, and finally realizing the automated allocation of GPU cluster resources.

[0168] By dividing each functional module according to its corresponding function, this disclosure provides a computing power allocation device for large model training. This computing power allocation device for large model training can be a server, a terminal, or a chip applied to a server. Figure 5 This is a schematic block diagram of the functional modules of a computing power allocation device for large model training provided as an exemplary embodiment of this disclosure. Figure 5 As shown, the computing power allocation device for large model training includes:

[0169] The data acquisition module 61 is used to acquire the large model to be trained and the training dataset, and to acquire the current GPU resources;

[0170] The model building module 62 is used to build a cluster computing power resource allocation model; wherein, the cluster computing power resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints.

[0171] The computing resource allocation module 63 is used to obtain the input of the cluster computing resource allocation model based on the large model to be trained, the training dataset, and the current GPU resources, and output decision variables and optimal target values ​​through the cluster computing resource allocation model; wherein, the decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch processing quantity, and pipeline micro-batch size, and the optimal target value includes minimizing the training time of the large model to be trained.

[0172] In another embodiment provided in this disclosure, the cluster computing power resource allocation model includes: a GPU intelligent computing cluster key performance indicator module, a large model training key performance indicator module, and a GPU cluster computing power allocation module for large model training.

[0173] The GPU intelligent computing cluster key performance index module is used to calculate key performance indexes in the GPU intelligent computing cluster. The large model training key performance index module is used to calculate key indexes of the large model to be trained in distributed training. The GPU cluster computing power allocation module for large model training is used to allocate computing power resources to the large model to be trained by optimizing the objective function under the constraints.

[0174] In another embodiment provided in this disclosure, the key performance indicators in the GPU intelligent computing cluster include: GPU graphics card parameter performance indicators, GPU card interconnection topology indicators within the server, and GPU card interconnection indicators across servers.

[0175] In another embodiment provided in this disclosure, the GPU graphics card performance parameters include: data read time, computation time, computation intensity, and available GPU computing power; wherein, the data read time is determined based on the ratio between the amount of data read and written by the GPU and the chip bandwidth, the computation time is determined based on the ratio between the number of floating-point operations of the GPU and the peak computing power of the chip, the computation intensity is determined based on the ratio between the total number of floating-point operations and the amount of memory accesses, and the available GPU computing power is determined based on the peak computing power of the chip, the computation intensity, and the chip bandwidth.

[0176] In another embodiment provided in this disclosure, the key metrics for distributed training of the large model to be trained include: key metrics for large model training, key metrics for distributed training, and distributed training communication metrics for the GPU cluster.

[0177] In another embodiment provided in this disclosure, the key indicators for large model training include: GPU memory usage, computational cost, and large model computational intensity; wherein, the GPU memory usage is determined based on the GPU memory usage of model parameters, gradients, optimizer parameters, and activation values; the computational cost is determined based on the forward pass computation of the large model to be trained and the number of model layers; the large model computational intensity is determined based on the computational cost, the number of available GPUs, and the available computing power of the GPUs; the key indicators for distributed training include GPU key indicators under data parallelism, pipelined parallelism, and tensor parallelism respectively; the communication indicators for GPU cluster distributed training include communication indicators under data parallelism, pipelined parallelism, and tensor parallelism respectively.

[0178] This disclosure also provides an electronic device, including: at least one processor; a memory for storing processor-executable instructions; wherein the at least one processor is configured to execute the instructions to implement the methods disclosed in this disclosure.

[0179] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0180] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the method described above.

[0181] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 7 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 7 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0182] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0183] like Figure 7 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0184] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0185] The computing unit 1901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0186] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0187] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0189] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0190] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0192] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0193] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0194] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0195] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for allocating computing power for training large models, characterized in that, The method includes: Obtain the large model to be trained and the training dataset, and obtain the current GPU resources; A cluster computing power resource allocation model is constructed; wherein, the cluster computing power resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints; The cluster computing power resource allocation model is obtained based on the large model to be trained, the training dataset, and the current GPU resources. The model outputs decision variables and optimal target values. The decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch size, and pipeline micro-batch size. The optimal target value includes minimizing the training time of the large model to be trained.

2. The method according to claim 1, characterized in that, The cluster computing power resource allocation model includes: a GPU intelligent computing cluster key performance indicator module, a large model training key performance indicator module, and a GPU cluster computing power allocation module for large model training. The GPU intelligent computing cluster key performance index module is used to calculate key performance indexes in the GPU intelligent computing cluster. The large model training key performance index module is used to calculate key indexes of the large model to be trained in distributed training. The GPU cluster computing power allocation module for large model training is used to allocate computing power resources to the large model to be trained by optimizing the objective function under the constraints.

3. The method according to claim 2, characterized in that, Key performance indicators in the GPU intelligent computing cluster include: GPU graphics card parameter performance indicators, GPU card interconnection topology indicators within the server, and GPU card interconnection indicators across servers.

4. The method according to claim 3, characterized in that, The GPU graphics card performance parameters include: data read time, computation time, computation intensity, and available GPU computing power; wherein, the data read time is determined based on the ratio between the amount of data read and written by the GPU and the chip bandwidth, the computation time is determined based on the ratio between the number of floating-point operations of the GPU and the peak computing power of the chip, the computation intensity is determined based on the ratio between the total number of floating-point operations and the amount of memory accesses, and the available GPU computing power is determined based on the peak computing power of the chip, the computation intensity, and the chip bandwidth.

5. The method according to claim 2, characterized in that, The key metrics for the distributed training of the large model to be trained include: key metrics for large model training, key metrics for distributed training, and metrics for distributed training communication of the GPU cluster.

6. The method according to claim 5, characterized in that, The key metrics for large model training include: GPU memory usage, computational cost, and large model computational intensity; wherein, the GPU memory usage is determined based on the GPU memory usage of model parameters, gradients, optimizer parameters, and activation values; the computational cost is determined based on the forward pass computation of the large model to be trained and the number of model layers; the large model computational intensity is determined based on the computational cost, the number of available GPUs, and the available computing power of the GPUs; the key metrics for distributed training include GPU key metrics under data parallelism, pipelined parallelism, and tensor parallelism respectively; the communication metrics for GPU cluster distributed training include communication metrics under data parallelism, pipelined parallelism, and tensor parallelism respectively.

7. A computing power allocation device for large model training, characterized in that, The device includes: The data acquisition module is used to acquire the large model to be trained and the training dataset, as well as the current GPU resources; The model building module is used to build a cluster computing resource allocation model; wherein, the cluster computing resource allocation model includes constraints and an objective function; the constraints include: GPU cluster resource constraints, GPU memory usage constraints, tensor parallelism constraints, pipeline parallelism constraints, pipeline micro-batch constraints, and total training time constraints. The computing resource allocation module is used to obtain the input of the cluster computing resource allocation model based on the large model to be trained, the training dataset, and the current GPU resources, and output decision variables and optimal target values ​​through the cluster computing resource allocation model; wherein, the decision variables include tensor parallelism, pipeline parallelism, data parallelism, batch processing quantity, and pipeline micro-batch size, and the optimal target value includes minimizing the training time of the large model to be trained.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images