Training strategy determination method for computing power cluster using chips of various models in mixed mode and related device

Abstract

The invention provides a training strategy determination method for a computing power cluster using chips of various models in a mixed mode and a related device, and relates to the technical field of graphic processing chips, computing power clusters, large models, model training and the like. The method comprises the following steps: determining each target model chip forming a computing power cluster and a corresponding quantity proportion; for each target model chip, determining performance evaluation of the target model chip in a preset performance type and weight distribution of each performance type under each alternative training strategy; for each alternative training strategy, performing weighting processing on the performance evaluation under each corresponding preset performance type in combination with each corresponding number proportion by using the weight distribution, matched with each target model chip, of the alternative training strategy, so as to obtain corresponding comprehensive training evaluation; and determining a target training strategy based on the evaluation parameter of each comprehensive training evaluation. By applying the scheme, the training strategy which is suitable for being used by the computing power cluster which mixedly uses multiple types of chips during training of a large model can be quickly determined at low cost.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to technical fields such as graphics processing chips, computing power clusters, large models, model training, and more particularly to a method, device, electronic device, computer-readable storage medium, and computer program product for determining a training strategy for a computing power cluster that uses a mixture of chips of various models. Background Art

[0002] With the vigorous development of the AI ​​(Artificial Intelligence) industry, the underlying AI computing technology is also constantly updated and iterated, and the industry has been actively looking for new computing solutions to meet challenges such as cost and supply.

[0003] In this wave of large models, the computing power required for a single training task has also expanded from tens of cards to tens of thousands or even hundreds of thousands of cards. The supply challenges of using a single chip to build a super-large cluster are much greater than before. And considering that companies have purchased a large number of various models of AI chips or GPUs (Graphics Processing Units) in the past, they need to give full play to the computing power of these old chips, such as using multiple models of chips to build a mixed computing power cluster.

[0004] At the same time, in order to overcome the scalability problems brought about by large generative language models, current AI frameworks have introduced optimization strategies such as tensor parallelism, recomputation, memory offload (memory reasoning technology, which can also be understood as hybrid deployment or hybrid reasoning technology, such as unloading objects to external resources or distributing them across multiple devices to reduce the memory load of the main device), and sequence parallelism to support larger and larger models, longer and longer inputs, and special model architectures.

[0005] At the same time, due to the differences in model size and data scale, the scale of the computing power cluster built by multiple graphics processing units (GPUs) at the bottom layer is also different. Under different model sizes, data scales, and computing power clusters of different sizes, there will be obvious differences in the performance of different training strategy combinations.

[0006] How to quickly and cost-effectively select a suitable training strategy from a variety of alternative training strategies to train the required generative large language model according to the training task by mixing the computing power cluster of multiple chip models is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0007] The embodiments of the present disclosure propose a method, device, electronic device, computer-readable storage medium, and computer program product for determining a training strategy for a computing power cluster that uses a mixture of chips of multiple models.

[0008] In the first aspect, an embodiment of the present disclosure proposes a method for determining a training strategy for a computing power cluster that uses a mixture of multiple chip models, including: determining the target chip models that constitute the computing power cluster and the number ratio of each target chip model; for each target chip model, determining the performance evaluation of the target chip model under a preset performance type and the weight distribution of each preset performance type under each alternative training strategy; for each alternative training strategy, using the alternative training strategy to match the weight distribution of each target chip model, respectively, the performance evaluation of each corresponding target chip model under the preset performance type is weighted in combination with the number ratio of each corresponding target chip model to obtain a corresponding comprehensive training evaluation; based on the evaluation parameters of each comprehensive training evaluation, determining the target training strategy.

[0009] In the second aspect, the embodiments of the present disclosure propose a training strategy determination device for a computing power cluster that uses a mixture of multiple chip models, including: a model and quantity ratio determination unit, which determines the target model chips and the quantity ratio of each target model chip that constitute the computing power cluster; a performance evaluation and weight distribution determination unit, which is configured to determine, for each target model chip, the performance evaluation of the target model chip under a preset performance type and the weight distribution of each preset performance type under each alternative training strategy; a comprehensive training evaluation determination unit, which is configured to use the weight distribution of the alternative training strategy matched to each target model chip to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance type combined with the quantity ratio of each corresponding target model chip for each alternative training strategy to obtain a corresponding comprehensive training evaluation; a target training strategy determination unit, which is configured to determine the target training strategy based on the evaluation parameters of each comprehensive training evaluation.

[0010] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it is possible to implement the method for determining a training strategy for a computing power cluster that uses a mixture of multiple chip models as described in the first aspect.

[0011] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method for determining a training strategy for a computing power cluster that uses a mixture of multiple chip models as described in the first aspect when executed.

[0012] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the method for determining a training strategy for a computing power cluster that uses a mixture of chips of multiple models as described in the first aspect.

[0013] The present disclosure provides a training strategy determination scheme for a computing power cluster that uses a mixture of multiple chip models. After receiving a training task that instructs to train a generative large language model, the computing power cluster composed of multiple chip models used for training, the target chip models specifically used by the computing power cluster, and the quantity ratio of each target chip model are first determined. Then, for each target chip model, its performance evaluation under a preset performance category and the weight distribution of each preset performance category under different alternative training strategies are determined. Then, for each alternative training strategy, the alternative training strategy is used to match the weight distribution of each target chip model, and the performance evaluation of each corresponding target chip model under the preset performance category is combined with the quantity ratio of each corresponding target chip model to perform weighted processing, so as to obtain a comprehensive training evaluation corresponding to the alternative training strategy, and finally a suitable target training strategy is selected according to the evaluation parameters of each comprehensive training evaluation.

[0014] That is, by applying the technical solution provided by the present invention, it is not necessary to determine the appropriate target training strategy by trial training each alternative training strategy one by one, thereby reducing the resource overhead and time overhead required for trial training. The target training strategy suitable for training a generative large language model in a computing power cluster composed of multiple different types of chips can be quickly determined at a low cost, so that the training task can be completed with less time and resource overhead, thereby indirectly improving the model training efficiency.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is an exemplary system architecture in which the present disclosure may be applied; Figure 2 A flowchart of a method for determining a training strategy for a computing cluster that uses a mixture of chips of various models provided in an embodiment of the present disclosure; Figure 3 A flowchart of a method for determining a performance evaluation of a target model chip under a preset performance category provided by an embodiment of the present disclosure; Figure 4 A flowchart of a method for training a performance prediction model provided in an embodiment of the present disclosure; Figure 5 A flowchart of a method for determining a comprehensive training evaluation provided in an embodiment of the present disclosure; Figure 6 A flowchart of a method for scoring and determining a comprehensive training evaluation based on an embodiment of the present disclosure; Figure 7 A flowchart of a method for controlling a computing power cluster to perform training tasks according to a target training strategy provided by an embodiment of the present disclosure; Figure 8-1 A schematic diagram of the structure of a model training system for hybrid multi-model chips in combination with a specific application scenario provided by an embodiment of the present disclosure; Figure 8-2 A schematic diagram of the structural flow of an optimal training strategy determination system in combination with a specific application scenario provided by an embodiment of the present disclosure; Figure 8-3 A schematic diagram of the structural flow of a system for comparing different training strategies to determine an optimal training strategy in combination with a specific application scenario provided by an embodiment of the present disclosure; Fig. 9 A structural block diagram of a training strategy determination device for a computing power cluster that uses a mixture of chips of various models provided in an embodiment of the present disclosure; Fig.10 A schematic diagram of the structure of an electronic device suitable for executing a method for determining a training strategy for a computing power cluster that uses a mixture of chips of various models, provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0018] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] Figure 1An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for determining a training strategy for a computing cluster that uses a mixture of chips of various models of the present disclosure can be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include a server 110, a network 120, and a computing cluster 130. The computing cluster 130 includes multiple graphics processors, such as 131, 132, 133, 134, and 135 (not just 5, Figure 1 Only a few numbers are used as examples), and the graphics cards 131, 132, 133, 134 and 135 include at least two different graphics processor models, that is, the computing power cluster 130 is a computing power cluster composed of a mixture of multiple different models of graphics processors; the network 120 is used as a medium for providing a communication link between the server 110 and the computing power cluster 130, and the network 120 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0021] The user can use the terminal device to interact with the server 110 through the network 120 to receive or send messages, etc. The server 110, the computing cluster 130 and the terminal device can be installed with various applications for realizing information communication between the two, such as training task delivery applications, training strategy optimization applications, instant messaging applications, etc.

[0022] The terminal device and the server 110 can be hardware or software. When the terminal device is hardware, it can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.; when the terminal device is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, without specific limitation here. When the server 110 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server 110 is software, it can be implemented as multiple software or software modules, or as a single software or software module, without specific limitation here. The graphics processor constituting the computing power cluster 130 is usually manifested as hardware, and of course it can also be manifested as software or software running products in special scenarios (such as simulation scenarios), which is not specifically limited here.

[0023] The server 110 can provide various services through various built-in applications. Taking the training strategy optimization application that can provide training strategy optimization service as an example, the server 110 can achieve the following effects when running the training strategy optimization application: first, determine the target model chips and the quantity ratio of each target model chip that constitute the computing power cluster 130; then, for each target model chip, determine the performance evaluation of the target model chip under the preset performance type and the weight distribution of each preset performance type under each alternative training strategy; next, for each alternative training strategy, use the weight distribution of the alternative training strategy that matches each target model chip to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance type combined with the quantity ratio of each corresponding target model chip to obtain the corresponding comprehensive training evaluation; finally, based on the evaluation parameters of each comprehensive training evaluation, determine the target training strategy.

[0024] Furthermore, after determining the target training strategy, the server 110 may also send the information to the computing power cluster 130 through the network 120, so that the computing power cluster 130 trains the large model by using the target training strategy.

[0025] Since selecting the best one from a variety of alternative training strategies requires more computing resources and stronger computing power, the training strategy determination method for a computing power cluster that uses a mixture of multiple chip models provided in the subsequent embodiments of the present disclosure is generally executed by a server 110 with stronger computing power and more computing resources. Correspondingly, the training strategy determination device for a computing power cluster that uses a mixture of multiple chip models is generally also set in the server 110.

[0026] It should be understood that Figure 1 The number of servers, networks, computing clusters, and graphics cards in the example is only illustrative. Any number of servers, networks, computing clusters, and graphics cards may be provided as required.

[0027] Please refer to Figure 2 , Figure 2 A flowchart of a method for determining a training strategy for a computing cluster that uses a mixture of chips of various models is provided in an embodiment of the present disclosure, wherein process 200 includes the following steps: Step 201: Determine the target model chips and the quantity ratio of the target model chips constituting the computing power cluster; This step is intended to determine the execution subject of the method by the training strategy for the computing power cluster that uses a mixture of multiple chip models (for example Figure 1The server 110 shown in the figure determines the various types of chips (i.e., the target type chips) used by the computing power cluster composed of multiple different types of chips, as well as the proportion of each type of chip in the computing power cluster (i.e., the proportion of the actual number of chips of this model to the total number of chips).

[0028] Among them, it should be noted that the computing power cluster is determined based on the pre-received large model training task (for example, the training task clearly records or carries relevant information indicating the need to use the computing power cluster, such as the computing power cluster number, name, etc.), that is, the computing power cluster is needed to specifically perform the training task of the generative large language model, and in the present disclosure, the computing power cluster is composed of a variety of different models of graphics processing chips, and the number of graphics processing chips of each model can be the same or different, and is not specifically limited. Therefore, this step actually determines the multiple models of graphics processing chips used by the computing power cluster used to perform the large model training task in the future - multiple target models, and then determines the graphics processing chips with each target model as the target model chip described in this step, and then determines the number ratio of each target model chip.

[0029] An implementation method including but not limited to may be: First, based on the model information of the various image processing chips used by the computing power cluster, the target model chips are obtained, and then the proportion of each target model chip is calculated based on the actual number of each target model chip in the computing power cluster and the total number of chips.

[0030] Step 202: for each target model chip, determine the performance evaluation of the target model chip under a preset performance category and the weight distribution of each preset performance category under each candidate training strategy; Based on step 201, this step aims to have the above-mentioned execution subject perform performance evaluation of each target chip model under the preset performance category and determine the weight distribution of each preset performance category under each alternative training strategy. It can be divided into two parts: Part 1: For each target chip model, the performance evaluation of the target chip model under the preset performance category is determined; The performance evaluation can be expressed as a specific performance level, performance score, etc. Among them, the preset performance type refers to the performance type of the evaluation target model chip that is preset or selected, such as computing performance, storage performance, communication performance, etc., that is, the preset performance type may include: at least one of computing performance, storage performance and communication performance. The following will respectively describe these three performances in detail: Computing performance is one of the core advantages of GPU, which is usually measured by the following aspects: 1) Number of cores and parallel computing power: GPUs contain thousands of small computing cores (for example, NVIDIA's A100 GPU has 6912 CUDA cores). These cores are a parallel computing platform and programming model that can perform a large number of simple computing tasks in parallel. Unlike the few powerful cores of the CPU (Central Processing Unit), the GPU is designed to maximize parallel processing performance. This architecture enables the GPU to far exceed the computing power of the CPU when processing large-scale matrix operations and vector operations. When training deep learning models (such as neural networks), millions of matrix multiplication and addition operations are just right for the GPU's parallel processing mode.

[0031] 2) Floating Point Operations Per Second (FLOPs): Floating point operations are the basic type of calculations in deep learning training, and the computing performance of GPUs is usually measured in FLOPs. The computing performance of GPUs is relatively high. For example, the FP32 (32-bit Single Precision Floating Point) computing power of the A100 GPU is 19.5 TFLOPs (19.5 trillion floating point operations per second), and under mixed precision (FP16, 16-bit Single Precision Floating Point, 16-bit half-precision floating point), the performance of A100 can reach 156 TFLOPs. Such powerful computing power makes GPUs more efficient than CPUs when training large-scale neural networks.

[0032] 3) Tensor Core: A hardware unit dedicated to accelerating tensor calculations (matrix multiplication) in deep learning. Tensor Core can support more efficient mixed precision calculations, such as 16-bit floating point numbers (FP16), and significantly improve the training speed of deep learning tasks. Especially in large model training, the application of Tensor Core can reduce training time.

[0033] 4) CUDA architecture and library support: The CUDA platform provides developers with programming interfaces and optimized libraries, enabling GPUs to efficiently execute computing tasks in deep learning frameworks (such as TensorFlow and PyTorch). Using dedicated programming frameworks such as CUDA, developers can fully tap the computing potential of GPUs.

[0034] Storage performance mainly includes memory bandwidth and memory capacity, both of which are crucial for training large-scale models: 1) GPU memory bandwidth determines the speed at which data is transferred between the computing core and the memory. With the increase in large-scale deep learning models and data sets, memory bandwidth has become particularly important. GPUs typically use high-speed video memory (such as GDDR6, HBM2, etc.) to achieve high bandwidth. Taking A100 as an example, its HBM2 video memory provides a bandwidth of 1555GB / s, which means that the GPU can read and write large amounts of data very quickly, which is very important for training deep neural networks, especially when it is necessary to process high-resolution images or large-scale data.

[0035] 2) The size of the video memory determines the size of the data set that the GPU can process. In deep learning tasks, especially when training large-scale neural networks, the video memory needs to be able to accommodate the entire model and training data set. The video memory capacity is usually the determining factor in whether the GPU can process large models on a single card. For example, the A100 is equipped with 40GB or 80GB of video memory, which enables it to process large pre-trained models such as BERT or GPT, while smaller GPU video memory may cause frequent data exchange and reduce performance.

[0036] In addition, memory access patterns also affect storage performance to a certain extent. The memory in the GPU is usually divided into different levels, such as global memory, shared memory, constant memory, etc. The access pattern of each memory is different, which affects the computing efficiency. For example, the shared memory of the GPU usually has lower latency, while the global memory is suitable for storing large amounts of data, but the access speed is slower. Therefore, in large-scale model training, how to optimize the use and access pattern of memory is the key to improving GPU performance.

[0037] The communication performance of the GPU, especially the synchronization mechanism in data transmission and parallel computing, directly affects the efficiency of large-scale model training: 1) Communication between GPUs (NVLink / PCIe, two different communication methods): Multi-card training is particularly common in distributed deep learning. In this scenario, multiple GPUs work together, so the data transmission performance between GPUs is particularly important. In distributed training, the synchronization speed of data between different GPUs (such as gradient synchronization) directly affects the training time and model convergence speed.

[0038] 2) PCIe 4.0 / 5.0: PCIe 4.0 and PCIe 5.0 are important communication interfaces between GPU and CPU, and GPU and storage devices. The bandwidth of PCIe 4.0 is 16 GT / s, and PCIe 5.0 is increased to 32 GT / s. For single-card systems, these interface bandwidths are usually sufficient for most tasks, but when training multiple cards in parallel, bandwidth may become a bottleneck.

[0039] 3) Network bandwidth (for distributed training): In distributed training, GPUs usually communicate with each other through high-speed networks (such as InfiniBand or high-speed Ethernet). Network bandwidth and latency will directly affect the efficiency of training, especially in synchronous training mode where gradients or parameters need to be exchanged frequently. Low-latency and high-bandwidth networks can reduce the communication waiting time between GPUs and improve the overall training speed.

[0040] In addition to the three performance factors mentioned above that affect the GPU's ability to train large models, the following performance factors also affect the GPU's ability to train large models to a certain extent: 1) Co-optimization of hardware and software: The performance of a GPU is not only determined by the hardware itself, but software optimization is also crucial. Deep learning frameworks (such as TensorFlow and PyTorch) accelerate specific computing tasks through highly optimized libraries (such as cuDNN and cuBLAS, the full English names are: CUDA Deep Neural Network library and CUDA Basic Linear Algebra Subprograms, the former is optimized for deep learning frameworks, and the latter is optimized for linear algebra operations). These libraries are deeply optimized for GPU hardware and can fully utilize the computing and storage capabilities of the GPU. In addition, optimizations at the operating system, driver, compiler, and other levels will also have an important impact on the final performance of the GPU.

[0041] 2) Power and thermal management: When training large-scale models, the high-load operation of the GPU generates a lot of heat, so the GPU's heat dissipation capacity and power stability are also factors that affect performance. Insufficient heat dissipation may cause the GPU to overheat, thereby reducing performance (for example, frequency reduction) or causing system instability. Efficient thermal design and power management are essential to keep the GPU running efficiently for a long time.

[0042] 3) Precision and mixed precision computing: For deep learning tasks, especially large model training, mixed precision computing (such as FP16, BF16, where BF16 is also a 16-bit floating point number, but has fewer bits than FP16) can significantly improve computing speed and efficiency while reducing memory consumption. GPUs usually support mixed precision computing to improve performance without significantly losing model accuracy.

[0043] 4) Hardware acceleration features: In addition to Tensor Cores, GPUs may also include other hardware acceleration units, such as accelerators for specific tasks (such as video encoding and decoding, encryption and decryption, etc.). These hardware acceleration features can reduce the burden on the GPU in specific scenarios, thereby improving training efficiency.

[0044] In general, the performance of GPU plays a vital role in training large-scale deep learning models, mainly reflected in computing performance, storage performance and communication performance. The powerful parallel computing capability, huge memory bandwidth, low-latency communication mechanism, and coordinated optimization of hardware and software of GPU determine its advantages in training large models. In addition, factors such as precision optimization, heat dissipation and power management will also affect the performance of GPU. In practical applications, choosing a suitable GPU and optimizing related configurations can significantly improve training efficiency and speed up the training of large models.

[0045] Part 2: For each target model chip, the weight distribution of each preset performance type of the target model chip under each alternative training strategy is determined.

[0046] This step is intended to allow the above-mentioned execution subject to determine the weight distribution of each preset performance type of the target model chip under each alternative training strategy for each target model chip. Specifically, the above-mentioned execution subject can first determine the different effects of different training methods provided by different alternative training strategies on different types of performance of the target model chip, and then determine the weight distribution of each preset performance type of the target model chip under each alternative training strategy based on the different effects of different training methods on different types of performance of the target model chip. Finally, the weight distribution of each preset performance type of each target model chip under each alternative training strategy is obtained. Among them.

[0047] Simply put, the reason why there are multiple training strategies is that each training strategy calls or processes the computing power cluster differently for the same large model training task, that is, each training strategy provides a different training method, and the differences between different training methods will be reflected in the degree of influence on the preset performance types corresponding to the GPU, and different GPUs will also be affected accordingly due to the different performance strengths of different performance types. Still taking the three performances of computing performance, storage performance and communication performance as an example, the corresponding computing performance refers to the part of operations that need to be calculated in the training method, the storage performance corresponds to the part of operations that need to store certain data, and the communication performance refers to the part of operations that transmit certain data within the GPU or between GPUs. Different training methods usually decompose the same training task into the above three parts, and different proportions will be obtained, that is, some training methods may result in a larger number of operations related to calculations, and some training methods may result in a larger number of operations related to communication.

[0048] Therefore, this step is to determine the weight distribution of each preset performance type under each alternative training strategy based on the different effects of different training methods provided by different alternative training strategies on different preset performance types. Generally speaking, if a certain performance type is affected more (positively, that is, it is affected by increasing the number of operations corresponding to this type of performance), it can also be understood that the weight of this performance type is relatively larger than that of other performance types. Conversely, if the greater the degree of impact refers to a negative impact (that is, it refers to the impact of reducing the number of operations corresponding to this type of performance), it can also be adaptively understood that the weight of this performance type is relatively smaller than that of other performance types.

[0049] Step 203: For each candidate training strategy, the weight distribution of the candidate training strategy matched to each target model chip is used to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation; This step is intended to enable the above-mentioned execution subject to use each alternative training strategy to match the weight distribution of each target model chip respectively, and to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the proportion of the number of each corresponding target model chip, so as to obtain a comprehensive training evaluation corresponding to the alternative training strategy.

[0050] For ease of understanding, taking the alternative training strategy X as an example and assuming that there are three types of chips, Y1, Y2, and Y3, it is necessary to use the weights that match the alternative training strategy X to Y1, Y2, and Y3 respectively, and perform weighted processing on the original performance evaluations under each preset performance type of Y1, Y2, and Y3 combined with the respective quantity proportions of Y1, Y2, and Y3, so as to obtain the comprehensive training evaluation of the computing power cluster composed of Y1, Y2, and Y3 when performing training tasks according to the alternative training strategy X.

[0051] That is, step 202 and step 203 first clarify the several performance types of each type of GPU that mainly affect the ability to train large models, and obtain the original performance evaluation, and then determine the weight distribution based on the influence of different training methods provided by different alternative training strategies on these performance types, and use the weighted calculation method to clarify the actual contribution of these performance types of each type of GPU to the ability to train large models under each alternative training strategy (that is, considering the proportion of the number of GPUs of each type), and finally obtain a relatively more accurate comprehensive training evaluation corresponding to each alternative training strategy under the calculation method using the correct weight weighting. Similarly, the comprehensive training evaluation can also be an evaluation level or score evaluation.

[0052] Step 204: Determine a target training strategy based on the evaluation parameters of each comprehensive training evaluation.

[0053] On the basis of step 203, this step aims to determine the target training strategy that better meets the actual needs based on the evaluation parameters of each comprehensive training evaluation by the above-mentioned execution subject. The actual needs can be specifically manifested as the demand indicators that minimize the overall training time and consume the least resources in the overall training process. The alternative training strategy with the comprehensive evaluation parameters that better meet the demand indicators can be determined as the target training strategy.

[0054] Considering that under normal circumstances, we only need to determine the alternative training strategy with the best evaluation parameters as the target training strategy, then we can simply determine the alternative training strategy with the highest evaluation parameters as the target training strategy; among which, the highest evaluation parameters include: the highest evaluation level, the highest evaluation score, and the highest evaluation degree.

[0055] The method for determining a training strategy for a computing power cluster that uses a mixture of multiple types of chips provided by the embodiment of the present disclosure, after receiving a training task instructing to train a generative large language model, first determines the computing power cluster composed of multiple types of chips used in the training, the target type chips specifically used by the computing power cluster, and the quantity ratio of each target type chip, then determines, for each target type chip, its performance evaluation under a preset performance category and the weight distribution of each preset performance category under different alternative training strategies, then, for each alternative training strategy, uses the alternative training strategy to match the weight distribution of each target type chip respectively, and performs weighted processing on the performance evaluation of each corresponding target type chip under the preset performance category combined with the quantity ratio of each corresponding target type chip, to obtain a comprehensive training evaluation corresponding to the alternative training strategy, and finally selects a suitable target training strategy according to the evaluation parameters of each comprehensive training evaluation.

[0056] That is, by applying the technical solution provided in this embodiment, it is not necessary to determine the appropriate target training strategy by trial training each alternative training strategy one by one, thereby reducing the resource overhead and time overhead required for trial training. The target training strategy suitable for training a generative large language model in a computing power cluster composed of multiple different types of chips can be quickly determined at a low cost, so that the training task can be completed with less time and resource overhead, thereby indirectly improving the model training efficiency.

[0057] Please refer to Figure 3 , Figure 3A flowchart of a method for determining a performance evaluation of a target model chip under a preset performance category provided in an embodiment of the present disclosure is a specific implementation method for the first part mentioned in step 202: "For each target model chip, determine the performance evaluation of the target model chip under the preset performance category" in "how to determine the performance evaluation of the target model chip under the preset performance category". For ease of understanding, the description is based on a single target model chip as the perspective, wherein process 300 includes the following steps: Step 301: Determine a target performance prediction model corresponding to a target model chip; This step aims to determine the target performance prediction model corresponding to the target model chip by the above-mentioned execution subject. The target performance prediction model is used to characterize the corresponding relationship between the performance evaluations of different models of chips for various data sizes under preset performance categories, and the target performance prediction model can be pre-trained using training samples with the above-mentioned corresponding relationship.

[0058] Furthermore, considering that different types of chips under the same chip architecture often have the same chip architecture, the predicted performance between different types of chips has continuity or inertia. That is, the performance prediction model trained using the training samples of the A1 type chip and the A2 type chip under the A chip architecture can often be used when actually determining the performance evaluation of the new A3 type chip, and a relatively high-accuracy predicted performance evaluation can be obtained. However, different types of chips under different chip architectures cannot be accurately predicted due to different chip architectures. Therefore, in the case of considering the chip architecture, the matching performance prediction model can also be determined through the following steps: First, determine the target chip architecture used by the target model chip, and then determine the performance prediction model corresponding to the target chip architecture as the target performance prediction model; wherein a matching performance prediction model is pre-trained for each chip architecture, that is, assuming that there are three different chip architectures A, B, and C, a performance prediction model is trained for each of the three different chip architectures A, B, and C, for example, performance prediction model Z1 corresponds to chip architecture A, performance prediction model Z2 corresponds to chip architecture B, and performance prediction model Z3 corresponds to chip architecture C.

[0059] Step 302: Input the chip model of the target model chip into the target performance prediction model to obtain an output performance evaluation under a preset performance category.

[0060] On the basis of step 301, this step aims at having the above-mentioned execution subject input the chip model of the target model chip into the target performance prediction model, and obtain the performance evaluation under the preset performance category output by the target performance prediction model.

[0061] Specifically, the training samples used in the training of the target performance prediction model may include real performance evaluations that exceed the types of performance that are actually required to be output, that is, assuming that the target performance prediction model currently only needs to output predicted performance evaluations under computing performance, storage performance, and communication performance, then the training samples used in the training of the target performance prediction model may only include actual performance evaluations under computing performance, storage performance, and communication performance, and may also further include actual performance evaluations of other performance types. This allows the target performance prediction model to directly meet its requirements if it is required to output performance evaluations under more performance types in the future.

[0062] This embodiment provides a solution of first determining a target performance prediction model that matches a target model chip, and then using the target performance prediction model to output the required performance evaluation of the target model chip under a preset performance category through steps 301 and 302. That is, by pre-training a target performance pre-stored model that can characterize the corresponding relationship between the performance evaluations of different model chips for various data sizes under preset performance categories, it is possible to quickly obtain a relatively accurate predicted performance evaluation without the need for actual testing when the actual performance evaluation of the preset performance category of the target model chip is not clearly known, thereby reducing the time consumption of determination in the actual use stage and improving the determination efficiency.

[0063] Furthermore, in order to deepen the understanding of how to pre-train performance prediction models for different chip architectures, Figure 4 A performance prediction model training method is provided, wherein the process 400 includes the following steps: Step 401: Obtain known chip models under the same chip architecture; For example, the A1 chip and the A2 chip under the A chip architecture in the above example, where it is assumed that the A2 chip has more CUDA units and larger video memory than the A1 chip.

[0064] Step 402: obtaining actual performance evaluations of image processing chips of known chip models for various data sizes under preset performance categories; For example, obtain the actual performance evaluation of A1 model chips and A2 model chips for various data sizes under preset performance categories, such as the actual computing performance evaluation, actual storage performance evaluation and actual communication performance evaluation of 2MB, 4MB, 8MB or even larger data in terms of computing performance, storage performance and communication performance respectively.

[0065] Step 403: Taking the known chip model as sample input and the corresponding actual performance evaluation as expected sample output, constructing a training sample; That is, the training samples include multiple sample pairs, each of which is composed of a sample input of a known chip model and an expected sample output of the actual performance evaluation corresponding to the known chip model. Specifically, the sample input is not just a model name, but the core parameters corresponding to the model (such as video memory capacity, video memory type, number of cores, and computing power accuracy, etc.), so that the prediction model can learn the corresponding relationship between the differences in core parameters of different models and the degree of differences in performance parameters by using such training samples.

[0066] Step 404: Use the training samples to train the prediction model to be trained to obtain a trained performance prediction model.

[0067] This step aims to use the training samples to train the prediction model to be trained to obtain a trained performance prediction model.

[0068] It should be noted that Figure 4 The execution subject of the solution for training the performance prediction model provided may be the same execution subject as the above embodiments, for example, the same execution subject performs the same training as the above embodiments at a certain stage before executing the above embodiments. Figure 4 The performance prediction model training operation shown in the figure may also be another execution subject different from that used in the above embodiments, for example, different from Figure 1 Other servers besides the server 110 shown, for example, a server dedicated to training a performance prediction model.

[0069] In order to deepen the understanding of how to use the weighted calculation method to determine the comprehensive training evaluation corresponding to each candidate training strategy, this embodiment also uses Figure 5 A flow chart of a method for determining a comprehensive training evaluation is shown, wherein the flow 500 includes the following steps: Step 501: Determine the performance evaluation of the target model chip in terms of computing, storage and communication, and obtain computing performance evaluation, storage performance evaluation and communication performance evaluation accordingly; This step is based on the assumption that the preset performance types specifically include computing, storage and communication, and the computing performance evaluation, storage performance evaluation and communication performance evaluation of the target model chip are then determined by the above-mentioned execution subject.

[0070] The computing performance evaluation, storage performance evaluation and communication performance evaluation may be evaluation levels that are differentiated by semantic description, or may be evaluation scores that are differentiated by numerical values, which are not specifically limited here.

[0071] Step 502: Determine the weight distribution of computing performance, storage performance and communication performance under each candidate training strategy according to the different effects of different training methods on computing performance, storage performance and communication performance. This step aims to further determine the weight distribution of computing performance, storage performance and communication performance under each alternative training strategy according to the different effects of different training methods on computing performance, storage performance and communication performance. For example, the weight distribution can be expressed as: computing performance: storage performance: communication performance = 2:2:1, and it can be seen that the alternative training strategy corresponding to this weight distribution will contain relatively more calculation-related operations and storage-related operations, and relatively fewer communication-related operations.

[0072] Step 503: For each candidate training strategy, the following comprehensive training evaluation determination steps are performed:

[0073] Step 5031: Determine the weight distribution of candidate training strategies that match each target model chip; Continuing with the above example, for example, the current candidate training strategies match the weights of computing performance, storage performance, and communication performance of three types of chips, Y1, Y2, and Y3, respectively: 2:2:1 (corresponding to Y1), 2:2:1 (corresponding to Y2), and 3:3:1 (corresponding to Y3). It can be seen that even if the same candidate training strategy acts on different types of chips, the degree of impact on the same preset performance types may be different. The reason should be mainly attributed to the different performance strengths of different types of chips themselves. For example, some types of chips specifically enhance computing performance, while some types of chips specifically enhance communication performance.

[0074] Step 5032: For each target model chip, perform the following weighted processing steps: Step 50321: Determine the weights corresponding to the computing performance, storage performance, and communication performance of the target chip using the weight distribution of the target chip; Step 50322: using the weights corresponding to the computing performance, storage performance and communication performance of the target chip, weighted processing is performed on the corresponding computing performance evaluation, storage performance evaluation and communication performance evaluation to obtain weighted computing performance evaluation, storage performance evaluation and communication performance evaluation; Continuing with the above example, under the alternative training strategy X, taking the Y1 model chip as an example, if the sum of the total weights needs to satisfy unit 1, the weight corresponding to the computing performance will be adaptively changed to: 0.4, that is, 0.4×computing performance evaluation; the weight corresponding to the storage performance will be adaptively changed to: 0.4, that is, 0.4×storage performance evaluation; the weight corresponding to the communication performance will be adaptively changed to: 0.2, that is, 0.2×communication performance evaluation.

[0075] That is, 0.4+0.4+0.2=1. If there is no need to adaptively change the original weight size according to unit 1, the weighted performance evaluation needs to be standardized to unify the comparison standard.

[0076] Step 50323: Based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity ratio corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated; Continuing from the above example, under the alternative training strategy X, the sub-comprehensive training evaluation corresponding to the Y1 model chip should be the number of Y1 model chips corresponding to the proportion and then the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation of the Y1 model chip should be weighted twice or similarly processed. Step 5033: Obtain a comprehensive training evaluation corresponding to the candidate training strategy according to the sub-comprehensive training evaluations corresponding to each target model chip; Continuing from the previous example, this step aims to obtain the comprehensive training evaluation of the computing power cluster composed of the three types of chips Y1, Y2 and Y3 when performing the training task according to the alternative training strategy X by summarizing the sub-comprehensive training evaluations of the three types of chips Y1, Y2 and Y3 under the alternative training strategy X.

[0077] An implementation including but not limited to can be found in Figure 6 The flowchart of a method for determining comprehensive training evaluation based on performance scoring is shown, still taking a target model chip as the starting point, and the process 600 includes the following steps: Step 601: quantizing the weighted evaluation parameters of computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip to obtain a computing performance score, a storage performance score and a communication performance score; That is, this embodiment quantifies the evaluation parameters of the weighted computing performance evaluation, storage performance evaluation, and communication performance evaluation to obtain computing performance scores, storage performance scores, and communication performance scores that are easier to distinguish between high and low performance and the degree of high and low performance. There are many ways to quantify, as long as they can meet the needs, and no specific restrictions are made here.

[0078] For example, taking the Y1 model chip as an example, this step aims to quantify the evaluation parameters of its corresponding weighted computing performance evaluation, storage performance evaluation and communication performance evaluation, and then obtain corresponding scores.

[0079] Step 602: performing an addition operation on the computing performance score, the storage performance score and the communication performance score to obtain a performance score sum; This step aims to obtain the sum of the computing performance score, the storage performance score and the communication performance score by the above-mentioned execution subject through addition operation to obtain the performance score sum.

[0080] Step 603: Based on the performance score and the percentage of the number of chips corresponding to the target model, determine the sub-comprehensive training evaluation corresponding to the alternative training strategy.

[0081] On the basis of step 602, this step aims to determine the sub-comprehensive training evaluation corresponding to the alternative training strategy based on the performance score and the percentage of the number corresponding to the target model chip by the above-mentioned execution subject, that is, the sub-comprehensive evaluation training is determined at least according to the performance score and the percentage of the number of the target model chip. For example, the product of the performance score and the percentage of the number corresponding to the target model chip can be directly used as the sub-comprehensive training evaluation corresponding to the alternative training strategy, or the sub-comprehensive training evaluation can be determined based on the product combined with some other coefficients or bases.

[0082] Also refer to Figure 6 The provided solution adapts step 5033 to obtain the following implementation steps: First, the sub-comprehensive training evaluations corresponding to each target model chip are quantized to obtain each sub-comprehensive training score, and then each sub-comprehensive training score is added to obtain the sum of the sub-comprehensive training scores, and finally the comprehensive training evaluation corresponding to the alternative training strategy is determined based on the sum of the sub-comprehensive training scores. That is, it can also be quantized and summed, or the sum of the sub-comprehensive training scores can be directly used as the comprehensive training evaluation corresponding to the alternative training strategy, or the comprehensive training evaluation can be determined based on the sum of the sub-comprehensive training scores combined with some other coefficients or bases. On the basis of any of the above embodiments, in order to reduce the amount of calculation as much as possible and avoid invalid calculations, before entering the calculation comprehensive training evaluation provided by step 203, the alternative training strategies that do not meet the requirements of the actual received training tasks (for example, they obviously do not meet the requirements at the training rule level) and / or the alternative training strategies that have obvious video memory conflicts (for example, strategies in which the required storage amount obviously exceeds the available video memory ratio) can be removed from all the alternative training strategies, thereby obtaining the remaining training strategies, so that the subsequent calculation only needs to be based on the performance evaluation after weighting each corresponding performance type according to each weight distribution, and the comprehensive training evaluation corresponding to each remaining training strategy can be calculated.

[0083] Based on any of the above embodiments, considering that the above steps of determining the comprehensive training evaluation corresponding to each candidate training strategy actually perform data calculation operations, and compared with the GPU, the CPU obviously has more powerful data calculation capabilities, so all or part of the calculation tasks of the comprehensive training evaluation can be assigned to the CPU to perform: For example, the above-mentioned execution subject can control the CPU to create multiple parallel computing threads, and control each computing thread to use any alternative training strategy to be calculated to match the weight distribution of each target model chip, and perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance type combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

[0084] That is, a message queue can be set up in advance, and the alternative training strategies that need to be subjected to comprehensive training evaluation calculations can be stored in the message queue, and the CPU can enable each of its computing threads to take out a specific alternative training strategy from the message queue respectively, thereby completing the calculation operation of comprehensive training evaluation of the obtained alternative training strategies in each computing thread.

[0085] If only some of the alternative training strategies to be subjected to comprehensive training evaluation are stored in the message queue, it is equivalent to the CPU playing an auxiliary computing role in this process. If all the alternative training strategies to be subjected to comprehensive training evaluation are stored in the message queue, it is equivalent to the CPU replacing the GPU to play an accelerated computing role in this process.

[0086] On the basis of any of the above embodiments, taking into account that the computing power cluster uses a mixture of multiple different types of chips, in order to facilitate the connection of chips of different models to the computing power cluster or enable them to correctly respond to the scheduling of the computing power cluster, each target model chip can also be connected to the computing power cluster and accept the scheduling of the computing power cluster through a unified basic operation instruction pre-defined for each target model chip. The unified basic operation instruction is a basic operation instruction supported by each target model chip.

[0087] On the basis of any of the above embodiments, after selecting the target training strategy suitable for the computing cluster composed of the target model chips, the above execution subject (for example Figure 1 The server 110 shown in FIG. 1 controls a computing cluster (eg, Figure 1The computing cluster 130 shown in the figure performs the training task of the generative large language model according to the target training strategy. It should be noted that since the computing cluster used in each embodiment of the present disclosure is constructed by multiple types of graphics processing chips, each type of graphics processing chip often has different processing capabilities. Therefore, when executing the training task, the processing tasks (including computing tasks, storage tasks, and communication tasks) assigned to different target model chips should match the performance strengths thereof, that is, they are usually uneven, and each processing task is decomposed from the overall training task facing the computing cluster.

[0088] An implementation including but not limited to can be found in Figure 7 The flowchart shown in FIG. 700 includes the following steps: Step 701: Split the training task of the generative large language model into tasks to be processed corresponding to each target model chip according to the actual performance and quantity ratio of different target model chips; Step 702: Send each task to be processed to the corresponding target model chip; Step 703: Control each target model chip to execute the received tasks to be processed according to the target training strategy.

[0089] That is, this embodiment first splits the overall training task according to the actual performance and quantity ratio of different target model chips, and then sends the different tasks to be processed to the corresponding target model chips, so that each target model chip can finally execute the received tasks to be processed according to the target training strategy.

[0090] Since the tasks to be processed can be further divided into computing tasks, communication tasks, and storage tasks according to the task type, the following methods can be used to improve the efficiency when executing different types of tasks: For the first inter-chip communication task in the tasks to be processed, the execution subject may control each target model chip to communicate with another target model chip in the same communication domain through a collective communication library (CCL). For different chips that can be in the same communication domain, the inter-chip communication method provided by CCL has a higher communication rate and a higher data throughput per unit time. In order to clarify which chips can be in the same communication domain, an active detection component can be added to confirm which models of chips can be in the same communication domain by actively sending inquiries or test packets to each target model chip.

[0091] For the second inter-chip communication task in the tasks to be processed, each target model chip is controlled to communicate with another target model chip in a different communication domain in a forwarding manner through the central processor.

[0092] For different chips that cannot be in the same communication domain, the inter-chip communication method provided by CCL cannot be used, and the only way to achieve inter-chip communication is to forward communication through the CPU. On the basis of the conventional technology of forwarding communication through the CPU, the standard method of forwarding communication through the CPU can be optimized by combining remote direct data access technology (RDMA, Remote Direct Memory Access) and / or (Host-Device, referring to GPU and CPU) copy pipelining technology to improve the communication rate and data throughput per unit time under this method.

[0093] Furthermore, for the computing tasks in the tasks to be processed, each target model chip can be controlled to call the corresponding large model operator to perform the corresponding computing operation according to the unified large model operator calling instruction corresponding to the computing task.

[0094] That is, the unified large model operator calling instruction is a large model operator calling instruction that is pre-defined for each target model chip and can be recognized and run by each target model chip. The large model operator calling instruction is used to call the operator provided by the graphics processing chip to support the training of the generative large language model.

[0095] To deepen understanding, this disclosure also combines specific application scenarios and provides a complete set of implementation solutions: In order to achieve mixed training of multiple chips with high computing efficiency, several major problems need to be solved: 1. Rapid access to multiple chips. Each chip can be quickly accessed, and the changes must be small enough; 2. Multiple chips must be able to communicate on a large scale, so that after a large model is split onto different chips, parameter synchronization during training can proceed normally; 3. Heterogeneous slicing strategy: Different chips have different specifications and parameters. In order to fully exert the computing power, the configuration of strategies such as recalculation is different on different chips; 4. Automatic parallel strategy optimization. After the introduction of heterogeneous segmentation strategies, the entire strategy space has increased by several orders of magnitude compared to single-chip training. Traditional manual tuning alone cannot fully utilize the computing power.

[0096] Therefore, the existing related technologies lack the following solutions: 1. Lack of a fast access mechanism for multiple chips. The access to the underlying operators of each chip requires developers to have a very good understanding of the architecture of the solution, which greatly increases the cost of secondary development to access new chips; 2. The communication between chips only uses the cross-CPU forwarding mechanism. There is no differentiated consideration for different communication environments. For example, different models of GPUs may still communicate at high speed through NCCL (NVIDIA CCL, specifically a high-performance collective communication library developed by NVIDIA). This design will affect the overall training performance; 3. It does not support heterogeneous slicing strategies. Even if the chips with different video memory have the same strategies in the tasks, 4. There is no automatic parallel mechanism, which requires users to have very strong parallel strategy tuning capabilities or do a lot of testing experiments.

[0097] In order to solve the above problems, we first define a global MFU (global computing power or global throughput) indicator to describe the actual computing power of the hybrid training system. The formula is as follows: ; Achieved TPS: the actual throughput achieved, in Tokens / device / sec. N: model size, unit b; L: number of model layers; H: Model Hidden size; S: Model Sequence length, the number of tokens in each data; : The number of the i-th chip; : The theoretical maximum computing power of the ith chip, in Tflops / sec.

[0098] With a single chip, the MFU can reach more than 60% when training a large Dense model (a deep neural network model consisting of a large number of fully connected layers). In order to quickly access various chips (the access cost of the current solution is very high), and in the case of multi-core mixed training, it can also approach this level (the current solution is only less than 50%). The overall solution diagram can be seen in Figure 8-1 .

[0099] Specifically, this solution provides the following improvement mechanisms: 1) Multi-core access framework: Combining the current large model structure and the characteristics of the underlying chip, a device operator layer custom_device is abstracted. In this layer, basic operations describing the device (such as context maintenance, memory operation, signal synchronization, etc.) and operators custom_ops required by the large model are defined, such as RMSNorm (a lightweight normalization method, commonly used in Transformer models), Self-Attention (capable of capturing the relationship between elements in a sequence and widely used in Transformer models, literally translated as self-attention mechanism in Chinese), GLU (an activation function that combines the gating mechanism and linear transformation, which can enhance the representation ability of the model) three common operators and other operators. In this way, developers or users do not need to pay special attention to the current large model framework, but only need to implement the corresponding operators and operations according to the abstraction. custom_device will register the underlying implementation of these chips into the framework.

[0100] 2) Flexible inter-chip communication mechanism. By default, after a new chip is registered, inter-chip communication is implemented according to the CPU forwarding logic, which ensures that multi-core hybrid training can run. But on this basis, this solution adds a communication capability detection function. During initialization, it will detect whether various chips can join the same communication domain (mainly for GPUs of different models). If so, they will communicate directly through CCL (or NCCL) (this solution can also be expanded to more chips that follow the same protocol in the future), otherwise it will degenerate to CPU forwarding communication. In the case of simultaneous GPUs, MFU can improve by nearly 5 points.

[0101] In addition, the following optimizations will be made to improve performance under the CPU forwarding communication mechanism to ensure that the underlying communication is not a bottleneck: ①: Host-Device copy pipeline technology hides the copy overhead between GPU and CPU; ②: Use RDMA technology to create multiple connections to communicate together.

[0102] 3) Support for heterogeneous parallel splitting strategies. This solution has made major changes to the framework. It supports configuring different operation strategies at the by layer, including PP (Pipeline Parallelism), TP (Tensor Parallelism) and DP (Data Parallelism) splitting, recalculation strategies, and graphics memory offload strategies. In this way, after different chips are assigned tasks, they can set fine-grained strategies according to the characteristics of their own chips, and the framework will ensure the correctness of strategy synchronization.

[0103] Specifically, considering the different capabilities of each chip, in order to balance the load, it supports splitting into different numbers of PP stages (literally translated as: different stages under pipeline parallelism); based on the characteristic that the memory overhead of PP stages decreases as the PP rank (rank is used to refer to different GPUs, and different GPUs in each parallel group have different rank numbers), the front stages use cards with larger memory, and the back chips use cards with smaller memory; enable the PP stage granularity recalculation strategy, allocate fewer Layers (network layers, functional layers), use more recalculations, and meet the memory requirements while ensuring load balancing; fine-grained segmentation, the task unit of a single PP stage is Attention (attention mechanism), MLP (classic artificial neural network, composed of multiple fully connected layers stacked) or Attention + MLP (Transfomer Layer, Transformer layer, where Attention and MLP work in series in sequence, and each part has its own input and output), and on this basis, uneven segmentation is performed on the PP stages.

[0104] Similar to the uneven splitting of the PP layer, in order to balance the load, uneven micro batch size (referring to the amount of data processed by the model at one time in each training step, for each training step) splitting can also be supported between different TPs and DPs.

[0105] 4) Supports automatic parallel strategy for multi-core hybrid training. This solution abstracts the operating performance of various operations of various chips. When a task is initiated, the solution will detect all cluster environments, understand the specifications and characteristics of all chips, and then automatically traverse all possible segmentation strategies, calculate the cost of each strategy, and find the optimal parallel strategy for MFU. Combined with the above heterogeneous parallel segmentation strategy, MFU can reach more than 57% in a 100-card GPU training environment, which is only 3% lower than the loss of single-chip training.

[0106] In addition to the above MFU formula, we can also abstract the time it takes to train a specified large model in one iter under a certain strategy. The formula is as follows: ; Among them, time_cost: the time spent on an iter (which can be simply understood as a training stage or an iterative training), which can be converted into training throughput or computing power MFU for training tasks. The smaller the index, the more it meets the actual needs and has a lower cost, and the more it should be selected as the target training strategy; gemm_ops: the number of computational operations performed on each matrix; flops: the actual floating-point computing capability per cycle of the chip at each matrix size; mem_ops: the number of memory access operations performed by each non-matrix operation; mem_bws: memory access bandwidth of the chip at various memory access operation sizes; comm_ops: the amount of communication required under different communication modes; comm_bws: Communication bandwidth of the chip under different communication models and sizes.

[0107] The above two formulas can be linked using the following formula for flexible conversion: = ; Among them, B: global batchsize, the number of data entries input in each iteration.

[0108] In order to obtain some indicators in the above formula, the following method is designed, see Figure 8-2 ,exist Figure 8-2 In the implementation shown, the following mechanisms are mainly used to achieve the above time estimation: 1. Basic performance of the chip: 1) Offline testing of the computing, storage, and communication performance of various chips under various data sizes; 2) An XGBoost model (an integrated learning model based on the gradient boosting framework, which iteratively generates multiple decision trees to improve the prediction accuracy of the model. The basic idea is to combine multiple tree models with low classification accuracy into a model with higher accuracy) is used for fitting. In this way, the computing MFU CostModel, memory bandwidth CostModel, and communication bandwidth CostModel under different chips are obtained, which are used to subsequently calculate the performance of different data volumes under different strategies.

[0109] That is, in the early offline stage, it is necessary to use the real performance parameters of some known chip models to train and build the XGBoost model, so that for the new chip models input later, the XGBoost model that has learned the relevant knowledge can output the predicted performance parameters for the relevant performance types according to the chip hardware parameters that constitute the new chip models.

[0110] 2. Strategy abstraction: 1) According to the logic of various parallel strategies, the computational graph of the entire model is divided to obtain the required computational load, memory access, communication volume, storage volume, and pipeline bubble under different strategies; 2) To consider the scenario of insufficient video memory, the offline profile mechanism (which can be understood as a trial run mechanism, that is, the video memory parameters are clarified by providing similar offline running conditions) is used to obtain the size of the video memory system buffer, temporary variables, and other spaces, thereby determining the proportion of video memory available for model training.

[0111] 3. Strategy space: 1) Traverse all strategy combinations and filter them through some simple strategies (such as strategies that obviously do not comply with the rules), thereby reducing the strategy space that needs to be calculated; 2) Traverse all candidate strategies, calculate the required storage amount, compare it with the available video memory ratio, and exclude strategies that will cause video memory OOM (insufficient video memory); 3) Traverse the remaining strategies, call the strategy abstraction to obtain the required computing, memory access, and communication volume, and then calculate the time_cost of the above formula with the basic performance set of the chip, and select the strategy with the smallest time_cost, that is, the strategy with the best performance. This process can be seen in Figure 8-3 (Taking training strategies A and B as examples, the entire task execution time is calculated based on the operator execution time + waiting time of the last group of chips, and these parameters are specifically summarized by the bubbles before and after the previous chips, the execution time of various types of operators, the synchronization point communication waiting time, the recompute execution time and the recalculation time, etc.)

[0112] In the implementation scheme provided in the above content of this embodiment, compared with the existing related technologies, in addition to traversing test and profile related operations, it also has the following advantages and characteristics: 1) In addition to the basic offline performance test in the early stage, no actual resources are required during the model development process to search for the optimal parallel strategy; 2) The entire process is purely computational, and the strategies are not dependent on each other, so the CPU can be used for parallel computing, and the optimal strategy can be found in minutes; 3) The coverage strategy is wider than the previous solution, and the new strategy only requires logical abstraction, so the accuracy is very high, reaching 95%.

[0113] By applying the technical solution provided in this embodiment, the access and efficiency issues of mixed training of multiple chips can be better solved. Compared with the previous solutions, there are two significant improvements: 1) When connecting a new chip, users do not need to understand the architecture of the solution and can connect it according to standard specifications; 2) Through automatic search, heterogeneous parallel strategies and underlying communication optimization, the efficiency of mixed training of multiple chips can be improved to a level close to that of single chip training, making the solution more widely used.

[0114] It is currently found that the following scenarios have good application potential: 1) Training of ultra-large models. For example, GPT5 (1.75 trillion parameters) requires 50,000 H100s, and GPT6 requires 100,000 H100s. Single chip supply cannot meet the demand. A mixed supply solution of multiple chips can be used, mainly for some intelligent computing centers or enterprises that train trillion-scale models. 2) Expansion of existing clusters. Some companies have already purchased A800 and other computing power. Due to supply constraints, they purchase domestic computing power chips. In order to make full use of the existing computing power, new and old chips form a mixed cluster, mainly in some privatization scenarios; 3) Reuse of existing clusters: Some companies have purchased a variety of computing power in the past and want to upgrade their technology through large models. The existing cluster is a hybrid cluster, which is mainly used in some privatized scenarios.

[0115] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training strategy determination device for a computing power cluster that uses a mixture of chips of various models. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0116] like Fig. 9 As shown, the training strategy determination device 800 for a computing power cluster that uses a variety of mixed chip models in this embodiment may include: a model and quantity ratio determination unit 901, a performance evaluation and weight distribution determination unit 902, a comprehensive training evaluation determination unit 903, and a target training strategy determination unit 904. Among them, the model and quantity ratio determination unit 901 determines each target model chip and the quantity ratio of each target model chip that constitutes the computing power cluster; the performance evaluation and weight distribution determination unit 902 is configured to determine, for each target model chip, the performance evaluation of the target model chip under a preset performance category and the weight distribution of each preset performance category under each alternative training strategy; the comprehensive training evaluation determination unit 903 is configured to use the weight distribution of the alternative training strategy that matches each target model chip to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity ratio of each corresponding target model chip for each alternative training strategy, and obtain the corresponding comprehensive training evaluation; the target training strategy determination unit 904 is configured to determine the target training strategy based on the evaluation parameters of each comprehensive training evaluation.

[0117] In this embodiment, for the training strategy determination device 900 for a computing power cluster that uses a mixture of multiple types of chips: the specific processing of the model and quantity ratio determination unit 901, the performance evaluation and weight distribution determination unit 902, the comprehensive training evaluation determination unit 903, and the target training strategy determination unit 904 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0118] In some optional implementations of this embodiment, the model and quantity ratio determining unit 901 may be further configured to: According to the model information of various image processing chips used by the computing power cluster, each target model chip is obtained; Based on the actual number of chips of each target model in the computing power cluster and the total number of chips, the proportion of chips of each target model is calculated.

[0119] In some optional implementations of this embodiment, the performance evaluation and weight distribution determination unit 902 may include a performance evaluation determination subunit configured to determine the performance evaluation of the target model chip under a preset performance category, and the performance evaluation determination subunit may include: A target performance prediction model determination module is configured to determine a target performance prediction model corresponding to a target model chip; wherein the target performance prediction model is used to characterize the corresponding relationship between performance evaluations of different models of chips for various data sizes under preset performance categories; The model using subunit is configured to input the chip model of the target model chip into the target performance prediction model to obtain an output performance evaluation under a preset performance category.

[0120] In some optional implementations of this embodiment, the target performance prediction model determination module may be further configured to: Determine the target chip architecture used by the target model chip; A performance prediction model corresponding to the target chip architecture is determined as a target performance prediction model; wherein a matching performance prediction model is pre-trained for each chip architecture.

[0121] In some optional implementations of this embodiment, the training strategy determination device 900 for a computing power cluster that uses a mixture of chips of multiple models may further include: a performance prediction model training unit, and the performance prediction model training unit may be further configured to: Get known chip models under the same chip architecture; Obtaining actual performance evaluation of image processing chips of known chip models for various data sizes under preset performance categories; The known chip model is used as sample input and the corresponding actual performance evaluation is used as the expected sample output to construct the training sample; The training samples are used to train the prediction model to be trained to obtain a trained performance prediction model.

[0122] In some optional implementations of this embodiment, the performance evaluation and weight distribution determination unit 902 may include a first weight distribution determination subunit configured to determine the weight distribution of each preset performance category under each candidate training strategy, and the first weight distribution determination subunit may include: A different impact determination module is configured to determine different impacts of different training methods provided by different candidate training strategies on different types of performance of the target model chip; The weight distribution determination module is configured to determine the weight distribution of each preset performance type of the target model chip under each alternative training strategy according to the different impacts of different training methods on different types of performance of the target model chip; wherein the weight of any type of performance is proportional to the degree of impact.

[0123] In some optional implementations of this embodiment, the preset performance type includes: at least one of computing performance, storage performance, and communication performance; Correspondingly, the performance evaluation and weight distribution determining unit 902 may include a performance evaluation determining subunit configured to determine the performance evaluation of the target model chip under a preset performance category, and the performance evaluation determining subunit may be further configured to: Determine the performance evaluation of the target model chip in terms of computing performance, storage performance and communication performance, and obtain computing performance evaluation, storage performance evaluation and communication performance evaluation accordingly; Correspondingly, the first weight distribution determination module may be further configured as follows: According to the different impacts of different training methods on computing performance, storage performance and communication performance, the weight distribution of computing performance, storage performance and communication performance under each alternative training strategy is determined.

[0124] In some optional implementations of this embodiment, the comprehensive training evaluation determination unit 903 may include: A second weight distribution determination subunit is configured to determine the weight distribution of the candidate training strategies that are matched to each target model chip; The weighted processing execution subunit is configured to perform the following weighted processing steps for each target model chip: using the weight distribution of the target model chip to determine the weights corresponding to the computing performance, storage performance and communication performance of the target model chip; using the weights corresponding to the computing performance, storage performance and communication performance of the target model chip, respectively weighted processing is performed on the corresponding computing performance evaluation, storage performance evaluation and communication performance evaluation to obtain the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation; based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity proportion corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated; The comprehensive evaluation determination subunit is configured to obtain a comprehensive training evaluation corresponding to the candidate training strategy based on the sub-comprehensive training evaluations corresponding to each target model chip.

[0125] In some optional implementations of this embodiment, based on the weighted computing performance evaluation, storage performance evaluation, and communication performance evaluation corresponding to the target model chip and the quantity proportion corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated, including: Quantifying the evaluation parameters of the weighted computing performance evaluation, storage performance evaluation, and communication performance evaluation corresponding to the target model chip to obtain a computing performance score, a storage performance score, and a communication performance score; Add the computing performance score, storage performance score and communication performance score to obtain the performance score sum; Based on the performance score and the percentage of the number of chips corresponding to the target model, the sub-comprehensive training evaluation corresponding to the alternative training strategy is determined.

[0126] In some optional implementations of this embodiment, based on the performance score and the number ratio corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is determined, including: The product of the performance score and the percentage of the number of chips corresponding to the target model is used as the sub-comprehensive training evaluation corresponding to the alternative training strategy.

[0127] In some optional implementations of this embodiment, the comprehensive evaluation determination subunit may include: A quantization processing module is configured to quantize the sub-comprehensive training evaluations corresponding to each target model chip to obtain each sub-comprehensive training score; A summation processing module is configured to perform an addition operation on each sub-comprehensive training score to obtain a sum of the sub-comprehensive training scores; The comprehensive training evaluation determination module is configured to obtain a comprehensive training evaluation corresponding to the candidate training strategy based on the sub-comprehensive training scores and determination.

[0128] In some optional implementations of this embodiment, the comprehensive training evaluation determination module may be further configured to: The sum of the sub-comprehensive training scores is used as the comprehensive training evaluation corresponding to the alternative training strategies.

[0129] In some optional implementations of this embodiment, the training strategy determination device 900 for a computing power cluster that uses a mixture of chips of multiple models may also include: A screening processing unit is configured to remove the candidate training strategies that do not meet the requirements of the actually received training tasks and / or remove the candidate training strategies that have obvious video memory conflicts from all the candidate training strategies to obtain the remaining training strategies; Correspondingly, the comprehensive training evaluation determination unit 903 may be further configured as follows: For each remaining training strategy, the remaining training strategy is used to match the weight distribution of each target model chip, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

[0130] In some optional implementations of this embodiment, the comprehensive training evaluation determination unit 903 may be further configured to: Control the central processing unit to create multiple parallel computing threads, and control each computing thread to use any alternative training strategy to be calculated to match the weight distribution of each target model chip, and perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

[0131] In some optional implementations of this embodiment, the target training strategy determination unit 904 may be further configured to: The alternative training strategy with the highest evaluation parameter is determined as the target training strategy; wherein the highest evaluation parameter includes: the highest evaluation level, the highest evaluation score, and the highest evaluation degree.

[0132] In some optional implementations of this embodiment, the training strategy determination device 900 for a computing power cluster that uses a mixture of chips of multiple models may also include: The multi-model chip access and scheduling acceptance unit is configured to connect each target model chip to the computing power cluster and accept the scheduling of the computing power cluster through unified basic operation instructions pre-defined for each target model chip; wherein the unified basic operation instructions are basic operation instructions supported by each target model chip.

[0133] In some optional implementations of this embodiment, the training strategy determination device 900 for a computing power cluster that uses a mixture of chips of multiple models may also include: The training task control execution unit is configured to control the computing power cluster to execute the training task of the generative large language model according to the target training strategy; wherein, different target model chips under the computing power cluster will be assigned to pending tasks that match their actual performance, and the pending tasks are decomposed from the training tasks.

[0134] In some optional implementations of this embodiment, the training task control execution unit may include: The task splitting subunit is configured to split the training task of the generative large language model into tasks to be processed corresponding to each target model chip according to the actual performance and quantity ratio of different target model chips; The sending subunit is configured to send each task to be processed to the corresponding target model chip; The control execution subunit is configured to control each target model chip to execute the received tasks to be processed according to the target training strategy.

[0135] In some optional implementations of this embodiment, the control execution subunit may include: A first inter-chip communication task execution control module is configured to control each target model chip to communicate with another target model chip in the same communication domain through a collective communication library for the first inter-chip communication task in the tasks to be processed; The second inter-chip communication task execution control module is configured to control each target model chip to communicate with another target model chip in a different communication domain through forwarding by the central processor for the second inter-chip communication task in the task to be processed.

[0136] In some optional implementations of this embodiment, the second inter-chip communication task execution control module may be further configured to: Each target model chip is controlled to communicate with another target model chip in a different communication domain by adopting a remote direct data access technology and / or a central processor forwarding method improved by copying pipeline technology.

[0137] In some optional implementations of this embodiment, the control execution subunit may include: The computing task control execution module is configured to control each target model chip to call the corresponding large model operator to perform corresponding computing operations according to the unified large model operator calling instruction corresponding to the computing task for the computing task to be processed; wherein the unified large model operator calling instruction is a large model operator calling instruction pre-defined for each target model chip and can be recognized and run by each target model chip, and the large model operator calling instruction is used to call the operator provided by the graphics processing chip to support the training of the generative large language model.

[0138] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The training strategy determination device for a computing power cluster that uses a mixture of multiple types of chips provided by this embodiment, after receiving a training task instructing to train a generative large language model, first determines the computing power cluster composed of multiple types of chips used in the training, as well as the target type chips specifically used by the computing power cluster and the quantity ratio of each target type chip. Then, for each target type chip, its performance evaluation under a preset performance type and the weight distribution of each preset performance type under different alternative training strategies are determined. Then, for each alternative training strategy, the alternative training strategy is used to match the weight distribution of each target type chip respectively, and the performance evaluation of each corresponding target type chip under the preset performance type is combined with the quantity ratio of each corresponding target type chip to perform weighted processing, so as to obtain a comprehensive training evaluation corresponding to the alternative training strategy, and finally, a suitable target training strategy is selected according to the evaluation parameters of each comprehensive training evaluation. That is, by applying the technical solution provided in this embodiment, it is not necessary to determine the appropriate target training strategy by trial training each alternative training strategy one by one, thereby reducing the resource overhead and time overhead required for trial training. The target training strategy suitable for training a generative large language model in a computing power cluster composed of multiple different types of chips can be quickly determined at a low cost, so that the training task can be completed with less time and resource overhead, thereby indirectly improving the model training efficiency.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the training strategy determination method for a computing power cluster that uses a mixture of multiple chip models as described in any of the above embodiments when executing.

[0140] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for determining a training strategy for a computing power cluster that uses a mixture of multiple chip models as described in any of the above embodiments when executed.

[0141] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the training strategy determination method for a computing power cluster that uses a mixture of multiple types of chips as described in any of the above embodiments.

[0142] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0143] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0144] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0145] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a training strategy determination method for a computing power cluster that uses a variety of mixed chip models. For example, in some embodiments, a training strategy determination method for a computing power cluster that uses a variety of mixed chip models may be implemented as a computer software program that is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training strategy determination method for a computing power cluster that uses a variety of mixed chip models described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other appropriate manner (for example, by means of firmware) to execute a training strategy determination method for a computing power cluster that uses a mixture of chips of various models.

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0151] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0152] According to the technical solution of the embodiment of the present disclosure, after receiving a training task instructing to train a generative large language model, first determine the computing power cluster composed of multiple types of chips used in the training, the target type chips specifically used by the computing power cluster, and the number ratio of each target type chip. Then, for each target type chip, determine its performance evaluation under a preset performance type and the weight distribution of each preset performance type under different alternative training strategies. Then, for each alternative training strategy, use the alternative training strategy to match the weight distribution of each target type chip respectively, and perform weighted processing on the performance evaluation of each corresponding target type chip under the preset performance type combined with the number ratio of each corresponding target type chip to obtain a comprehensive training evaluation corresponding to the alternative training strategy. Finally, select a suitable target training strategy according to the evaluation parameters of each comprehensive training evaluation. That is, by applying the technical solution provided in this embodiment, it is not necessary to determine the appropriate target training strategy by trial training each alternative training strategy one by one, thereby reducing the resource overhead and time overhead required for trial training. The target training strategy suitable for training a generative large language model in a computing power cluster composed of multiple different types of chips can be quickly determined at a low cost, so that the training task can be completed with less time and resource overhead, thereby indirectly improving the model training efficiency.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a training strategy for a computing cluster that uses a mixture of chips of various models, comprising: Determine the target model chips and the quantity ratio of each target model chip that constitutes the computing power cluster; For each target model chip, determining the performance evaluation of the target model chip under a preset performance category and the weight distribution of each preset performance category under each candidate training strategy; For each of the candidate training strategies, the weight distribution of the candidate training strategies respectively matched to each of the target model chips is used to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation; Based on the evaluation parameters of each of the comprehensive training evaluations, a target training strategy is determined.

2. The method according to claim 1, wherein: Determine the target model chips and the quantity ratio of each target model chip that constitutes the computing power cluster, including: According to the model information of the multiple image processing chips used by the computing power cluster, the target model chips are obtained; According to the actual number of each target model chip in the computing power cluster and the total number of chips, the quantity ratio of each target model chip is calculated.

3. The method according to claim 1, wherein: Determining the performance evaluation of the target model chip under a preset performance category includes: Determine a target performance prediction model corresponding to the target model chip; wherein the target performance prediction model is used to characterize the corresponding relationship between the performance evaluations of different model chips for various data sizes under the preset performance category; The chip model of the target model chip is input into the target performance prediction model to obtain an output performance evaluation under the preset performance category.

4. The method according to claim 3, wherein: Determining a target performance prediction model corresponding to the target model chip includes: Determine the target chip architecture adopted by the target model chip; A performance prediction model corresponding to the target chip architecture is determined as the target performance prediction model; wherein a matching performance prediction model is pre-trained for each chip architecture.

5. The method according to claim 4, wherein: The performance prediction model is trained by the following steps: Get known chip models under the same chip architecture; Obtaining actual performance evaluation of the image processing chip of the known chip model for various data sizes under the preset performance category; The known chip model is used as a sample input and the corresponding actual performance evaluation is used as an expected sample output to construct a training sample; The training samples are used to train the prediction model to be trained to obtain a trained performance prediction model.

6. The method according to claim 1, wherein: Determining the weight distribution of each of the preset performance categories under each alternative training strategy includes: Determine different effects of different training methods provided by different candidate training strategies on different types of performance of the target model chip; According to the different impacts of the different training methods on the different types of performance of the target model chip, the weight distribution of each preset performance type of the target model chip under each of the alternative training strategies is determined; wherein the weight of any type of performance is proportional to the degree of impact.

7. The method according to claim 6, wherein: The preset performance types include: at least one of computing performance, storage performance and communication performance; Correspondingly, determining the performance evaluation of the target model chip under a preset performance category includes: Determine the performance evaluation of the target model chip in terms of computing performance, storage performance and communication performance, respectively, and obtain computing performance evaluation, storage performance evaluation and communication performance evaluation accordingly; Correspondingly, according to the different effects of the different training methods on the different types of performance of the target model chip, the weight distribution of each preset performance type of the target model chip under each of the alternative training strategies is determined, including: According to the different effects of the different training methods on the computing performance, the storage performance and the communication performance, respectively, the weight distribution of the computing performance, the storage performance and the communication performance under each of the alternative training strategies is determined.

8. The method according to claim 7, wherein: The alternative training strategies are used to match the weight distribution of each target model chip respectively, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted in combination with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation, including: Determine the weight distribution of the candidate training strategies that match each of the target model chips; For each target model chip, the following weighted processing steps are performed: Determine the weights corresponding to the computing performance, storage performance and communication performance of the target model chip respectively by using the weight distribution of the target model chip; Using the weights corresponding to the computing performance, storage performance and communication performance of the target chip, weighted processing is performed on the corresponding computing performance evaluation, storage performance evaluation and communication performance evaluation to obtain weighted computing performance evaluation, storage performance evaluation and communication performance evaluation; Based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity ratio corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated; According to the sub-comprehensive training evaluations respectively corresponding to each of the target model chips, a comprehensive training evaluation corresponding to the alternative training strategy is obtained.

9. The method according to claim 8, wherein: Based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity proportion corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated, including: Quantifying the weighted evaluation parameters of the computing performance evaluation, storage performance evaluation, and communication performance evaluation corresponding to the target model chip to obtain a computing performance score, a storage performance score, and a communication performance score; Adding the computing performance score, the storage performance score and the communication performance score to obtain a performance score sum; Based on the performance score and the percentage of the number of chips corresponding to the target model, a sub-comprehensive training evaluation corresponding to the alternative training strategy is determined.

10. The method according to claim 9, wherein: Based on the performance score and the proportion of the number of chips corresponding to the target model, a sub-comprehensive training evaluation corresponding to the alternative training strategy is determined, including: The product of the performance score and the quantity ratio corresponding to the target model chip is used as the sub-comprehensive training evaluation corresponding to the alternative training strategy.

11. The method according to claim 8, wherein: According to the sub-comprehensive training evaluations corresponding to each of the target model chips, a comprehensive training evaluation corresponding to the candidate training strategy is obtained, including: Quantifying the sub-comprehensive training evaluations corresponding to the target model chips to obtain the sub-comprehensive training scores; Performing an addition operation on each of the sub-comprehensive training scores to obtain a sum of the sub-comprehensive training scores; A comprehensive training evaluation corresponding to the candidate training strategy is obtained based on the sub-comprehensive training scores and determined.

12. The method according to claim 11, wherein: Obtaining a comprehensive training evaluation corresponding to the candidate training strategy based on the sub-comprehensive training scores and determination includes: The sum of the sub-comprehensive training scores is used as a comprehensive training evaluation corresponding to the alternative training strategy.

13. The method according to claim 1, wherein: Before obtaining the corresponding comprehensive training evaluation, the method further includes: for each candidate training strategy, the weight distribution of each target model chip is matched with the candidate training strategy to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip, and obtaining the corresponding comprehensive training evaluation. Removing the candidate training strategies that do not meet the requirements of the actually received training tasks and / or removing the candidate training strategies that have obvious video memory conflict problems from all the candidate training strategies to obtain the remaining training strategies; Correspondingly, for each of the alternative training strategies, the alternative training strategies are used to match the weight distribution of each of the target model chips respectively, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted in combination with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation, including: For each of the remaining training strategies, the remaining training strategies are used to match the weight distribution of each of the target model chips, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted in combination with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

14. The method according to claim 1, wherein: For each of the candidate training strategies, the candidate training strategies are used to match the weight distribution of each of the target model chips respectively, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted in combination with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation, including: Control the central processing unit to create multiple parallel computing threads, and control each of the computing threads to use any alternative training strategy to be calculated to match the weight distribution of each target model chip, and perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

15. The method according to claim 1, wherein: Based on the evaluation parameters of each comprehensive training evaluation, a target training strategy is determined, including: The candidate training strategy with the highest evaluation parameter is determined as the target training strategy; wherein the highest evaluation parameter includes: the highest evaluation level, the highest evaluation score, and the highest evaluation degree.

16. The method according to claim 1, further comprising: Each target model chip is connected to the computing power cluster and is scheduled by the computing power cluster through a unified basic operation instruction pre-defined for each target model chip; wherein the unified basic operation instruction is a basic operation instruction supported by each target model chip.

17. The method according to any one of claims 1 to 16, further comprising: Control the computing power cluster to execute the training task of the generative large language model according to the target training strategy; wherein, different target model chips under the computing power cluster will be assigned to tasks to be processed that match their actual performance, and the tasks to be processed are decomposed from the training tasks.

18. The method according to claim 17, wherein: Controlling the computing power cluster to perform the training task of the generative large language model according to the target training strategy includes: Splitting the training task of the generative large language model into tasks to be processed corresponding to each target model chip according to the actual performance and quantity ratio of different target model chips; Sending each of the tasks to be processed to the corresponding target model chip; Control each of the target model chips to execute the received tasks to be processed according to the target training strategy.

19. The method according to claim 18, wherein: Controlling each of the target model chips to execute the received tasks to be processed according to the target training strategy, including: For the first inter-chip communication task in the tasks to be processed, controlling each of the target model chips to communicate with another target model chip in the same communication domain through the collective communication library; For the second inter-chip communication task in the tasks to be processed, each of the target model chips is controlled to communicate with another target model chip in a different communication domain in a forwarding manner through a central processing unit.

20. The method according to claim 19, wherein: Controlling each of the target model chips to communicate with another target model chip in a different communication domain by forwarding through a central processor, including: Each of the target model chips is controlled to communicate with another target model chip in a different communication domain by adopting a remote direct data access technology and / or a central processor forwarding method improved by copying pipeline technology.

21. The method according to claim 18, wherein: Controlling each of the target model chips to execute the received tasks to be processed according to the target training strategy, including: For the computing tasks in the tasks to be processed, control each of the target model chips to call the corresponding large model operator to perform corresponding computing operations according to the unified large model operator calling instruction corresponding to the computing task; wherein the unified large model operator calling instruction is a large model operator calling instruction pre-defined for each of the target model chips, which can be recognized and run by each of the target model chips, and the large model operator calling instruction is used to call the operator provided by the graphics processing chip to support the training of the generative large language model.

22. A training strategy determination device for a computing cluster that uses a mixture of chips of various models, comprising: A model and quantity ratio determination unit is used to determine the target model chips and the quantity ratio of each target model chip constituting the computing power cluster; The performance evaluation and weight distribution determination unit is configured to determine, for each target model chip, the performance evaluation of the target model chip under a preset performance category and the weight distribution of each preset performance category under each candidate training strategy; The comprehensive training evaluation determination unit is configured to, for each of the candidate training strategies, use the weight distribution of the candidate training strategies respectively matched to each of the target model chips to perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain a corresponding comprehensive training evaluation; The target training strategy determination unit is configured to determine the target training strategy based on the evaluation parameters of each of the comprehensive training evaluations.

23. The device according to claim 22, wherein: The model and quantity ratio determination unit is further configured to: According to the model information of the multiple image processing chips used by the computing power cluster, the target model chips are obtained; According to the actual number of each target model chip in the computing power cluster and the total number of chips, the quantity ratio of each target model chip is calculated.

24. The device according to claim 22, wherein: The performance evaluation and weight distribution determination unit includes a performance evaluation determination subunit configured to determine the performance evaluation of the target model chip under a preset performance category, and the performance evaluation determination subunit includes: A target performance prediction model determination module is configured to determine a target performance prediction model corresponding to the target model chip; wherein the target performance prediction model is used to characterize the corresponding relationship between the performance evaluations of different models of chips for various data sizes under the preset performance category; The model using subunit is configured to input the chip model of the target model chip into the target performance prediction model to obtain an output performance evaluation under the preset performance category.

25. The device according to claim 24, wherein: The target performance prediction model determination module is further configured to: Determine the target chip architecture adopted by the target model chip; A performance prediction model corresponding to the target chip architecture is determined as the target performance prediction model; wherein a matching performance prediction model is pre-trained for each chip architecture.

26. The apparatus according to claim 25, further comprising: A performance prediction model training unit, wherein the performance prediction model training unit is further configured to: Get known chip models under the same chip architecture; Obtaining actual performance evaluation of the image processing chip of the known chip model for various data sizes under the preset performance category; The known chip model is used as a sample input and the corresponding actual performance evaluation is used as an expected sample output to construct a training sample; The training samples are used to train the prediction model to be trained to obtain a trained performance prediction model.

27. The device according to claim 22, wherein: The performance evaluation and weight distribution determination unit includes a first weight distribution determination subunit configured to determine the weight distribution of each of the preset performance categories under each candidate training strategy, and the first weight distribution determination subunit includes: A different impact determination module is configured to determine different impacts caused by different training methods provided by different candidate training strategies on different types of performance of the target model chip; The weight distribution determination module is configured to determine the weight distribution of each preset performance type of the target model chip under each of the alternative training strategies according to the different impacts of the different training methods on the different types of performance of the target model chip; wherein the weight of any type of performance is proportional to the degree of impact.

28. The device according to claim 27, wherein The preset performance type includes: at least one of computing performance, storage performance and communication performance; Correspondingly, the performance evaluation and weight distribution determination unit includes a performance evaluation determination subunit configured to determine the performance evaluation of the target model chip under a preset performance category, and the performance evaluation determination subunit is further configured to: Determine the performance evaluation of the target model chip in terms of computing performance, storage performance and communication performance, respectively, and obtain computing performance evaluation, storage performance evaluation and communication performance evaluation accordingly; Correspondingly, the first weight distribution determination module is further configured to: According to the different effects of the different training methods on the computing performance, the storage performance and the communication performance, respectively, the weight distribution of the computing performance, the storage performance and the communication performance under each of the alternative training strategies is determined.

29. The device according to claim 28, wherein The comprehensive training evaluation determination unit comprises: A second weight distribution determination subunit is configured to determine the weight distribution of the candidate training strategies respectively matched to each target model chip; The weighted processing execution subunit is configured to perform the following weighted processing steps for each target model chip: using the weight distribution of the target model chip to determine the weights corresponding to the computing performance, storage performance and communication performance of the target model chip; using the weights corresponding to the computing performance, storage performance and communication performance of the target model chip, respectively weighted processing is performed on the corresponding computing performance evaluation, storage performance evaluation and communication performance evaluation to obtain the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation; based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity proportion corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated; The comprehensive evaluation determination subunit is configured to obtain a comprehensive training evaluation corresponding to the candidate training strategy based on the sub-comprehensive training evaluations corresponding to each target model chip.

30. The device according to claim 29, wherein: Based on the weighted computing performance evaluation, storage performance evaluation and communication performance evaluation corresponding to the target model chip and the quantity proportion corresponding to the target model chip, a sub-comprehensive training evaluation corresponding to the alternative training strategy is calculated, including: Quantifying the weighted evaluation parameters of the computing performance evaluation, storage performance evaluation, and communication performance evaluation corresponding to the target model chip to obtain a computing performance score, a storage performance score, and a communication performance score; Adding the computing performance score, the storage performance score and the communication performance score to obtain a performance score sum; Based on the performance score and the percentage of the number of chips corresponding to the target model, a sub-comprehensive training evaluation corresponding to the alternative training strategy is determined.

31. The device according to claim 30, wherein Based on the performance score and the proportion of the number of chips corresponding to the target model, a sub-comprehensive training evaluation corresponding to the alternative training strategy is determined, including: The product of the performance score and the quantity ratio corresponding to the target model chip is used as the sub-comprehensive training evaluation corresponding to the alternative training strategy.

32. The device according to claim 29, wherein: The comprehensive evaluation determination subunit includes: A quantization processing module is configured to quantize the sub-comprehensive training evaluations corresponding to the target model chips to obtain sub-comprehensive training scores; A summation processing module is configured to perform an addition operation on each of the sub-comprehensive training scores to obtain a sum of the sub-comprehensive training scores; The comprehensive training evaluation determination module is configured to obtain a comprehensive training evaluation corresponding to the candidate training strategy based on the sub-comprehensive training scores and determination.

33. The device according to claim 32, wherein: The comprehensive training evaluation determination module is further configured to: The sum of the sub-comprehensive training scores is used as a comprehensive training evaluation corresponding to the alternative training strategy.

34. The apparatus of claim 22, further comprising: A screening processing unit is configured to remove the candidate training strategies that do not meet the requirements of the actually received training tasks and / or remove the candidate training strategies that have obvious video memory conflicts from all the candidate training strategies to obtain the remaining training strategies; Correspondingly, the comprehensive training evaluation determination unit is further configured to: For each of the remaining training strategies, the remaining training strategies are used to match the weight distribution of each of the target model chips, and the performance evaluation of each corresponding target model chip under the preset performance category is weighted in combination with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

35. The apparatus of claim 22, wherein: The comprehensive training evaluation determination unit is further configured to: Control the central processing unit to create multiple parallel computing threads, and control each of the computing threads to use any alternative training strategy to be calculated to match the weight distribution of each target model chip, and perform weighted processing on the performance evaluation of each corresponding target model chip under the preset performance category combined with the quantity proportion of each corresponding target model chip to obtain the corresponding comprehensive training evaluation.

36. The apparatus of claim 22, wherein: The target training strategy determination unit is further configured to: The candidate training strategy with the highest evaluation parameter is determined as the target training strategy; wherein the highest evaluation parameter includes: the highest evaluation level, the highest evaluation score, and the highest evaluation degree.

37. The apparatus of claim 22, further comprising: The multi-model chip access and scheduling acceptance unit is configured to connect each target model chip to the computing power cluster and accept the scheduling of the computing power cluster through a unified basic operation instruction pre-defined for each target model chip; wherein the unified basic operation instruction is a basic operation instruction supported by each target model chip.

38. The apparatus according to any one of claims 22 to 37, further comprising: The training task control execution unit is configured to control the computing power cluster to execute the training task of the generative large language model according to the target training strategy; wherein, different target model chips under the computing power cluster will be assigned to tasks to be processed that match their actual performance, and the tasks to be processed are decomposed from the training tasks.

39. The device according to claim 38, wherein The training task control execution unit includes: The task splitting subunit is configured to split the training task of the generative large language model into tasks to be processed corresponding to each target model chip according to the actual performance and quantity proportion of different target model chips; The sending subunit is configured to send each of the tasks to be processed to a corresponding target model chip; The control execution subunit is configured to control each of the target model chips to execute the received tasks to be processed according to the target training strategy.

40. The device according to claim 39, wherein The control execution subunit comprises: A first inter-chip communication task execution control module is configured to control each of the target model chips to communicate with another target model chip in the same communication domain through a collective communication library for the first inter-chip communication task in the tasks to be processed; The second inter-chip communication task execution control module is configured to control each of the target model chips to communicate with another target model chip in a different communication domain by forwarding through the central processor for the second inter-chip communication task in the task to be processed.

41. The device according to claim 40, wherein The second inter-chip communication task execution control module is further configured to: Each of the target model chips is controlled to communicate with another target model chip in a different communication domain by adopting a remote direct data access technology and / or a central processor forwarding method improved by copying pipeline technology.

42. The apparatus of claim 39, wherein: The control execution subunit comprises: The computing task control execution module is configured to control each of the target model chips to call the corresponding large model operator to perform corresponding computing operations according to the unified large model operator calling instruction corresponding to the computing task for the computing task to be processed; wherein the unified large model operator calling instruction is a large model operator calling instruction pre-defined for each of the target model chips and can be recognized and run by each of the target model chips, and the large model operator calling instruction is used to call the operator provided by the graphics processing chip to support the training of the generative large language model.

43. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training strategy determination method for a computing power cluster that uses a mixture of multiple chip models as described in any one of claims 1-21.

44. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method for determining a training strategy for a computing power cluster that uses a mixture of chips of multiple models as described in any one of claims 1-21.

45. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of a method for determining a training strategy for a computing power cluster that uses a mixture of chips of multiple models according to any one of claims 1-21.

Images