Power consumption prediction method in large model parallel training scene and related equipment

Abstract

The embodiment of the application provides a power consumption prediction method in a large model parallel training scene and related equipment, and belongs to the technical field of computer data processing. The method comprises the following steps: obtaining a configuration file, a total parameter set and a total training step number of a target large model; predicting the calculation amount information of the target large model in each sub-training step number according to a model type, and predicting the communication amount information of the target large model in each sub-training step number according to at least one parallel strategy; determining a calculation power consumption value based on the predicted calculation amount information and a calculation energy efficiency ratio, and determining a communication power consumption value based on the predicted communication amount information and a communication energy efficiency ratio; superimposing the calculation power consumption value and the communication power consumption value to obtain an every-step power consumption value corresponding to the current sub-training step number, multiplying the every-step power consumption value by the total training step number to obtain a total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set. The application can improve the power consumption prediction precision of the large model in the parallel training scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data processing, and in particular to a power consumption prediction method in a large model parallel training scene and related equipment. BACKGROUND

[0002] With the continuous development of technology, the computing power demand of AI large model training and inference is growing explosively, which directly pushes up the running power consumption of intelligent computing clusters. The generation of AI large models also involves complex training processes, and therefore the energy consumption of training a round of AI large models, especially top-level AI large models, is rising exponentially, with unit area power consumption far exceeding that of traditional Internet data centers. Such huge energy consumption (electricity load) poses a challenge to power grid dispatching. In this context, predicting the power consumption of large models in a parallel training scene helps optimize resource allocation.

[0003] In related technologies, empirical fitting or black box models such as machine learning based on historical data are used, or the modeling ideas of traditional data center general loads are followed for power consumption prediction. However, when these methods are applied to the parallel training scene of AI large models, the power consumption of the entire training task is often estimated as a whole, without effectively distinguishing and quantifying the model calculation itself and other power consumption contributions, resulting in relatively rough prediction results, which ultimately reduces the power consumption prediction accuracy of large models in the parallel training scene. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a power consumption prediction method in a large model parallel training scene and related equipment, aiming to improve the power consumption prediction accuracy of large models in the parallel training scene.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a power consumption prediction method in a large model parallel training scene, the method comprising:

[0006] obtaining a configuration file, a total parameter set and a total training step number of a target large model, wherein the total training step number comprises a plurality of same sub-training step numbers;

[0007] determining a model type to which the target large model belongs based on the configuration file, and predicting calculation amount information of the target large model in each sub-training step number according to the model type;

[0008] determining at least one parallel strategy of the target large model based on the configuration file, and predicting communication amount information of the target large model in each sub-training step number according to the at least one parallel strategy;

[0009] The computing energy efficiency ratio and the communication energy efficiency ratio of the target large model in each sub-training step are predicted, the computing power consumption value is determined based on the computing amount information and the computing energy efficiency ratio, and the communication power consumption value is determined based on the communication amount information and the communication energy efficiency ratio;

[0010] The computing power consumption value and the communication power consumption value are superimposed to obtain an energy consumption value of each step corresponding to the current sub-training step, and the energy consumption value of each step is multiplied by the total training step to obtain the total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set.

[0011] In some embodiments, the computing amount information of the target large model in each sub-training step is predicted according to the model type, including:

[0012] The total data set to be processed by the target large model at each sub-training step is obtained, and a preset floating point value is obtained; if the model type represents that the target large model is a dense large model, the preset floating point value, the total parameter set and the total data set are multiplied to obtain first computing amount information, and the first computing amount information is taken as the computing amount information of the target large model in each sub-training step;

[0013] If the model type represents that the target large model is a hybrid expert type large model, the model layer number and the expert layer number of the target large model are determined, and the difference between the model layer number and the expert layer number is calculated to obtain a layer number difference;

[0014] A first ratio is calculated between the expert layer number and the model layer number, and a second ratio is calculated between the layer number difference and the model layer number;

[0015] The number of activated target experts is determined from a plurality of initial experts preset by the target large model, and a ratio of the number of initial experts and the number of target experts is calculated to obtain an expert number ratio;

[0016] The expert number ratio and the first ratio are multiplied to obtain a third ratio, and the superimposed result of the second ratio and the third ratio is multiplied by the first computing amount information to obtain second computing amount information, and the second computing amount information is taken as the computing amount information of the target large model in each sub-training step.

[0017] In some embodiments, the target large model includes a plurality of operators;

[0018] The computing energy efficiency ratio and the communication energy efficiency ratio of the target large model in each sub-training step are predicted, including:

[0019] The total communication amount set of the target large model and the unit energy consumption value of the target large model per unit time are obtained;

[0020] determining a unit operation number, an execution time overhead and an execution utilization of each operator, multiplying the unit operation number, the execution time overhead and the execution utilization for each operator to obtain a current operator computation amount;

[0021] adding all current operator computation amounts to obtain a total operator computation amount, and determining a computing energy efficiency ratio of the target large model in each sub-training step based on a ratio of the total operator computation amount and the unit energy consumption value;

[0022] determining a communication energy efficiency ratio of the target large model in each sub-training step based on a ratio of the total communication amount set and the unit energy consumption value.

[0023] In some embodiments, the communication amount information includes tensor parallel communication amount information, data parallel communication amount information, pipeline parallel communication amount information and expert parallel communication amount information, and the communication energy efficiency ratio includes tensor parallel communication energy efficiency ratio, data parallel communication energy efficiency ratio, pipeline parallel communication energy efficiency ratio and expert parallel communication energy efficiency ratio;

[0024] determining the communication power consumption value based on the communication amount information and the communication energy efficiency ratio, comprising:

[0025] if the parallel strategy includes tensor parallel, data parallel, pipeline parallel and expert parallel, for the tensor parallel, predicting tensor parallel communication amount information and tensor parallel communication energy efficiency ratio in each sub-training step, and determining tensor parallel communication power consumption based on a ratio of the tensor parallel communication amount information and the tensor parallel communication energy efficiency ratio;

[0026] for the data parallel, predicting data parallel communication amount information and data parallel communication energy efficiency ratio in each sub-training step, and determining data parallel communication power consumption based on a ratio of the data parallel communication amount information and the data parallel communication energy efficiency ratio;

[0027] for the pipeline parallel, predicting pipeline parallel communication amount information and pipeline parallel communication energy efficiency ratio in each sub-training step, and determining pipeline parallel communication power consumption based on a ratio of the pipeline parallel communication amount information and the pipeline parallel communication energy efficiency ratio;

[0028] for the expert parallel, predicting expert parallel communication amount information and expert parallel communication energy efficiency ratio in each sub-training step, and determining expert parallel communication power consumption based on a ratio of the expert parallel communication amount information and the expert parallel communication energy efficiency ratio;

[0029] adding the tensor parallel communication power consumption, the data parallel communication power consumption, the pipeline parallel communication power consumption and the expert parallel communication power consumption to obtain the communication power consumption value.

[0030] In some embodiments, each sub-training step includes a plurality of micro-batch times;

[0031] predicting tensor parallel communication quantity information under each sub-training step, including:

[0032] determining a tensor parallel scale of the target large model, wherein the target large model comprises a first quantity of first acceleration devices;

[0033] determining a first data processing amount of each first acceleration device, determining a tensor parallel overhead of each first acceleration device based on the tensor parallel scale, and multiplying the first data processing amount and the tensor parallel overhead to obtain a first data propagation amount;

[0034] predicting tensor parallel communication quantity information under each sub-training step based on the micro-batch number, the first data propagation amount, and the first quantity.

[0035] In some embodiments, predicting data parallel communication quantity information under each sub-training step, includes:

[0036] determining a data parallel scale of the target large model, wherein the target large model comprises a second quantity of second acceleration devices;

[0037] determining a second data processing amount of each second acceleration device based on the total parameter set, determining a data parallel overhead of each second acceleration device based on the data parallel scale, and multiplying the second data processing amount and the data parallel overhead to obtain a second data propagation amount;

[0038] predicting data parallel communication quantity information under each sub-training step based on the second data propagation amount and the second quantity.

[0039] In some embodiments, predicting pipeline parallel communication quantity information under each sub-training step, includes:

[0040] determining a pipeline parallel scale of the target large model, wherein the target large model comprises a third quantity of communication groups;

[0041] determining a third data processing amount of each communication group, determining a pipeline parallel overhead of each communication group based on the pipeline parallel scale, and determining a third data propagation amount based on the third data processing amount and the pipeline parallel overhead;

[0042] predicting pipeline parallel communication quantity information under each sub-training step based on the micro-batch number, the third data propagation amount, and the third quantity.

[0043] In some embodiments, the expert parallel includes a regular expert parallel and a grouped expert parallel;

[0044] predicting expert parallel communication quantity information under each sub-training step, includes:

[0045] If the target large model is a hybrid expert type large model and the parallel strategy is a regular expert parallel, a fourth ratio is calculated by calculating a ratio between the preset unit value and the number of initial experts, and a second difference is calculated by calculating a difference between the preset unit value and the fourth ratio, wherein the target large model includes a target expert quantity of target experts;

[0046] A desired expert quantity is obtained by multiplying the target expert quantity and the second difference, and a fourth data processing amount of each target expert is determined, and a fourth data propagation amount is determined based on the fourth data processing amount and the desired expert quantity;

[0047] Based on the micro-batch number, the fourth data propagation amount, and the expert layer number, expert parallel communication amount information under each sub-training step is predicted;

[0048] If the target large model is a hybrid expert type large model and the parallel strategy is a grouped expert parallel, a grouped expert quantity is determined;

[0049] Based on the micro-batch number, the fourth data propagation amount, the expert layer number, and the grouped expert quantity, expert parallel communication amount information under each sub-training step is predicted.

[0050] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a power consumption prediction device in a large model parallel training scene, the device comprising:

[0051] An acquisition module is configured to acquire a configuration file, a total parameter set, and a total training step number of a target large model, wherein the total training step number includes a plurality of same sub-training step numbers;

[0052] A calculation amount information determination module is configured to determine a model type to which the target large model belongs based on the configuration file, and to predict calculation amount information of the target large model in each sub-training step according to the model type;

[0053] A communication amount information determination module is configured to determine at least one parallel strategy of the target large model based on the configuration file, and to predict communication amount information of the target large model in each sub-training step according to the at least one parallel strategy;

[0054] A local power consumption value determination module is configured to predict a calculation energy efficiency ratio and a communication energy efficiency ratio of the target large model in each sub-training step, to determine a calculation power consumption value based on the calculation amount information and the calculation energy efficiency ratio, and to determine a communication power consumption value based on the communication amount information and the communication energy efficiency ratio;

[0055] A total power consumption value determination module is configured to obtain an every-step power consumption value corresponding to the current sub-training step by superimposing the calculation power consumption value and the communication power consumption value, and to multiply the every-step power consumption value and the total training step number to predict a total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set.

[0056] To achieve the above object, the third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the power consumption prediction method in the large model parallel training scenario of the first aspect when executing the computer program.

[0057] To achieve the above object, the fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the power consumption prediction method in the large model parallel training scenario of the first aspect.

[0058] The power consumption prediction method in the large model parallel training scenario and the related device provided by the present application, by obtaining the configuration file, the total parameter set and the total training step number of the target large model, wherein the total training step number comprises a plurality of same sub-training step numbers; determining the model type to which the target large model belongs based on the configuration file, and predicting the calculation amount information of the target large model in each sub-training step number according to the model type; determining at least one parallel strategy of the target large model based on the configuration file, and predicting the communication amount information of the target large model in each sub-training step number according to the at least one parallel strategy; predicting the calculation energy efficiency ratio and the communication energy efficiency ratio of the target large model in each sub-training step number, determining the calculation power consumption value based on the calculation amount information and the calculation energy efficiency ratio, and determining the communication power consumption value based on the communication amount information and the communication energy efficiency ratio; superimposing the calculation power consumption value and the communication power consumption value to obtain the per-step power consumption value corresponding to the current sub-training step number, and multiplying the per-step power consumption value by the total training step number to predict the total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set. The present application decomposes the large model power consumption prediction problem in the parallel training scenario into independent calculation power consumption prediction and communication power consumption prediction, makes up for the major defect that the traditional method only focuses on power consumption calculation and completely ignores communication power consumption, and fundamentally corrects the power consumption prediction deviation in the traditional method by calculating the communication amount under different parallel strategies in detail, so that the power consumption prediction accuracy of the large model in the parallel training scenario is essentially improved. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a schematic diagram of an optional implementation environment of the power consumption prediction device for large model parallel training scenario provided by the embodiment of the present application;

[0060] Figure 2 is an optional flowchart of the power consumption prediction method for large model parallel training scenario provided by the embodiment of the present application;

[0061] Figure 3 is an optional implementation flowchart of step 102 in Figure 2

[0062] Figure 4 ​is Figure 2 is an optional implementation flowchart of step 104 in

[0063] Figure 5 is Figure 2 is another optional implementation flowchart of step 104 in

[0064] Figure 6 is Figure 5 is an optional implementation flowchart of step 104.2.1 in

[0065] Figure 7 is Figure 5 is an optional implementation flowchart of step 104.2.2 in

[0066] Figure 8 is Figure 5 is an optional implementation flowchart of step 104.2.3 in

[0067] Figure 9 is Figure 5 is an optional implementation flowchart of step 104.2.4 in

[0068] Figure 10 is an optional device module schematic diagram of the power consumption prediction device provided by the embodiment of the application in the large model parallel training scene;

[0069] Figure 11 is a hardware structure schematic diagram of the electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0071] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0073] First, the terms involved in the present application are analyzed:

[0074] Artificial Intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; Artificial intelligence is a branch of computer science, artificial intelligence tries to understand the essence of intelligence, and produce a new intelligent machine that can react in a similar way to human intelligence. The field of research includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0075] AI large model refers to a super large artificial intelligence model trained by massive data and with hundreds of billions or even trillions of parameters. It understands and generates multi-modal information such as text, image and audio based on deep learning technology. This kind of model has strong versatility and can flexibly handle multiple tasks such as question answering, translation, writing, programming and reasoning without the need for separate training for each task. Its core principle is to learn the patterns in the data through statistical rules and predict the most likely output when receiving input, thus showing human-like intelligence.

[0076] With the continuous development of technology, the computing power demand of AI large model training and inference is growing explosively, which directly pushes up the operation power consumption of intelligent computing clusters. The generation of AI large model also involves a complex training process, so the energy consumption of training a round of AI large model, especially the top AI large model, is increasing exponentially, and its unit area power consumption far exceeds that of traditional Internet data centers. Such huge energy consumption (power load) poses a challenge to power grid dispatching. In this context, it is helpful to optimize resource allocation to predict the power consumption of large models in parallel training scenarios.

[0077] In related technologies, empirical fitting or black box models such as machine learning based on historical data are used, or the modeling ideas of traditional data center general loads are followed to predict power consumption. However, when these methods are applied to the parallel training scenario of AI large models, the power consumption of the entire training task is often estimated as a whole, without effectively distinguishing and quantifying the model calculation itself and other power consumption contributions, resulting in rough prediction results and ultimately reducing the power consumption prediction accuracy of large models in parallel training scenarios.

[0078] Based on this, the embodiment of the present application provides a power consumption prediction method in a large model parallel training scene and related equipment. The power consumption prediction problem of a large model in a parallel training scene is decomposed into independent calculation power consumption prediction and communication power consumption prediction, so as to overcome the defect that the power consumption is regarded as a whole for rough estimation based on a black box model in the traditional method. Specifically, the embodiment of the present application predicts the calculation amount of the target large model in each sub-training step based on the model type to which the target large model belongs; then determines the parallel strategy of the large model based on the configuration file, so as to predict the communication amount of the target large model in each sub-training step according to at least one parallel strategy; finally, the power consumption contribution value of the target large model in the aspects of “calculation” and “power consumption” is determined and superimposed respectively combining the calculation energy efficiency ratio and the communication energy efficiency ratio, and then the total power consumption value of the target large model under the current parallel training condition is obtained. It can be understood that the embodiment of the present application considers that the target large model needs to exchange a large amount of data (such as gradient synchronization and activation value transmission) at a high frequency in the data parallel, tensor parallel and other parallel strategies in large-scale parallel training, in other words, the power consumption generated in the communication link process is an important component of the total energy consumption of the target large model training process, and then the analytical and white-box modeling framework proposed by the embodiment of the present application makes up for the major defect that the traditional method only focuses on power consumption calculation and completely ignores communication power consumption, and fundamentally corrects the power consumption prediction deviation in the traditional method by calculating the communication amount under different parallel strategies, and realizes the essential improvement of the power consumption prediction accuracy.

[0079] Exemplarily, as shown in Figure 1 , Figure 1is a schematic diagram of an optional implementation environment of the power consumption prediction device in a large model parallel training scenario provided by the embodiment of the present application, and the implementation environment includes a client 11 and a server 12, wherein the power consumption prediction device in a large model parallel training scenario (for the convenience of description, it can also be referred to as “prediction device” for short) is deployed on the server 12. The server 12 will receive the configuration file, the total parameter set and the total training step number sent from the client 11, and the total training step number includes a plurality of same sub-training step numbers; then, the prediction device determines the model type to which the target large model belongs based on the configuration file, and predicts the calculation amount information of the target large model in each sub-training step number according to the model type; then, at least one parallel strategy of the target large model is determined based on the configuration file, and the communication amount information of the target large model in each sub-training step number is predicted according to the at least one parallel strategy; then, the calculation energy efficiency ratio and the communication energy efficiency ratio are predicted based on the total parameter set, the calculation power consumption value is determined based on the calculation amount information and the calculation energy efficiency ratio, and the communication power consumption value is determined based on the communication amount information and the communication energy efficiency ratio; finally, the per-step power consumption value corresponding to the current sub-training step number is obtained by superimposing the calculation power consumption value and the communication power consumption value, and the per-step power consumption value is multiplied by the total training step number to predict the total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set.

[0080] Wherein, the server 12 can send the trained target defect detection model to the client 11, and the client 11 can perform subsequent processing on the obtained target detection image; or the server 12 can directly use the trained target defect detection model to perform subsequent data processing, and the embodiment of the present application does not limit this.

[0081] Wherein, the server 12 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. In addition, the server 12 can also be a node server in a blockchain network. The client 11 can be a mobile phone, a computer, a smart voice interaction device, a smart wearable device, a smart home appliance, a vehicle terminal, etc., but is not limited thereto. The client 11 and the server 12 can be directly or indirectly connected through wired or wireless communication, and the embodiment of the present application does not limit this.

[0082] It should be noted that in the embodiments of the present application, when it is necessary to obtain user's basic information or information related to user characteristics such as user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained first, and after obtaining the separate permission or separate consent of the user, the necessary data for enabling the embodiments of the present application to operate normally will be obtained. For example, before obtaining the configuration file, total parameter set and total training step number of the target large model, the embodiments of the present application will first obtain the authorization or consent of the relevant personnel managing the target large model. In addition, the other related data obtained by the prediction device proposed in the present application are all authorized data, which will not be described one by one here.

[0083] In the embodiments of the present application, the dimensions of the power consumption prediction device in the large model parallel training scene will be described, as shown in Figure 2 Figure 2 is an optional flowchart of the power consumption prediction method in the large model parallel training scene provided by the embodiments of the present application, Figure 2 The method in the above can include but is not limited to the following steps 101 to 105. When the power consumption prediction device in the large model parallel training scene executes the power consumption prediction method in the large model parallel training scene, the specific process is as follows. It should be first pointed out that the order of steps 101 to 105 in the above is not specifically limited, and the order of steps can be adjusted or some steps can be reduced, increased according to actual needs. Figure 2

[0084] Step 101, obtaining the configuration file, total parameter set and total training step number of the target large model, wherein the total training step number includes a plurality of same sub-training step numbers.

[0085] The step 101 will be described in detail below.

[0086] ​​The target large model refers to a large artificial intelligence model that is focused on and operated in a specific research, development or deployment task (i.e., the AI large model mentioned earlier). It usually has a large number of parameters (such as tens of billions to hundreds of billions) and has strong language understanding and generation capabilities. The target large model has a large number of parameters because its strong intelligence performance relies on a high-capacity model structure to capture complex language patterns, world knowledge and reasoning capabilities. Therefore, the more parameters the target large model has, the stronger its expression and memory capacity, and it can better fit large-scale training data to complete various tasks such as question answering, creation, translation, etc. However, such a large parameter size results in a computing load and memory requirement far exceeding the processing capacity of a single graphics processing unit (GPU) or neural processing unit (NPU). Therefore, during training, the large model usually adopts parallel training techniques to split the large model into a cluster composed of tens or even tens of thousands of computing cores, and collaboratively compute through data parallelism, model parallelism or pipeline parallelism to complete training within a reasonable time.

[0087] The configuration file is a text file used to store model structure or training environment parameters related to the target large model. The configuration file is usually stored in the code repository of the large model project, or saved in the local or cloud working directory corresponding to the training task, or integrated in the model publishing platform. Users can obtain the configuration file from sources including but not limited to open source project code, internal configuration management system or training task scheduling platform, etc., as long as their permissions allow.

[0088] The total parameter set refers to the set of all learnable parameters of the large model during training, including but not limited to the number of network layers, the dimension of hidden layers, the number of attention heads, the type of optimizer, the learning rate, the batch size, and the hardware resource configuration of the large model. The total parameter set is initialized at the beginning of training and updated according to the gradient backpropagation algorithm at each training step. These parameters are the core carriers for the model to "learn" data features. In the parallel training scenario, the total parameter set is usually split or duplicated to multiple computing devices for collaborative updating.

[0089] The total training step number refers to the total number of iterations required to complete a full large model training. The total training step includes multiple consecutive and similar time or computing load sub-training step numbers, and the total training step number is equal to the sum of all sub-training step numbers. In actual model training, each sub-training step usually involves the same calculation process repeated, such as each sub-training step corresponding to a set of forward propagation, loss calculation, backpropagation and parameter update processes.

[0090] Further, in large model training, the configuration file, the total parameter set, and the total training step number are three core elements guiding and defining the entire training task: the configuration file contains the model architecture (such as the number of layers, the hidden dimension), the optimizer type, the learning rate, the batch size, the parallel strategy (such as the scale of tensor parallel and data parallel), and other hyperparameters and runtime settings, which are the "blueprint" of the training process; the total parameter set represents all the learnable weights and biases of the model, which are the objects of optimization through backpropagation and gradient update in the training process, and the scale thereof directly determines the capacity and computational complexity of the model; the total training step number plans the length and iteration number of the entire training process, which is divided into multiple identical sub-training step numbers, each of which corresponds to a complete cycle of forward, backward, and parameter update, which collectively drives the model to gradually converge from the initial state to the final state with strong capabilities.

[0091] Step 102, determining the model type to which the target large model belongs based on the configuration file, and predicting the computational amount information of the target large model in each sub-training step number according to the model type.

[0092] The following describes step 102 in detail.

[0093] In some embodiments, after obtaining the configuration file, the prediction device will parse the model type to which the target large model belongs according to the configuration file. The model type is used to represent the category or structural feature of the large model in the architecture design, which defines the basic composition method and computational logic of the model, for example, it can be used to determine whether the target large model is a dense large model in which all parameters participate in the calculation, or a hybrid expert (Mixture of Experts, MoE) large model in which only part of the parameters are activated for calculation. Since there are essential differences in the calculation mode of the target large model under different model types, the embodiments of the present application will call different computational amount information determination methods according to different model types to accurately predict the computational amount information required by the target large model to complete the operation in a single sub-training step. The computational amount information is usually measured in floating point operation (FLOPs), which isolates the influence of hardware and parallel strategy, and purely reflects the computational complexity of the model algorithm.

[0094] The following will introduce how to determine the computational amount information of the target large model:

[0095] In some embodiments, as shown in Figure 3 , Figure 3 is Figure 2 an optional implementation flowchart of step 102 in

[0096] 102.1.1 obtain a total data set to be processed by the target large model at each sub-training step, and a preset floating point value;

[0097] 102.1.2 if the model type represents the target large model as a dense large model, multiply the preset floating point value, the total parameter set and the total data set to obtain first calculation information, and take the first calculation information as the calculation information of the target large model at each sub-training step;

[0098] 102.1.3 if the model type represents the target large model as a hybrid expert large model, determine the model layer number and the expert layer number of the target large model, and calculate the difference between the model layer number and the expert layer number to obtain a layer number difference;

[0099] 102.1.4 calculate the ratio between the expert layer number and the model layer number to obtain a first ratio, and calculate the ratio between the layer number difference and the model layer number to obtain a second ratio;

[0100] 102.1.5 determine the number of activated target experts from a plurality of initial experts preset by the target large model, and calculate the ratio between the number of initial experts and the number of target experts to obtain an expert number ratio;

[0101] 102.1.6 multiply the expert number ratio and the first ratio to obtain a third ratio, multiply the superimposed result of the second ratio and the third ratio with the first calculation information to obtain second calculation information, and take the second calculation information as the calculation information of the target large model at each sub-training step.

[0102] The following describes steps 102.1.1 to 102.1.6 in detail.

[0103] In some embodiments, first, the prediction device obtains a total data set to be processed by the target large model at each sub-training step, which reflects the data size to be processed by the large model in a single iteration. At the same time, the prediction device also obtains a preset floating point value, which is an empirical constant for estimating the number of floating point operations. This value is usually determined according to the calculation complexity of forward and backward propagation. Generally, the total parameter set of the target large model and the input data (token) require an average of 8 floating point operations (FLOP) in the calculation process, including 2 FLOP for forward propagation, 2 FLOP for re-computation, and 4 FLOP for backward propagation. Therefore, the preset floating point value is usually the numerical value 8.

[0104] Further, if the model type determined in the preceding step indicates that the target large model is a dense large model, the preset floating point value, the total parameter set, and the total data set are multiplied together, and the multiplication result is taken as the first calculation amount information. For a dense model, the first calculation amount information is directly taken as the final calculation amount information of the target large model at each sub-training step, and it also provides a benchmark value for subsequent calculation of the hybrid expert model.

[0105] For example, as shown in the following formula <1>, a certain dense large model such as a large language model meta AI (LLaMA series model) needs to process tokens for each step of training, and for the total parameter set N of the dense large model, the required calculation amount information (that is, the first calculation amount information) is:

[0106] <1>

[0107] wherein, represents the total parameter set; represents the total data set; represents the preset floating point value.

[0108] In some embodiments, if the model type determined in the preceding step indicates that the target large model is a hybrid expert large model, a refined calculation process for the sparse activation characteristics thereof is started. It should be noted that one of the characteristics of the hybrid expert large model is that not all parameters thereof are in an activated state. Therefore, the model layer number (that is, the total depth of the large model) and the expert layer number (that is, the number of layers of the large model using the MoE structure) of the large model need to be determined first; then, the difference between the model layer number and the expert layer number is calculated to obtain a layer number difference, which represents the number of layers of the large model using the traditional dense calculation mode.

[0109] Further, in order to quantify the proportion of different calculation modes in the model, the ratio between the expert layer number and the model layer number is then calculated to obtain a first ratio, which represents the proportion of the MoE structure in the entire model depth; then, the ratio between the layer number difference calculated in the previous step and the model layer number is calculated to obtain a second ratio, which represents the proportion of the traditional dense layer in the entire model depth. These two ratios decompose the model structure into sparse and dense parts in terms of calculation, laying a foundation for subsequent weighted calculation.

[0110] Further, the number of activated target experts is determined from a plurality of initial experts preset by the target large model; then, the expert number ratio is obtained by calculating the ratio between the total number of initial experts and the number of activated target experts, which is a key sparse factor directly reflecting that in the MoE layer, the actual participating parameters only account for a small part of the total expert parameters (initial experts).

[0111] Further, the expert number ratio is multiplied by the first ratio to obtain a third ratio, which represents the equivalent computing load proportion of the MoE layer after sparse activation reduction; then, the second ratio representing the dense layer computing load proportion is superimposed with the third ratio, thereby forming a comprehensive computing load weight factor. Finally, the computing load weight factor is multiplied by the benchmark first computing amount information calculated in step 102.1.1 to obtain the second computing amount information. The second computing amount information accurately reflects the final computing amount information of the hybrid expert type large model with sparse activation characteristics at each sub-training step.

[0112] For example, for MoE models such as DeepSeek series models, considering the model in the process of training, All e initial experts of the MoE layer only activate k target experts, so the computing amount of the MoE layer needs to be reduced by / The number of e initial experts and the number of k target experts are determined, and the computing amount information required by the hybrid expert type large model is determined by the following formula <2>: That is, the second computing amount information) is:

[0113] <2>

[0114] Wherein, represents the total parameter set; represents the total data set; represents a preset floating point value; represents the number of model layers; represents the number of expert layers; represents the layer difference value; represents the first ratio; represents the second ratio; represents the number of initial experts; represents the number of activated target experts; represents the expert number ratio; represents the third ratio; represents the superimposed result of the second ratio and the third ratio; represents the first computing amount information.

[0115] It can be understood that the embodiments of the present application no longer use a single model calculation amount estimation method for all types of large models as in the traditional method, but by distinguishing between dense large models and mixed expert large models, the model is targeted and refined. In particular, for mixed expert large models with complex structure, the method decomposes the proportion of dense and expert layers layer by layer to accurately quantify the calculation amount reduction caused by sparse activation, and the final calculation amount information is much more accurate than estimating it as a dense model. This fine modeling based on the internal structural differences of the model provides highly accurate input for subsequent determination of computing power values, ensuring the accuracy and reliability of the computing power prediction from the source.

[0116] Next, the following continues to introduce how to determine the communication amount information of the target large model:

[0117] Step 103, determining at least one parallel strategy of the target large model based on the configuration file, and predicting the communication amount information of the target large model in each sub-training step according to the at least one parallel strategy.

[0118] The following describes step 103 in detail.

[0119] In some implementations, based on the configuration file, at least one parallel strategy used in the current model training task is identified, such as Tensor Parallelism (TP), Data Parallelism (DP), Pipeline Parallelism (PP), Expert Parallelism (EP), etc. Each parallel strategy corresponds to a unique inter-node or inter-chip data exchange mode and communication data amount. Therefore, the embodiments of the present application will apply a specific communication amount calculation method to predict the communication amount information generated in a single sub-training step according to the specific parallel strategy combination determined. The communication amount information is usually quantified in bytes, which quantifies the total amount of data exchanged to maintain the cooperative work of multiple nodes, such as gradient synchronization, activation value transmission, or model parameter distribution, etc.

[0120] Step 104, predicting the computing energy efficiency ratio and the communication energy efficiency ratio of the target large model in each sub-training step, determining the computing power value based on the calculation amount information and the computing energy efficiency ratio, and determining the communication power value based on the communication amount information and the communication energy efficiency ratio.

[0121] The following describes step 104 in detail.

[0122] In some embodiments, after determining the computation amount information and the communication amount information of the target large model in terms of "computation" and "communication" respectively, the computation energy efficiency ratio of the target large model in each sub-training step in terms of "computation" and the communication energy efficiency ratio of the target large model in each sub-training step in terms of "communication" are determined respectively. The computation energy efficiency ratio represents the energy consumed by the target large model to perform a unit computation amount (e.g., 1 GFLOPs per second), and the communication energy efficiency ratio represents the energy consumed by the target large model to transmit a unit communication amount (e.g., 1 GB). Subsequently, the computation amount information obtained in step 102 and the communication amount information obtained in step 103 are converted into actual computation power consumption values and communication power consumption values respectively by using the computation energy efficiency ratio and the communication energy efficiency ratio as a bridge.

[0123] In order to enable the reader to better understand the embodiments of the present application, for step 104, it is divided into "energy efficiency ratio calculation" and "power consumption prediction" two aspects, and first introduce the "energy efficiency ratio calculation" aspect:

[0124] In some embodiments, as shown in Figure 4 Figure 4 is Figure 2 an optional implementation flowchart of step 104 in

[0125] 104.1.1 obtaining a total communication amount set of the target large model and a unit energy consumption value of the target large model within a unit time;

[0126] 104.1.2 determining a unit operation frequency, an execution time overhead and an execution utilization rate of each operator, and multiplying the unit operation frequency, the execution time overhead and the execution utilization rate for each operator to obtain a current operator computation amount;

[0127] 104.1.3 superimposing all current operator computation amounts to obtain a total operator computation amount, and determining a computation energy efficiency ratio of the target large model in each sub-training step based on a ratio of the total operator computation amount to the unit energy consumption value;

[0128] 104.1.4 determining a communication energy efficiency ratio of the target large model in each sub-training step based on a ratio of the total communication amount set to the unit energy consumption value.

[0129] The steps 104.1.1 to 104.1.4 are described in detail below.

[0130] ​In some embodiments, the target large model includes a plurality of operators. An operator is the most basic functional unit of a model for data processing and calculation. Common operators include mathematical operations (such as addition, subtraction, multiplication, division, matrix multiplication), activation functions (such as ReLU, Sigmoid), normalization (such as batch normalization, layer normalization), convolution (Conv2D), pooling (MaxPool), attention mechanism (Attention), etc. Generally, the target large model covers a variety of operators, each of which defines specific input, output, and specific calculation logic. The calculation process of the target large model is composed and stacked by these basic operators.

[0131] In traditional computation amount prediction methods, a single matrix multiplication operator such as general matrix multiplication (GeMM) is often used to measure the peak computation amount. This is because the GeMM operator has a very high computation density and a regular memory access pattern. In order to simplify the calculation, the GeMM operator is often used as a representative operator of the model in the traditional method to determine the computation amount information. However, such a calculation method ignores the various operators involved in the actual training of the model. These operators are often limited by memory bandwidth, control flow overhead, and low arithmetic intensity when performing actual data processing, resulting in a significant decrease in the utilization rate of the computing unit, and thus the actual average effective computing power is much lower than the peak value measured by running only the idealized operator.

[0132] Therefore, in order to more accurately predict the computation energy efficiency ratio and the communication energy efficiency ratio, the embodiments of the present application first need to obtain the total communication amount set of the target large model and its unit energy consumption value in unit time in a series of benchmark tests or profiling runs. The total communication amount set refers to the total amount of data actually transmitted between nodes or between chips under a specific test scenario, recorded by hardware monitoring or software profiling tools. The unit energy consumption value is the unit energy consumption measured by a power meter or an acceleration device management worker under the same test scenario.

[0133] Further, the embodiments of the present application perform fine-grained analysis on the actual execution efficiency of the computing task to determine the effective computing load (effective computing power). For each operator, the theoretical number of unit operations (i.e., FLOPs), the execution time overhead on the hardware, and the execution utilization of the runtime computing unit are determined. The number of unit operations refers to the total number of basic floating-point computation operations involved in the execution of an operator, including addition, subtraction, multiplication, and division floating-point operations. The number of unit operations can evaluate the overall computing amount and hardware computing power demand of the model. The execution time overhead refers to the actual time consumed by an operator from the start of execution to the completion and output of the result on a specific hardware (such as GPU, CPU, or AI accelerator), usually measured in milliseconds (ms) or microseconds (μs). The execution time overhead can measure the operator running efficiency and the overall inference / training speed of the model. The execution utilization measures the proportion of time or computing power occupancy during which the hardware computing resources (such as GPU cores, computing units of AI chips) are effectively used to execute computing instructions during the execution of the corresponding task by an operator. For example, if the peak computing power of a GPU is 10 trillion floating-point operations per second (TFLOPs) and the actual computing throughput reaches 8 TFLOPs when running a certain operator, the execution utilization is about 80%. High utilization means that the hardware computing resources of the large model are fully used; low utilization indicates that there is a bottleneck (such as memory bandwidth limitation, instruction delay, or insufficient parallelism), causing the computing unit to be in a waiting state frequently and failing to exert its full potential.

[0134] Further, for each operator, the number of unit operations, execution time overhead, and execution utilization are multiplied to obtain the current operator computing amount, which represents the effective computing workload contributed by the current operator in the actual running process. Then, the total operator computing amount is obtained by superimposing the computing amounts of all the current operators executed in the benchmark test, which accurately represents the total effective floating-point operations completed by the entire test program on the hardware. Finally, based on the ratio of the total operator computing amount to the unit energy consumption value measured in step 104.1.1, the computing energy efficiency ratio is determined. The computing energy efficiency ratio represents the hardware computing efficiency of the target large model in each sub-training step, i.e., how many effective floating-point operations the target large model can complete in the current sub-training step per joule of energy consumed (in the embodiments of the present application, the computing energy efficiency ratio corresponding to each sub-training step is set to be the same).

[0135] Exemplarily, a typical training node is selected, a training task of a target large model is deployed on the node, and only a single acceleration device or a small number of acceleration devices (such as an NPU or a GPU) are used to reduce measurement interference. During the running of the task, an acceleration device management tool is enabled to collect the power P (unit: watt) of the acceleration device in real time, where the acceleration device management tool can be a Nvidia system management interface such as the npu-smi or nvidia-smi tool. At the same time, a performance analysis tool is used to record performance data in the training process, for example, the performance analysis tool can be PyTorch Profiler. Of course, the acceleration device management tool and the performance analysis tool can be specifically selected according to actual conditions, and the embodiments of the present application do not limit this.

[0136] After the training task is completed, the calculation time T (unit: second) of each sub-training step of the model is extracted from the time interval recording information (step_trace_time) of PyTorch Profiler; then, the execution time overhead (aic_total_cycles) and the execution utilization ratio (aic_mac_ratio) of each operator i are read; then, the total calculation amount C (total operator calculation amount) of a single acceleration device (covering various operators) is calculated based on the following formula <3>:

[0137] <3>

[0138] where i represents the current operator, and there are I different operators in total; represents the unit operation times; represents the execution time overhead corresponding to the current operator; represents the execution utilization ratio of the current operator. The reason why the unit operation times are 8192 in this example is that in each calculation period, the calculation core of the operator can perform 16*16*16=4096 multiplication and addition operations, and each multiplication and addition includes 2 floating point operations of multiplication and addition, and finally the total operator calculation amount C is obtained.

[0139] Further, the calculation energy efficiency ratio is determined according to the following formula <4> :

[0140] <4>

[0141] wherein, represents the unit energy consumption value.

[0142] It can be understood that, unlike the traditional method, the embodiments of the present application do not rely on theory or general hardware specification parameters to predict the calculation power consumption of the model, but through operator-level fine-grained performance profiling and measurement of actual unit energy consumption values, the calculation energy efficiency ratio and the communication energy efficiency ratio of the model are respectively predicted. In this way, the embodiments of the present application fully consider various overheads and utilization loss of the model in actual operation, thereby ensuring the high reliability of the two key conversion coefficients, and in turn, the two key conversion coefficients provide a highly reliable basis for accurately converting the abstract calculation amount information and communication amount information into actual calculation power consumption values and communication power consumption values, greatly improving the real fitting degree and final accuracy of the overall power consumption prediction of the model.

[0143] Next, the "power consumption value calculation" aspect in step 104 is introduced:

[0144] In some embodiments, as shown in Figure 5 , Figure 5 is Figure 2 another optional implementation flowchart of step 104 in

[0145] 104.2.1 If the parallel strategy simultaneously includes tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism, for tensor parallelism, predict the tensor parallel communication amount information and the tensor parallel communication energy efficiency ratio at each sub-training step, and determine the tensor parallel communication power consumption based on the ratio of the tensor parallel communication amount information to the tensor parallel communication energy efficiency ratio;

[0146] 104.2.2 For data parallelism, predict the data parallel communication amount information and the data parallel communication energy efficiency ratio at each sub-training step, and determine the data parallel communication power consumption based on the ratio of the data parallel communication amount information to the data parallel communication energy efficiency ratio;

[0147] 104.2.3 For pipeline parallelism, predict the pipeline parallel communication amount information and the pipeline parallel communication energy efficiency ratio at each sub-training step, and determine the pipeline parallel communication power consumption based on the ratio of the pipeline parallel communication amount information to the pipeline parallel communication energy efficiency ratio;

[0148] 104.2.4 For expert parallelism, predict the expert parallel communication amount information and the expert parallel communication energy efficiency ratio at each sub-training step, and determine the expert parallel communication power consumption based on the ratio of the expert parallel communication amount information to the expert parallel communication energy efficiency ratio;

[0149] 104.2.5 Superimpose the tensor parallel communication power consumption, the data parallel communication power consumption, the pipeline parallel communication power consumption, and the expert parallel communication power consumption to obtain the communication power consumption value.

[0150] The following describes steps 104.2.1 to 104.2.5 in detail.

[0151] wherein tensor parallelism refers to splitting the computation task of a single large matrix operation (such as matrix multiplication of a fully connected layer) into multiple devices for parallel execution. For example, the weight matrix is split by row or column, and different devices are responsible for calculating a part of the output, and then the results are aggregated through communication. The tensor parallel computing method can reduce the memory occupation of a single device and improve the computing speed, but introduces communication overhead between multiple devices.

[0152] wherein data parallelism refers to splitting a training data batch into multiple sub-batches and distributing them to different devices (such as GPUs or NPUs), each device holding a complete copy of the model to perform independent forward calculation and back propagation, and finally synchronizing the gradient and updating the model parameters. When the model is large or the number of devices is large, the communication overhead of gradient synchronization becomes a bottleneck, that is, the communication overhead generated in the data parallel process is a part of the model communication power consumption that cannot be ignored.

[0153] wherein pipeline parallelism splits the model by layer so that different devices are responsible for different parts of the model, forming a processing structure similar to a “pipeline”. The data batch is further divided into micro-batches, which flow through each device in turn. Similarly, in the pipeline parallel process, the multiple devices of the model need to transmit activation values (forward) and gradients (backward) between each other, and these cross-device data transmissions bring communication overhead.

[0154] wherein expert parallelism is mainly applied to hybrid expert models, which contain multiple “expert” sub-networks, but only a few of them are activated at each forward propagation. Expert parallelism distributes these expert sub-networks to different devices, and each device only stores and calculates the selected experts. This way can expand the model capacity without significantly increasing the computing amount, but the uneven load between experts, as well as the communication of routing and activation results, will bring additional overhead.

[0155] Based on this, the embodiments of the present application need to determine the communication amount information and the communication energy efficiency ratio under each parallel strategy, and then determine the communication power consumption under the current parallel strategy.

[0156] Further, the embodiments of the present application deconstruct the communication power consumption in the mixed parallel scenario. Firstly, for the tensor parallel strategy, splitting a single operator (such as matrix multiplication) to multiple computing devices will inevitably lead to frequent exchange of intermediate calculation results between devices, therefore, the prediction device will predict the tensor parallel communication information generated at each sub-training step according to the model structure and the tensor parallel scale. At the same time, since the tensor parallel communication usually occurs on the high-speed, low-latency inter-chip interconnection, the embodiments of the present application will match a dedicated tensor parallel communication energy efficiency ratio for it. Finally, based on the ratio of the specific tensor parallel communication information and the tensor parallel communication energy efficiency ratio, the tensor parallel communication power consumption can be determined independently.

[0157] Further, the power consumption contribution of the data parallel strategy is independently calculated. Since data parallelism needs to accelerate training by replicating the entire model on multiple devices and splitting data batches, the core communication overhead is to synchronize the gradient between all devices after back propagation, therefore, the embodiments of the present application will predict the data parallel communication information required for gradient synchronization at each sub-training step to determine the communication overhead. Considering that data parallel communication often crosses nodes and uses networks such as Ethernet or InfiniBand, the energy efficiency characteristics are significantly different from inter-chip interconnection, therefore, the embodiments of the present application will also match an independent data parallel communication energy efficiency ratio for it. Subsequently, based on the ratio of the data parallel communication information and the data parallel communication energy efficiency ratio, the data parallel communication power consumption can be determined.

[0158] Further, the communication overhead introduced by the pipeline parallel strategy is calculated. Since pipeline parallelism is to place different layers (or stages) of the model on different devices to form a "pipeline", the communication overhead mainly occurs between adjacent stages for transmitting forward propagation activation values and backward propagation gradients, therefore, the embodiments of the present application will predict the pipeline parallel communication information generated by such cross-stage data transmission at each sub-training step. Since this point-to-point communication mode and link characteristics also have their own unique characteristics, the embodiments of the present application will calibrate a corresponding pipeline parallel communication energy efficiency ratio, and determine the pipeline parallel communication power consumption based on the ratio of the pipeline parallel communication information and the pipeline parallel communication energy efficiency ratio.

[0159] Further, the power consumption of the expert parallel strategy specific to the hybrid expert model is analyzed. The expert parallel distributes different "expert" modules on different devices, and data needs to be dynamically routed between devices to access the specified expert. The main overhead of this All-to-All mode of communication is the expert parallel communication. Therefore, the embodiments of the present application will predict the expert parallel communication volume information generated by data routing at each sub-training step, and match it with an expert parallel communication energy efficiency ratio that can reflect its communication energy consumption characteristics. Finally, based on the ratio of the expert parallel communication volume information and the expert parallel communication energy efficiency ratio, the expert parallel communication power consumption is determined.

[0160] Further, the tensor parallel communication power consumption, data parallel communication power consumption, pipeline parallel communication power consumption and expert parallel communication power consumption determined under the different parallel strategies are superimposed to determine a total communication power consumption value obtained by structured calculation. This value is composed of the accurate power consumption contributions of each parallel strategy and fully reflects the real communication energy consumption under complex parallel configurations.

[0161] Generally, the parallel strategy includes tensor parallel, data parallel, pipeline parallel and expert parallel, but it is also possible to include only one or several of them according to the actual situation. When the parallel strategy does not simultaneously include all four, the corresponding communication energy efficiency ratio of the involved parallel strategy is determined, and the communication volume information under the corresponding parallel strategy is determined according to steps A.1 to A.3, steps B.1 to B.3, steps C.1 to C.3, steps D.1 to D.5, and then the communication power consumption value under the corresponding parallel strategy is determined. Moreover, the calculation formula for determining the total power consumption value of the model in step 105 will also be adjusted, i.e., only the calculation power consumption value and the communication power consumption value under the involved parallel strategy are superimposed. The embodiments of the present application will be described for the case where the parallel strategy includes all four, and other cases can be derived from the embodiments of the present application, so they will not be described again.

[0162] Further, during the model training process, model parameters and intermediate results need to be frequently synchronized between a large number of computing devices, and these devices are interconnected through hardware architectures of different levels. Different connection methods (such as on-chip buses, intra-card interconnections, and network protocols) have great differences in physical distance, bandwidth, and delay, so the communication performance is usually different in different scenarios. In order to comprehensively evaluate and optimize the distributed training efficiency, independent testing must be performed on these significantly different connection levels, thus generating three types of communication scenarios: cross-chip, cross-card and cross-node.

[0163] Among them, cross-die communication refers to data exchange between different computing chips (Die) or computing core clusters within the same AI acceleration card (such as GPU, NPU card). Modern high-end acceleration cards often use multi-chip integration design to package multiple small chips together to work together.

[0164] Among them, cross-card communication refers to data transmission between different acceleration cards within the same server (node). For example, a server is equipped with 8 GPUs, and these GPUs need to synchronize the gradient. Cross-card communication is generally performed through a high-speed interconnection bus (such as PCIe) on the motherboard or a dedicated board-level interconnection technology, and its bandwidth and delay performance are usually between cross-die and cross-node.

[0165] Among them, cross-node communication refers to data exchange between different servers (computing nodes). In a large-scale cluster, training tasks are distributed across hundreds or even thousands of servers, and nodes are connected through external networks (such as InfiniBand, Ethernet). Cross-node communication has the farthest physical distance and relies on network switches, which usually have the lowest bandwidth and the highest delay.

[0166] Further, the total operator computation amount C, power P, and computation time T of different parallel strategies in different communication scenarios will have subtle differences in numerical values. In order to improve the accuracy of the calculation of the energy efficiency ratio by the embodiments of the present application, for each parallel strategy, the total operator computation amount C, power P, and computation time T in each communication scenario are measured, and the corresponding different calculation energy efficiency ratios of the current parallel strategy in cross-die communication, cross-card communication, and cross-node communication are determined 、 、 , so as to subsequently determine the corresponding communication power value according to the communication mode selected by the parallel strategy.

[0167] It can be understood that the embodiments of the present application take into account the great differences in communication mode, communication data volume, and physical links (such as cross-chip, cross-card, and cross-node) relied on by different parallel strategies, and therefore the corresponding communication energy consumption is not the same. The embodiments of the present application establish an independent communication power consumption calculation model for each parallel strategy, so as to be able to compare and analyze the energy efficiency performance of different parallel scales and granularities under a given hardware, so as to guide the model splitting strategy, job scheduling strategy, and even help users make power-sensitive deployment decisions among multiple NPU architectures, thereby realizing the optimal cost-effective model parallel configuration.

[0168] The following will introduce how to determine the corresponding communication amount information for different parallel strategies. It needs to be noted in advance that each sub-training step in each parallel strategy includes multiple micro-batch times, that is, one sub-training step includes multiple micro-batches:

[0169] In some embodiments, as Figure 6 shown, Figure 6 is Figure 5 an optional implementation flowchart of step 104.2.1 in FIG. 1, predicting tensor parallel communication amount information at each sub-training step, comprising the following steps:

[0170] A.1 determining a tensor parallel scale of a target large model, wherein the target large model comprises a first number of first acceleration devices;

[0171] A.2 determining a first data processing amount of each first acceleration device, determining a tensor parallel overhead of each first acceleration device based on the tensor parallel scale, and multiplying the first data processing amount and the tensor parallel overhead to obtain a first data propagation amount;

[0172] A.3 predicting tensor parallel communication amount information at each sub-training step based on the number of micro-batches, the first data propagation amount, and the first number.

[0173] The following describes steps A.1 to A.3 in detail.

[0174] In some embodiments, for tensor parallelism (TP), in order to accurately evaluate the power consumption contribution of TP, the communication data amount generated by TP needs to be modeled:

[0175] First, determine the tensor parallel scale t of the target large model. The tensor parallel scale t defines the parallel degree of a single computing operation (such as a matrix multiplication) that is divided and cooperatively executed, and this parameter can be determined by a configuration file. Moreover, under this configuration, the training task of the target large model will utilize the first number of first acceleration devices, which together form a tensor parallel group and cooperatively complete the computing task that would have been performed by a single device. The first acceleration device can be an NPU, a GPU, or other acceleration devices. The “first” in the first acceleration device is only to distinguish from the acceleration devices used under other parallel strategies, and has no other special meaning. In addition, the specific device type of the first acceleration device can be set according to actual conditions, and the first number of the first acceleration devices is also selected according to actual conditions, and the embodiments of the present application do not limit this.

[0176] Then, a first data processing amount corresponding to each first acceleration device is determined, the first data processing amount refers to the size of the intermediate activation tensor or gradient tensor held by a single first acceleration device that needs to be exchanged with other first acceleration devices in one forward or backward propagation; Next, a dimensionless tensor parallel overhead is calculated based on the tensor parallel scale, and the tensor parallel overhead factor quantifies the communication growth effect caused by a series of communication operations as the number of parallel devices increases; Finally, by multiplying the basic first data processing amount and the tensor parallel overhead factor, the first data propagation amount can be obtained, which accurately represents the total communication data amount generated by a single first acceleration device when processing a micro-batch data.

[0177] Exemplarily, for one of the Transformer blocks of the model, the total amount of data (first data processing amount) processed by a single first acceleration device (assuming NPU) in forward and backward propagation is bshR, because in the process of forward and backward propagation, each token will generate an h-dimensional representation vector, and the entire batch contains bs tokens.

[0178] Assuming that the Ring-AllReduce algorithm is used for TP communication, each NPU needs to send data for 2(t-1) times under the tensor parallel scale t, and each time the total data is 1 / t, so the tensor parallel overhead can be determined as 2(t-1) / t. Further, as shown in the following formula <5>, the first data propagation amount sent by each NPU to calculate a Transformer block is determined is:

[0179] <5>

[0180] wherein, bs represents the micro-batch size; s represents the sequence length; R represents the size of each basic data unit, for half-precision floating point number (FP16 precision data), R=2 bytes (Byte), and for single-precision floating point number (FP32 precision data), R=4 Byte; h represents the hidden layer dimension.

[0181] Further, considering that 4 TP communication operations are usually performed in each Transformer block, including attention layer, Multi-Layer Perceptron (MLP), forward propagation and backward propagation, if a large model includes l Transformer blocks, in the case of an example including four parallel strategies at the same time, assuming that the pipeline parallel scale is p, tpd (the first quantity) acceleration devices are used, and one sub-training step includes m micro-batch times, then a single NPU needs to process l / p Transformer blocks in one sub-training step. In this case, the total data amount (tensor parallel communication amount ) that all NPUs need to send when performing TP communication can be represented by the following formula <6>:

[0182] <6>

[0183] wherein, is the micro-batch time; is the first data propagation amount; is the first quantity.

[0184] It can be understood that the embodiments of the present application are aimed at tensor parallelism, first identify the benchmark load (i.e., the first data processing amount) of the current communication through the tensor parallel scale, and innovatively introduce the tensor parallel overhead to quantify the nonlinear impact of parallelism on communication amount, and finally consider and aggregate the micro-batch time and the first quantity parameter, so that the obtained tensor parallel communication amount information is highly close to the physical reality. This modeling method based on the first principle not only provides a solid and reliable data basis for subsequent accurate calculation of communication power consumption, but also enables the prediction device to quantitatively analyze and weigh the impact of the tensor parallel scale on the total communication overhead of the model, and is more convenient for relevant personnel to make more optimal tensor parallel strategy configuration decisions according to the calculated tensor parallel communication amount information.

[0185] In some embodiments, as Figure 7 illustrated, Figure 7 is Figure 5 an optional implementation flowchart of step 104.2.2 in the method for predicting data parallel communication amount information under each sub-training step, including the following steps:

[0186] B.1 determining the data parallel scale of the target large model, wherein the target large model includes a second quantity of second acceleration devices;

[0187] B.2 determining the second data processing amount of each second acceleration device based on the total parameter set, determining the data parallel overhead of each second acceleration device based on the data parallel scale, and multiplying the second data processing amount and the data parallel overhead to obtain the second data propagation amount;

[0188] B.3 predicting data parallel communication volume information at each sub-training step based on the second data propagation volume and the second quantity.

[0189] The following describes steps B.1 to B.3 in detail.

[0190] In some embodiments, for data parallelism (DP), in order to accurately evaluate the power consumption contribution of DP, the communication data volume generated by DP needs to be modeled:

[0191] First, determine the data parallelism scale d of the target large model. The data parallelism scale d defines the parallelism degree of the training data batch being cut and distributed to different devices for parallel processing, and this parameter can also be determined through a configuration file. And under this configuration, the training task of the target large model will utilize the second quantity of second acceleration devices, each of which holds a complete copy of the model in order to perform independent forward and backward propagation, and finally completes parameter synchronization through gradient aggregation, which together constitute a data parallel group. Among them, the second acceleration device can also be an NPU, GPU or other acceleration device, and the "second" in the second acceleration device is only to distinguish from the acceleration devices used in other parallel strategies, and has no other special meaning. In addition, the specific device type of the second acceleration device can be set according to actual conditions, and the second quantity of the second acceleration device is also selected according to actual conditions, and the embodiments of the present application do not limit this.

[0192] Next, determine the second data processing volume of each second acceleration device based on the total parameter set. In the data parallel scenario, the main communication operation is AllReduce, which is a distributed communication operation that refers to all participating devices reducing their data (such as summing) and broadcasting the final reduction result to all devices, so that each device obtains the same result. Therefore, the second data processing volume mainly refers to the total amount of gradients calculated on a single device after one backward propagation that needs to be synchronized with other devices in the group, which is directly related to the total parameter set of the model; Next, determine the data parallel overhead of each second acceleration device based on the data parallelism scale, which quantifies the communication protocol overhead brought about by the increase in parallel nodes; Finally, multiply the second data processing volume by the data parallel overhead factor to obtain the second data propagation volume, which represents the effective communication data volume generated by a single second acceleration device in one gradient synchronization.

[0193] Exemplarily, under the data parallel strategy, the total parameter set N of the target large model is evenly distributed in the p pipeline stages and t tensor parallel groups, so that the data processing amount of each NPU (second acceleration device) is N / (pt), and in the d-way data parallel AllReduce operation, it is determined that each NPU needs to send data for 2(d-1) times, and each time, 1 / d of the total data is sent, that is, the data parallel overhead is 2(d-1) / t. Further, as shown in the following formula <7>, the initial second data propagation amount that each NPU needs to accumulate and send is determined

[0194] <7>

[0195] wherein NR / pt represents the second data processing amount; and 2(d-1) / d represents the data parallel overhead. In addition, formula <5> only considers the DP synchronization of gradient data, and in actual use, if the optimizer state is stored in pieces, the optimizer state includes momentum and variance data, which also needs to be synchronized by DP, and the second data propagation amount will be expanded to 3 times the current calculation value, that is, In this case, the total data amount (data parallel communication amount ) that all NPU (tpd) needs to send when performing DP communication can be represented by the following formula <8>:

[0196] <8>

[0197] wherein represents the second data propagation amount; represents the second amount. In addition, since the gradient value and the optimizer state are generally stored in FP32 precision, at this time, R=4Byte.

[0198] It can be understood that by introducing the technical feature of data parallel overhead, the model can accurately capture the non-linear growth effect caused by the communication protocol itself as the number of parallel nodes changes. This fine prediction based on physical processes not only provides highly reliable input for subsequent calculation of data parallel communication power consumption, but also enables the prediction device to quantitatively analyze and weigh the impact of data parallel scale on the total communication overhead of the model, and is more convenient for relevant personnel to make more optimal data parallel strategy configuration decisions according to the calculated data parallel communication amount information.

[0199] In some embodiments, as shown in Figure 8 , step 104.2.3 in Figure 8 is an optional implementation flowchart for predicting the pipeline parallel communication amount information under each sub-training step, including the following steps: Figure 5

[0200] ​​C.1 determine a pipeline parallel scale of the target large model, wherein the target large model comprises a third quantity of communication groups;

[0201] C.2 determine a third data processing amount of each communication group, determine a pipeline parallel overhead of each communication group based on the pipeline parallel scale, and determine a third data propagation amount based on the third data processing amount and the pipeline parallel overhead;

[0202] C.3 predict pipeline parallel communication amount information under each sub-training step based on the micro-batch number, the third data propagation amount, and the third quantity.

[0203] The following describes steps C.1 to C.3 in detail.

[0204] In some embodiments, for pipeline parallel (PP), in order to accurately evaluate the power consumption contribution of PP, the communication data volume generated by PP needs to be modeled:

[0205] First, the pipeline parallel scale p of the target large model is determined. The pipeline parallel scale p defines the number of pipeline stages into which the model is vertically divided, and this parameter can be determined through a configuration file. Under this configuration, the target large model is distributed on a third quantity of communication groups, each of which usually contains one or more acceleration devices, and each communication group is responsible for executing a specific stage in the pipeline.

[0206] Next, the third data processing amount of each communication group is determined. The third data processing amount mainly refers to the total amount of data of the activation values that a pipeline stage needs to pass to the previous stage or the gradients that it needs to pass to the next stage during the calculation of a micro-batch; then, the pipeline parallel overhead of each communication group is determined based on the pipeline parallel scale, which can quantify the additional communication burden introduced by pipeline scheduling, handshake protocols, and possible "pipeline bubbles"; subsequently, the third data propagation amount is determined based on the third data processing amount and the pipeline parallel overhead, and the third data propagation amount represents the actual communication data volume generated by a single communication group when processing a micro-batch.

[0207] For example, in a micro-batch processing process under the pipeline parallel strategy, an NPU needs to send or receive data of size bshR to its predecessor or successor stage (third data processing amount). Since the data of each micro-batch needs to be transmitted between the first communication group (rank0) and the last communication group (rank(p-1)) in the forward or reverse direction, the pipeline parallel overhead can be determined as , and then, as shown in the following formula <9>, the cumulative communication amount (third data propagation amount ) of a single communication group under pipeline parallel can be determined as:

[0208] <9>

[0209] Further, for m mini-batches in one sub-training step, the total PP communication data volume (pipeline parallel communication volume information ) of all td (third quantity) communication groups will be calculated by the following formula <10>:

[0210] <10>

[0211] It can be understood that the embodiments of the present application include the complexity of pipeline scheduling and its actual impact on communication in the pipeline communication power consumption modeling category, so that the prediction result can accurately reflect the influence of the change of hyperparameters such as pipeline parallel scale and mini-batch size on the communication volume under the pipeline parallel strategy, thereby providing a highly reliable basis for subsequent pipeline parallel communication power consumption. At the same time, the pipeline parallel communication volume information and the pipeline parallel communication power consumption can also provide clear quantitative guidance for relevant personnel to optimize pipeline efficiency and reduce "bubble" overhead.

[0212] In some embodiments, as Figure 9 shown, Figure 9 is Figure 5 an optional implementation flowchart of step 104.2.4 in

[0213] D.1 If the target large model is a hybrid expert type large model and the parallel strategy is a conventional expert parallel, calculate the ratio between the preset unit value and the initial number of experts to obtain a fourth ratio, and calculate the difference between the preset unit value and the fourth ratio to obtain a second difference, wherein the target large model includes a target number of target experts;

[0214] D.2 Multiply the target number of experts and the second difference to obtain an expected number of experts, and determine a fourth data processing amount of each target expert, and determine a fourth data propagation amount based on the fourth data processing amount and the expected number of experts;

[0215] D.3 Based on the number of mini-batches, the fourth data propagation amount and the number of expert layers, predict the expert parallel communication volume information under each sub-training step;

[0216] D.4 If the target large model is a hybrid expert type large model and the parallel strategy is a grouped expert parallel, determine the number of grouped experts;

[0217] D.5 Based on the number of mini-batches, the fourth data propagation amount, the number of expert layers and the number of grouped experts, predict the expert parallel communication volume information under each sub-training step.

[0218] In some embodiments, the expert parallelism includes regular expert parallelism and grouped expert parallelism.

[0219] The following describes steps D.1 to D.5 in detail.

[0220] In some embodiments, the expert parallelism (EP) is further divided into regular expert parallelism and grouped expert parallelism.

[0221] (1) If the target large model is a hybrid expert type large model, and the parallel strategy is regular expert parallelism, first determine the expected number of experts by the following formula <11> :

[0222] <11>

[0223] wherein 1 represents a preset unit value; represents the fourth ratio; represents the second difference; represents the target number of experts.

[0224] In regular expert parallelism, there are e initial experts in total, and for a single token, there are expected to be active target experts in k positions, which need to be sent once forward and returned after calculation, and the reverse process is the same. Therefore, for each layer of the model, the fourth data propagation amount sent is which can be represented by the following formula <12>:

[0225] <12>

[0226] wherein represents the fourth data processing amount; represents the expected number of experts.

[0227] Further, since there are expert layers in the model, in expert parallelism case (1), the expert parallel communication amount information of the following expert layers in each sub-training step can be calculated by the following formula <13>:

[0228] <13>

[0229] wherein represents the number of expert layers.

[0230] (2) If the target large model is a hybrid expert type large model, and the parallel strategy is grouped expert parallelism, then k experts are always selected from expert groups, at this time, the communication amount is no longer multiplied by but by the number of expert groups Therefore, in the expert parallel case (2), the expert parallel communication information under each sub-training step is The following formula (14) can be used to calculate:

[0231] <14>

[0232] Wherein, represents the number of grouped experts.

[0233] It can be understood that the embodiments of the present application clearly distinguish between the two different expert parallel communication architectures of regular expert parallel and grouped expert parallel, and establish their respective exclusive prediction paths based on first principles. For different expert parallel strategies, the most basic data processing is started, and the key factor that can accurately reflect the communication mode (such as the second difference of All-to-All or the number of grouped experts of grouped communication) is introduced for quantization, so that no matter what expert parallel strategy the large model adopts, the predicted expert parallel communication information can highly approximate the physical reality, thereby providing a solid and reliable foundation for subsequent power consumption calculation, and providing a clear quantitative basis for users to select and optimize the expert parallel strategy.

[0234] After determining the communication information under each parallel strategy, the corresponding communication power consumption values of each parallel strategy are determined by the following formulas (15) to (18):

[0235] (1) For tensor parallel, the corresponding tensor parallel communication energy efficiency ratio is determined by step 104.1.1 , combined with the tensor parallel communication information obtained in steps A.1 to A.3 , the tensor parallel communication power consumption value is determined by the following formula (15) :

[0236] <15>

[0237] Wherein, in a feasible embodiment, when the tensor parallel scale t = 2, numerically equal to the communication energy efficiency ratio calculated under cross-core communication , when the tensor parallel scale t > 2, the communication energy efficiency ratio calculated under cross-card communication is used .

[0238] (2) For data parallel, the corresponding data parallel communication energy efficiency ratio is determined by step 104.1.2 , combined with the data parallel communication information obtained in steps B.1 to B.3 , the data parallel communication power consumption value is determined by the following formula (16) is:

[0239] <16>

[0240] wherein the data parallelism is usually cross-node communication, thus in a feasible embodiment, numerically equal to the communication energy efficiency ratio calculated under cross-node communication .

[0241] (3) For pipeline parallelism, determine the corresponding pipeline parallel communication energy efficiency ratio through step 104.1.3 , in combination with the pipeline parallel communication traffic information obtained in steps C.1 to C.3 , determine the pipeline parallel communication power value through the following formula (17) is:

[0242] <17>

[0243] wherein the data parallelism is usually cross-node communication, thus in a feasible embodiment, numerically equal to the communication energy efficiency ratio calculated under cross-node communication .

[0244] (4) For expert parallelism, determine the corresponding expert parallel communication energy efficiency ratio through step 104.1.4 , in combination with the expert parallel communication traffic information obtained in steps D.1 to D.5 , determine the expert parallel communication power value through the following formula (18) is:

[0245] <18>

[0246] wherein the expert parallelism is usually cross-card communication, thus in a feasible embodiment, numerically equal to the communication energy efficiency ratio calculated under cross-card communication .

[0247] Step 105, superimpose the calculation power value and the communication power value to obtain the per-step power consumption value corresponding to the current sub-training step number, and multiply the per-step power consumption value by the total training step number to predict the total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set.

[0248] The following describes step 105 in detail.

[0249] In some embodiments, the computational power consumption value and communication power consumption value independently determined in step 104 are superimposed to obtain a power consumption value for each step corresponding to the current sub-training step number that comprehensively reflects the dual overhead of computation and communication. Finally, the power consumption value for each step is multiplied by the total number of training steps obtained in step 101 to accurately predict the total power consumption value of the target large model under the parallel training conditions defined by the given configuration file and total parameter set. Specifically, this can be achieved through the following formula. <19> and <20> express:

[0250] Total power consumption = Power consumption per step * Total number of training steps <19>

[0251] Power consumption per step = computational load information per step / computational energy efficiency ratio + communication load information per step / communication energy efficiency ratio <20>

[0252] Among them, formula <18> The rationale for this is that the power consumption of a single training step is mainly determined by computational overhead, hardware efficiency, and data movement. Under fixed hardware environment and model structure, the power consumption value of each step remains relatively stable. The total number of training steps, on the other hand, is determined by human intervention. Larger models or complex tasks often require more training steps to achieve convergence.

[0253] Furthermore, in this embodiment, considering the differences between the four parallel methods, the actual calculation method for the power consumption value of each step is expressed as the following formula. <21> :

[0254] Power consumption per step = computational load per step / computational energy efficiency ratio + tensor parallel communication load per step / tensor parallel communication energy efficiency ratio + data parallel communication load per step / data parallel communication energy efficiency ratio + pipelined parallel communication load per step / pipelined parallel communication energy efficiency ratio + expert parallel communication load per step / expert parallel communication energy efficiency ratio <21>

[0255] It is understandable that the embodiments of this application abandon the rough estimation method of treating power consumption as an indivisible whole in the traditional method, and innovatively decouple the total power consumption into two independent dimensions: computing power consumption and communication power consumption. In the communication power consumption calculation dimension, it covers mainstream multi-dimensional parallel strategies such as data parallelism, pipeline parallelism, tensor parallelism and expert parallelism, thereby improving the accuracy of the final determined total power consumption value. As a result, the embodiments of this application can provide pre-assessment in the stages of budget application, resource application, power load planning and carbon emission control during large model training, effectively avoiding unreasonable resource allocation and schedule delays, and providing basic support for the operation and maintenance management of intelligent computing centers.

[0256] like Figure 10 As shown, Figure 10An optional device module schematic diagram of the power consumption prediction device in the large model parallel training scenario is provided in the embodiment of the present application. The power consumption prediction device in the large model parallel training scenario can include the following modules 201 to 205:

[0257] The acquisition module 201 is configured to acquire a configuration file, a total parameter set and a total training step number of a target large model, wherein the total training step number includes a plurality of same sub-training step numbers.

[0258] The calculation amount information determination module 202 is configured to determine a model type to which the target large model belongs based on the configuration file, and to obtain calculation amount information of the target large model in each sub-training step number according to the model type.

[0259] The communication amount information determination module 203 is configured to determine at least one parallel strategy of the target large model based on the configuration file, and to obtain communication amount information of the target large model in each sub-training step number according to the at least one parallel strategy.

[0260] The local power consumption value determination module 204 is configured to obtain a calculation energy efficiency ratio and a communication energy efficiency ratio based on the total parameter set, to determine a calculation power consumption value based on the calculation amount information and the calculation energy efficiency ratio, and to determine a communication power consumption value based on the communication amount information and the communication energy efficiency ratio.

[0261] The total power consumption value determination module 205 is configured to obtain an every-step power consumption value corresponding to the current sub-training step number by superimposing the calculation power consumption value and the communication power consumption value, and to obtain a total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set by multiplying the every-step power consumption value and the total training step number.

[0262] The specific implementation of the prediction device is basically the same as that of the above-mentioned specific embodiment of the prediction method, and will not be repeated here.

[0263] The embodiment of the present application further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program. The processor implements the above-mentioned prediction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0264] As shown in Figure 11 , the hardware structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in Figure 11 . The electronic device includes:

[0265] The processor 301 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0266] The memory 302 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 302 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 302 and are called and executed by the processor 301 to implement the prediction method of the embodiments of the present application.

[0267] The input / output interface 303 is configured to implement information input and output.

[0268] The communication interface 304 is configured to implement the communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0269] The bus 305 is configured to transmit information between the components (for example, the processor 301, the memory 302, the input / output interface 303, and the communication interface 304) of the device.

[0270] The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are connected to each other by the bus 305 to realize the communication connection between the components in the device.

[0271] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to implement the prediction method.

[0272] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0273] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0274] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0275] The device embodiments described above are only schematic, and units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0276] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0277] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0278] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0279] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0280] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0281] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0282] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0283] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A power consumption prediction method in a large model parallel training scene, characterized in that, The method comprises the following steps: obtaining a configuration file, a total parameter set and a total training step number of a target large model, wherein the total training step number comprises a plurality of same sub-training step numbers; determining a model type to which the target large model belongs based on the configuration file, and predicting calculation amount information of the target large model in each of the sub-training step numbers according to the model type; determining at least one parallel strategy of the target large model based on the configuration file, and predicting communication amount information of the target large model in each of the sub-training step numbers according to the at least one parallel strategy; predicting a calculation energy efficiency ratio and a communication energy efficiency ratio of the target large model in each of the sub-training step numbers, determining a calculation power consumption value based on the calculation amount information and the calculation energy efficiency ratio, and determining a communication power consumption value based on the communication amount information and the communication energy efficiency ratio; superimposing the calculation power consumption value and the communication power consumption value to obtain an energy consumption value of each step corresponding to the current sub-training step number, and multiplying the energy consumption value of each step by the total training step number to predict a total power consumption value of the target large model under parallel training conditions of the configuration file and the total parameter set.

2. The method of claim 1, wherein the method is performed in a large model parallel training scenario. The prediction of the calculation amount information of the target large model in each of the sub-training step numbers according to the model type comprises: obtaining a total data set to be processed by the target large model at each of the sub-training step numbers, and a preset floating point value; if the model type represents that the target large model is a dense large model, multiplying the preset floating point value, the total parameter set and the total data set to obtain first calculation amount information, and taking the first calculation amount information as the calculation amount information of the target large model in each of the sub-training step numbers; if the model type represents that the target large model is a hybrid expert type large model, determining a model layer number and an expert layer number of the target large model, calculating a difference value between the model layer number and the expert layer number to obtain a layer number difference value; calculating a ratio between the expert layer number and the model layer number to obtain a first ratio, and calculating a ratio between the layer number difference value and the model layer number to obtain a second ratio; determining a number of activated target experts from a plurality of initial experts of the target large model, and calculating a ratio between the number of initial experts and the number of target experts to obtain an expert number ratio; multiplying the expert number ratio and the first ratio to obtain a third ratio, multiplying a superimposition result of the second ratio and the third ratio and the first calculation amount information to obtain second calculation amount information, and taking the second calculation amount information as the calculation amount information of the target large model in each of the sub-training step numbers.

3. The method of claim 1, wherein the method is performed in a large model parallel training scenario. The target large model comprises a plurality of operators. The prediction of the calculation energy efficiency ratio and the communication energy efficiency ratio of the target large model in each of the sub-training step numbers comprises: obtaining a total communication amount set of the target large model, and a unit energy consumption value of the target large model in a unit time; determining a unit operation number, an execution time overhead and an execution utilization of each of the operators, multiplying the unit operation number, the execution time overhead and the execution utilization for each of the operators to obtain a current operator calculation amount; superimposing all the current operator calculation amounts to obtain a total operator calculation amount, and determining the computing energy efficiency ratio of the target large model in each of the sub-training steps based on a ratio of the total operator calculation amount to the unit energy consumption value; determining the communication energy efficiency ratio of the target large model in each of the sub-training steps based on a ratio of the total communication amount set to the unit energy consumption value.

4. The method of claim 2, wherein the method is performed by a processor. The communication amount information includes tensor parallel communication amount information, data parallel communication amount information, pipeline parallel communication amount information and expert parallel communication amount information, and the communication energy efficiency ratio includes tensor parallel communication energy efficiency ratio, data parallel communication energy efficiency ratio, pipeline parallel communication energy efficiency ratio and expert parallel communication energy efficiency ratio. The determining of the communication power consumption value based on the communication amount information and the communication energy efficiency ratio includes: If the parallel strategy simultaneously includes tensor parallel, data parallel, pipeline parallel and expert parallel, for the tensor parallel, the tensor parallel communication amount information and the tensor parallel communication energy efficiency ratio in each of the sub-training steps are predicted, and tensor parallel communication power consumption is determined based on a ratio of the tensor parallel communication amount information to the tensor parallel communication energy efficiency ratio; for the data parallel, the data parallel communication amount information and the data parallel communication energy efficiency ratio in each of the sub-training steps are predicted, and data parallel communication power consumption is determined based on a ratio of the data parallel communication amount information to the data parallel communication energy efficiency ratio; for the pipeline parallel, the pipeline parallel communication amount information and the pipeline parallel communication energy efficiency ratio in each of the sub-training steps are predicted, and pipeline parallel communication power consumption is determined based on a ratio of the pipeline parallel communication amount information to the pipeline parallel communication energy efficiency ratio; for the expert parallel, the expert parallel communication amount information and the expert parallel communication energy efficiency ratio in each of the sub-training steps are predicted, and expert parallel communication power consumption is determined based on a ratio of the expert parallel communication amount information to the expert parallel communication energy efficiency ratio; The communication power consumption value is obtained by superimposing the tensor parallel communication power consumption, the data parallel communication power consumption, the pipeline parallel communication power consumption and the expert parallel communication power consumption.

5. The method of claim 4, wherein the method is performed in a large model parallel training scenario. Each of the sub-training steps includes a plurality of micro batch times; The prediction of the tensor parallel communication amount information in each of the sub-training steps includes: determining a tensor parallel scale of the target large model, wherein the target large model includes a first number of first acceleration devices; determining a first data processing amount of each of the first acceleration devices, determining a tensor parallel overhead of each of the first acceleration devices based on the tensor parallel scale, and multiplying the first data processing amount and the tensor parallel overhead to obtain a first data propagation amount; predicting the tensor parallel communication amount information in each of the sub-training steps based on the micro batch times, the first data propagation amount and the first number.

6. The method of claim 5, wherein the method is performed in a large model parallel training scenario. The prediction of the data parallel communication quantity information under each of the sub-training steps comprises: determining a data parallel scale of the target large model, wherein the target large model comprises a second number of second acceleration devices; determining a second data processing amount of each of the second acceleration devices based on the total parameter set, determining a data parallel overhead of each of the second acceleration devices based on the data parallel scale, and multiplying the second data processing amount and the data parallel overhead to obtain a second data propagation amount; based on the second data propagation amount and the second number, predicting the data parallel communication quantity information under each of the sub-training steps.

7. The method of claim 5, wherein the method is performed in a large model parallel training scenario. The prediction of the pipeline parallel communication quantity information under each of the sub-training steps comprises: determining a pipeline parallel scale of the target large model, wherein the target large model comprises a third number of communication groups; determining a third data processing amount of each of the communication groups, determining a pipeline parallel overhead of each of the communication groups based on the pipeline parallel scale, and determining a third data propagation amount based on the third data processing amount and the pipeline parallel overhead; based on the micro-batch number, the third data propagation amount and the third number, predicting the pipeline parallel communication quantity information under each of the sub-training steps.

8. The method of claim 5, wherein the method is performed in a large model parallel training scenario. The expert parallel comprises a regular expert parallel and a grouped expert parallel; The prediction of the expert parallel communication quantity information under each of the sub-training steps comprises: if the target large model is the hybrid expert type large model and the parallel strategy is the regular expert parallel, calculating a fourth ratio between a preset unit value and the number of initial experts to obtain a fourth ratio, and calculating a second difference between the preset unit value and the fourth ratio to obtain a second difference, wherein the target large model comprises a target expert number of target experts; multiplying the target expert number and the second difference to obtain an expected expert number, and determining a fourth data processing amount of each of the target experts, determining a fourth data propagation amount based on the fourth data processing amount and the expected expert number; based on the micro-batch number, the fourth data propagation amount and the expert layer number, predicting the expert parallel communication quantity information under each of the sub-training steps; if the target large model is the hybrid expert type large model and the parallel strategy is the grouped expert parallel, determining a grouped expert number; based on the micro-batch number, the fourth data propagation amount, the expert layer number and the grouped expert number, predicting the expert parallel communication quantity information under each of the sub-training steps.

9. A power consumption prediction apparatus in a large model parallel training scene, characterized in that, comprises: an acquisition module configured to acquire a configuration file, a total parameter set and a total training step number of a target large model, wherein the total training step number comprises a plurality of same sub-training steps; a calculation amount information determination module configured to determine a model type to which the target large model belongs based on the configuration file, and predict calculation amount information of the target large model in each of the sub-training steps according to the model type; The communication amount information determination module is configured to determine at least one parallel strategy of the target large model based on the configuration file, and predict communication amount information of the target large model in each of the sub-training steps according to the at least one parallel strategy; The local power consumption value determination module is configured to predict a calculation energy efficiency ratio and a communication energy efficiency ratio of the target large model in each of the sub-training steps, determine a calculation power consumption value based on the calculation amount information and the calculation energy efficiency ratio, and determine a communication power consumption value based on the communication amount information and the communication energy efficiency ratio; The total power consumption value determination module is configured to superimpose the calculation power consumption value and the communication power consumption value to obtain an energy consumption value of each step corresponding to the current sub-training step, and multiply the energy consumption value of each step by the total training step to predict a total power consumption value of the target large model under the parallel training condition of the configuration file and the total parameter set.

10. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the power consumption prediction method in the large model parallel training scenario according to any one of claims 1 to 8 when executing the computer program. 11.A computer readable storage medium, storing a computer program, characterized in that, The computer program is executed by the processor to implement the power consumption prediction method in the large model parallel training scenario according to any one of claims 1 to 8.

Images