Method and device for adjusting large model training task based on computing power usage state of intelligent computing center cloud platform

Abstract

The application provides a method and device for adjusting large model training tasks based on computing power use states of an intelligent computing center cloud platform, and relates to the technical fields of intelligent computing centers, intelligent computing centers, computing power infrastructure and intelligent computing cloud technology. The method for adjusting large model training tasks based on computing power use states of an intelligent computing center cloud platform comprises the following steps: S1: obtaining a computing power use state, which is used to represent the use state of computing power in the intelligent computing center cloud platform during the training of a large model; S2: determining whether the computing power use state matches an expected state; and S3: adjusting the training task of the large model according to the computing power use state in the case where the computing power use state does not match the expected state. In this way, the phenomenon of wasted computing power can be reduced, and the training efficiency of the large model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent computing centers, wisdom computing centers, computing power infrastructure and wisdom computing cloud, and in particular to a method and device for adjusting large model training tasks based on computing power usage state of an intelligent computing center cloud platform. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "wisdom computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that uses large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, to provide the required computing power, data and algorithms for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model inference scenarios). An intelligent computing center covers facilities, hardware and software, and can provide full-stack capabilities from bottom computing power to top application enablement.

[0004] An "intelligent computing center" includes but is not limited to a "wisdom computing center".

[0005] A "wisdom computing center", also known as an artificial intelligence computing center, is a type of computing power infrastructure based on artificial intelligence theory and using artificial intelligence computing architecture to provide computing power services, data services and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing centers" and "wisdom computing centers". It is the ability of computer equipment or computing / data centers to process information. It is the ability of computer hardware and software to work together to perform certain computing needs. It is the computing power that achieves the output of target results by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity and data storage power. It mainly provides services to society through computing power infrastructure.

[0007] When training a large model on a current intelligent computing center cloud platform, the computing power of the intelligent computing center cloud platform is usually used to train the large model. However, when the usage state of the computing power does not reach the expected state during the training of the large model on the current intelligent computing center cloud platform, the computing power is easily wasted, and the training efficiency of the large model is very low. Therefore, since the emergence of intelligent computing centers, when the usage state of the computing power does not reach the expected state, the computing power is easily wasted, and the training efficiency of the large model is very low, which is a problem that needs to be solved. SUMMARY

[0008] The present application provides a method and device for adjusting large model training tasks based on computing power usage state of an intelligent computing center cloud platform, which solves the problem of wasting computing power and low training efficiency of large models when the usage state of the computing power does not reach the expected state.

[0009] To solve the above technical problems, the present application is implemented as follows:

[0010] In a first aspect, the present application provides a method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform, comprising:

[0011] Step S1: obtaining a computing power usage state, the computing power usage state being used to represent a usage state of computing power in the intelligent computing center cloud platform during training of a large model;

[0012] Step S2: determining whether the computing power usage state matches an expected state;

[0013] Step S3: in the case where the computing power usage state does not match the expected state, adjusting the training task of the large model according to the computing power usage state.

[0014] Optionally, the computing power usage state comprises at least one of the following: a computing card core utilization rate, a bandwidth of data exchange between memory and a computing card, a video memory occupancy rate, a video memory throughput, and a throughput between different computing cards.

[0015] Optionally, the computing power usage state comprises a computing card core utilization rate, and the step S3 comprises:

[0016] Step S31: in the case where a fluctuation rate of the computing card core utilization rate within a preset period is greater than an expected fluctuation rate, determining that the computing power usage state does not match the expected state;

[0017] Step S32: in the case where the computing power usage state does not match the expected state, adjusting a storage level of training data in the training task according to a fluctuation state of the computing card core utilization rate, the fluctuation state comprising the fluctuation rate.

[0018] Optionally, the computing power usage state comprises a throughput between different computing cards, and the step S3 comprises:

[0019] Step S33: in the case where the throughput between different computing cards is greater than a first throughput, determining that the computing power usage state does not match the expected state;

[0020] Step S34: in the case where the computing power usage state does not match the expected state, adjusting a batch sample quantity of a single computing card according to the throughput between different computing cards.

[0021] Optionally, the computing power usage state comprises a video memory throughput, and the step S3 comprises:

[0022] Step S35: In the case that the calculated GPU throughput is greater than the second throughput, it is determined that the computing power usage state does not match the expected state.

[0023] Step S36: In the case that the computing power usage state does not match the expected state, the parallelism of the computing power is adjusted according to the GPU throughput.

[0024] Optionally, the computing power usage state includes the bandwidth of data exchange between the memory and the computing card and the GPU occupancy rate, and the step S3 includes:

[0025] Step S37: In the case that the GPU occupancy rate does not reach the preset occupancy rate and the bandwidth of data exchange between the memory and the computing card reaches the preset bandwidth, it is determined that the computing power usage state does not match the expected state.

[0026] Step S38: The proportion of model weight data unloaded into the memory is adjusted according to the bandwidth of data exchange between the memory and the computing card and the GPU occupancy rate.

[0027] In a second aspect, the present application provides a device for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform, comprising:

[0028] An obtaining module is configured to obtain a computing power usage state, the computing power usage state being used to represent a usage state of computing power of the intelligent computing center cloud platform in a process of training a large model by the intelligent computing center cloud platform.

[0029] A determining module is configured to determine whether the computing power usage state matches an expected state.

[0030] An adjusting module is configured to, in the case that the computing power usage state does not match the expected state, adjust a training task of the large model according to the computing power usage state.

[0031] In a third aspect, the present application provides an electronic device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program is executed by the processor to implement the steps of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the first aspect.

[0032] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the first aspect.

[0033] In a fifth aspect, the present application provides a computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the first aspect described above.

[0034] In the present application, the computing power usage state is obtained, and the computing power usage state is used to represent the usage state of the computing power of the intelligent computing center cloud platform in the process of training the large model. It is determined whether the computing power usage state matches an expected state. In the case where the computing power usage state does not match the expected state, the training task of the large model is adjusted according to the computing power usage state. In this way, when it is determined that the computing power usage state does not match the expected state, the training task of the large model can be adjusted according to the computing power usage state, thereby reducing the phenomenon of waste of computing power and improving the training efficiency of the large model. BRIEF DESCRIPTION OF DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the present application. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0036] Figure 1 a flowchart of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the present application;

[0037] Figure 2 a flowchart of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the present application;

[0038] Figure 3 a structural schematic diagram of the device for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform according to the present application;

[0039] Figure 4 a structural schematic diagram of the electronic device according to the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described content is only a part of the present application, and not all. Based on the content of the present application, all other content obtained by those of ordinary skill in the art without creative labor is within the scope of the present application.

[0041] The ''computing power'' described in the application refers to: the ability of computer equipment or computing / data center to process information, the ability of computer hardware and software to cooperate to perform certain computing requirements, the computing power to achieve target result output by processing information data, the new type of productivity integrating information computing power, network carrying capacity and data storage power, mainly providing services to society through computing power infrastructure.

[0042] The ''computing power'' (Computational Power, CP) described in the application refers to: a kind of ability of data center server to process data and realize result output, a comprehensive index for measuring the computing power of data center, including general computing power, supercomputing power and intelligent computing power. The commonly used unit of measurement is the number of floating point operations per second (FLOPS, 1EFLOPS=10^18FLOPS), and the larger the value represents the stronger the comprehensive computing power. According to the estimation, 1EFLOPS is about 5 Tianhe 2A or 500,000 mainstream server CPUs or 2 million mainstream notebook computing power output. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 .

[0043] The ''carrying capacity'' (Network Power, NP) described in the application refers to: the performance of data transmission capacity of computing power infrastructure, including network architecture, network bandwidth, transmission delay, intelligent management and scheduling, and other comprehensive capabilities, involving network transmission within and between data centers, and is a comprehensive index for measuring network transmission scheduling capability.

[0044] The ''storage power'' (Storage Power, SP) described in the application refers to: the comprehensive ability of data center in data storage capacity, performance, safety and reliability, and green low carbon, which is a comprehensive index for measuring the data storage capacity of data center, including external storage devices such as storage arrays and built-in storage devices of servers. The commonly used unit of measurement for storage capacity is exabyte (EB, 1EB=2^60bytes), the commonly used unit of measurement for performance is the number of read / write per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB), and the disaster recovery ratio is an important performance of safety and reliability.

[0045] The ''computing power infrastructure'' described in the application refers to: a new type of information infrastructure integrating information computing power, network carrying capacity and data storage power, which can realize centralized computing, storage, transmission and application of information.

[0046] The "new information infrastructure" described in the application refers to: mainly including 5G network, fiber broadband network, backbone network, international communication network, satellite Internet and other network infrastructure, data center, general computing power center, intelligent computing center, supercomputing center and other computing power infrastructure, artificial intelligence, blockchain, quantum computing and other new technology facilities.

[0047] The "computing power" described in the application includes: general computing power, intelligent computing power and supercomputing power.

[0048] The "general computing power" described in the application refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0049] The "intelligent computing power" described in the application refers to: for various types of artificial intelligence innovation applications, based on GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) and other special chip large-scale deployment of computing platform, such as natural language processing, machine vision, etc.

[0050] The "supercomputing power" described in the application refers to: the computing power provided by supercomputers and other high-performance computing clusters, which utilizes the centralized computing resources of multiple computer systems working in parallel and processes extremely complex or data-intensive problems through a dedicated operating system, mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, gene analysis, etc.

[0051] The "intelligent computing center" described in the application refers to: by using large-scale heterogeneous computing power resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), mainly for artificial intelligence applications (such as artificial intelligence deep learning model development, model training and model inference, etc. Scene) to provide the required computing power, data and algorithm facilities. Intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from bottom computing power to top application enablement.

[0052] The "intelligent computing center cloud platform" described in the application is referred to as "intelligent computing cloud", which refers to: a cloud computing platform based on the comprehensive services of hardware resources and software resources of intelligent computing center.

[0053] The "intelligent computing center" described in the application includes but is not limited to "intelligent computing center".

[0054] The "intelligence calculation center" of the application is an artificial intelligence calculation center based on artificial intelligence theory, adopts an artificial intelligence calculation architecture, and provides a kind of computing power infrastructure required for artificial intelligence application, data service and algorithm service.

[0055] The "computing power center" of the application refers to a facility mainly composed of infrastructure such as wind, fire, water and electricity and IT hardware and software devices, with computing power, carrying capacity and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0056] The "supercomputing center" of the application refers to a supercomputing data center based on supercomputers or large-scale computing clusters, which can provide large-scale computing, storage and network services, and is widely used in aerospace, national defense, oil exploration, climate modeling and genome sequencing.

[0057] The "computing power resource" of the application refers to the technology and facilities required for the development of digital society, including but not limited to CPU, GPU, etc. computing resource switch, router, etc. network resource, storage array, distributed storage, etc. storage resource, firewall, intrusion detection system, etc. security resource, and wind, fire, water and electricity supporting resources.

[0058] The "model" of the application includes but is not limited to "large language model" and "multimodal large model". The "large model" of the application can be understood as the above-mentioned "model".

[0059] The "large language model" of the application refers to a large-scale language model (LLM), which is a language model with a large number of parameters, aiming to understand and generate human language, and is trained through a large amount of text data, and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0060] The "multimodal large model" of the application refers to a model that jointly trains text, image, video, audio and other multimodal information, including but not limited to multimodal large language model.

[0061] The "computing power usage state" of the application refers to the usage state of computing power in the intelligent computing center cloud platform during the training of large models.

[0062] Please refer to Figure 1 , Figure 1 is a flowchart of the method for adjusting the training task of a large model based on the computing power usage state of the intelligent computing center cloud platform provided by the application, as shown in Figure 1As shown, comprising the following steps:

[0063] Step S1: Obtain a computing power usage state, the computing power usage state is used to represent the usage state of computing power in the intelligent computing center cloud platform in the process of training a large model.

[0064] Wherein, the specific types of state parameters included in the computing power usage state are not limited here, and optionally, the computing power usage state includes at least one of the following: GPU core utilization, memory and GPU exchange data bandwidth, video memory occupancy rate, video memory throughput and throughput between different GPUs. The above-mentioned GPU core utilization, memory and GPU exchange data bandwidth, video memory occupancy rate, video memory throughput and throughput between different GPUs can be referred to as state parameters included in the computing power usage state, so as to increase the diversity and flexibility of the types of state parameters included in the computing power usage state.

[0065] It should be noted that each type of state parameter included in the computing power usage state has a corresponding expected state, i.e., the state parameter and the expected state can be one-to-one.

[0066] In addition, optionally, the state parameter, the training parameter and the expected state can also be one-to-one, and the types of training parameters are not limited here, for example: the training parameters can include at least one of the following: training time period, training process, training sample, training method, i.e. when the training time period is different, the training process is different, the training sample input into the large model is different, and the training method for the large model is different, the corresponding expected state can also be different.

[0067] For example: when the state parameter included in the computing power usage state is GPU core utilization, and the training parameter includes training time period, the expected state corresponding to the GPU core utilization in the first time period can be greater than 70%, and the expected state corresponding to the GPU core utilization in the second time period can be greater than 80%, it can be seen that when the state parameter included in the computing power usage state is the same, the training parameter is different, and the corresponding expected state is also different.

[0068] It should be noted that the more types of state parameters included in the computing power usage state, the more accurate the result of determining whether the computing power usage state matches the expected state.

[0069] Step S2: Determine whether the computing power usage state matches the expected state.

[0070] When the state parameter included in the computing power usage state is one kind, the state parameter included in the computing power usage state can be directly matched with the corresponding expected state; when the state parameter included in the computing power usage state is multiple kinds, each kind of state parameter can be matched with the corresponding expected state, and when all the state parameters match the corresponding expected states, it can be determined that the computing power usage state matches the expected state; when at least one state parameter does not match the corresponding expected state, it can be determined that the computing power usage state does not match the expected state.

[0071] Wherein, matching each kind of state parameter with the corresponding expected state can be understood as: calculating the difference value of the state parameter and the corresponding expected state, if the difference value is within the preset range, it can be determined that the state parameter matches the corresponding expected state, if the difference value is not within the preset range, it can be determined that the state parameter does not match the corresponding expected state.

[0072] For example: the state parameter can be the utilization rate of the computing card core, the expected state is that the utilization rate is 70%, if the utilization rate of the computing card core is 69.5%, and the preset range is less than 1%, then 70%-69.5%=0.5%, which is obviously less than 1%, so it can be determined that the utilization rate of the computing card core of 69.5% matches the expected state; if the utilization rate of the computing card core is 68.5%, and the preset range is less than 1%, then 70%-68.5%=1.5%, which is obviously greater than 1%, so it can be determined that the utilization rate of the computing card core of 69.5% does not match the expected state.

[0073] Step S3: in the case that the computing power usage state does not match the expected state, adjusting the training task of the large model according to the computing power usage state.

[0074] In the case that the computing power usage state does not match the expected state, it indicates that the computing power usage state is bottlenecked at this time, and the training task of the large model needs to be adjusted, so as to improve the utilization rate of computing power, reduce the waste of computing power, and improve the training efficiency of the large model.

[0075] In the present application, by steps S1 to S3, the computing power usage state is obtained, which is used to represent the usage state of computing power of the intelligent computing center cloud platform in the process of training the large model; it is determined whether the computing power usage state matches the expected state; in the case that the computing power usage state does not match the expected state, the training task of the large model is adjusted according to the computing power usage state. In this way, when it is determined that the computing power usage state does not match the expected state, the training task of the large model can be adjusted according to the computing power usage state, thereby reducing the phenomenon of waste of computing power, and improving the training efficiency of the large model.

[0076] It should be noted that the types of state parameters included in the computing power usage state are different, and the adjustment mode and adjustment parameters of the large model training task are also different.

[0077] Optionally, the computing power usage state includes a computing card core utilization rate, and the step S3 includes:

[0078] Step S31: In the case where the fluctuation rate of the computing card core utilization rate in the preset period is greater than the expected fluctuation rate, it is determined that the computing power usage state does not match the expected state.

[0079] Step S32: In the case where the computing power usage state does not match the expected state, the storage level of training data in the training task is adjusted according to the fluctuation state of the computing card core utilization rate, and the fluctuation state includes the fluctuation rate.

[0080] The fluctuation rate of the computing card core utilization rate in the preset period can be obtained according to the difference between the maximum utilization rate and the minimum utilization rate of the computing card in the preset period. In addition, the fluctuation state of the computing card core utilization rate can include the fluctuation rate of the computing card core utilization rate in the preset period, and the fluctuation state of the computing card core utilization rate can also include the fluctuation duration, the fluctuation time period, etc.

[0081] In the present application, when the fluctuation rate of the computing card core utilization rate in the preset period is greater than the expected fluctuation rate, it indicates that there is a problem with the data reading speed of the computing card, and therefore the storage level of training data in the training task can be adjusted according to the fluctuation state of the computing card core utilization rate to improve the data reading speed. The storage level of training data in the training task can be understood as the priority of storage and reading, and the higher the storage level, the higher the priority of reading and storing training data.

[0082] In addition, optionally, when the fluctuation rate of the computing card core utilization rate in the preset period is greater than the expected fluctuation rate, it can also indicate that there is a problem with the computing power allocation of the computing card, and therefore the computing power allocated by the computing card can be adjusted. By adjusting the computing power allocated by the computing card, the fluctuation rate of the computing card core utilization rate is reduced, so that the computing card core utilization rate always reaches the expectation, i.e. the waste of computing power is reduced, and at the same time the training efficiency of the large model can be improved.

[0083] Optionally, the computing power usage state includes the throughput between different computing cards, and the step S3 includes:

[0084] Step S33: In the case where the throughput between different computing cards is greater than a first throughput, it is determined that the computing power usage state does not match the expected state.

[0085] Step S34: in the case that the computing power usage state does not match the expected state, adjusting the batch sample quantity of a single computing card according to the throughput between different computing cards.

[0086] The batch sample quantity of a single computing card can be referred to as batchsize, and the total quantity of samples can be understood as the quantity of samples input into the large model. Since the training task of the large model is executed on the intelligent computing center cloud platform, and the computing power of the intelligent computing center cloud platform can include multiple computing cards, the training task of the large model can be split into multiple subtasks, and each computing card can execute part of the subtasks. Thus, each computing card can input the samples corresponding to the subtasks to be executed (the quantity of the samples can be referred to as the batch sample quantity of a single computing card) to realize the training of the subtasks.

[0087] The throughput between different computing cards can be understood as the data communication quantity between different computing cards.

[0088] In the present application, when the throughput between different computing cards is greater than the first throughput, it can be understood that the communication between different computing cards is too frequent, that is, the quantity of samples allocated on the computing card can be unreasonable, resulting in the need to frequently communicate with other computing cards. Therefore, the batch sample quantity of a single computing card can be adjusted according to the throughput between different computing cards to improve the utilization rate of the computing power of each computing card, thereby enhancing the training efficiency of the subtasks and improving the training efficiency of the large model.

[0089] Optionally, the computing power usage state includes a GPU throughput, and the step S3 includes:

[0090] Step S35: in the case that the GPU throughput is greater than a second throughput, determining that the computing power usage state does not match the expected state.

[0091] Step S36: in the case that the computing power usage state does not match the expected state, adjusting the parallelism of the computing power according to the GPU throughput.

[0092] In the present application, the GPU throughput can be understood as the data communication quantity between the computing card and the memory. When the GPU throughput is greater than the second throughput, it means that the load of the computing card has reached the upper limit at this time. At this time, the parallelism of the computing power can be adjusted, such as increasing the quantity of computing cards included in the computing power, so as to further improve the training efficiency of the large model.

[0093] Optionally, the computing power usage state includes a bandwidth of data exchange between the memory and the computing card and a GPU occupancy rate, and the step S3 includes:

[0094] Step S37: In the case that the GPU memory occupancy rate does not reach the preset occupancy rate and the bandwidth of data exchange between the memory and the computing card reaches the preset bandwidth, it is determined that the computing power usage state does not match the expected state.

[0095] Step S38: Adjusting the proportion of model weight data unloaded into the memory according to the bandwidth of data exchange between the memory and the computing card and the GPU memory occupancy rate adjustment model weight data.

[0096] Wherein, the GPU memory occupancy rate can be understood as the usage rate of the computing card and the memory, and the bandwidth of data exchange between the GPU memory and the computing card can be understood as the bandwidth for data exchange between the memory and the computing card, and the above-mentioned data exchange can also be understood as data communication.

[0097] Wherein, the model weight data can also be referred to as the model weight coefficient or the model weight, and the model weight data can represent the parameters of neuron connection in each large model. These weights are continuously adjusted in the training process of the large model, so that the large model can more accurately predict the output. And the model weight data determines how the input data of the large model is processed and converted through the model.

[0098] In the present application, when the GPU memory occupancy rate does not reach the preset occupancy rate and the bandwidth of data exchange between the memory and the computing card reaches the preset bandwidth, it indicates that the data exchange between the memory and the computing card is too frequent at this time, and therefore the proportion of model weight data unloaded into the memory can be reduced to reduce the communication between the computing card and the memory, thereby improving the training efficiency of the sub-tasks allocated on the computing card, and further improving the training efficiency of the large model.

[0099] It should be noted that the above-mentioned bandwidth of data exchange between the memory and the computing card can be calculated by the GPU memory occupancy rate or the amount of communication between the computing card and the memory, and when the GPU memory occupancy rate or the amount of communication between the computing card and the memory is larger, the bandwidth of data exchange between the memory and the computing card is larger, and the three can be linearly related.

[0100] Wherein, the computing power includes types which are not specifically limited here, and optionally, the computing power can include the corresponding computing power of the central processing unit (CPU), the graphics processing unit (GPU) and the memory.

[0101] It should be noted that the above-mentioned adjusting the storage level of the training data in the training task, adjusting the batch sample quantity of a single computing card, adjusting the parallel degree of the computing power, and adjusting the proportion of model weight data unloaded into the memory are all for improving the training efficiency of the large model training task.

[0102] It should be noted that the process of the large model training task can include the following steps:

[0103] 1. Set training parameters, which can include the learning rate of the large model and the number of samples;

[0104] 2. Prepare training samples, which can be referred to as samples, training data or dataloader;

[0105] 3. Forward propagation of the large model, which can be understood as inputting the sample into the large model for iterative training, and the process of the large model outputting the prediction result;

[0106] 4. Calculate the loss, input the prediction result output by the large model and the actual label of the sample into the loss function to calculate the loss, and the specific type of the loss function is not limited here, optionally, the loss function can include mean square error function, cross entropy function, etc. ;

[0107] 5. Back propagation, calculate the gradient of the loss function with respect to the parameters of the large model, and update the model parameters according to the gradient;

[0108] 6. Verification and adjustment, after the training of each batch of samples on the large model is completed, the performance of the large model can be evaluated using the validation set of the sample, if the performance of the large model does not improve or starts to decline during the evaluation process of the validation set, it is possible that the large model is overfitting, at this time, the hyperparameters of the large model can be adjusted, or the regularization technique is used to adjust the large model, and the regularization technique can include: dropout, L1 / L2 regularization or early stopping method, etc.

[0109] Referring to Figure 2 , Figure 2 the training method of the large model is shown in FIG. 1, which can include the following steps: Figure 2

[0110] Step S1, load training data; the training data can be understood as samples or sample data;

[0111] Step S2, calculate the loss; the loss can be calculated by using the loss function, and the specific calculation method can be referred to the above description;

[0112] Step S3, subsequent calculation, which can include the calculation process in the back propagation, verification and adjustment.

[0113] Referring to Figure 3 , Figure 3 the structure of the device for adjusting the large model training task of the intelligent computing center cloud platform based on the computing power usage state provided by the present application is shown in FIG. 2, which can include the following steps: Figure 3 ​As shown, the device 300 for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform comprises:

[0114] An acquisition module 301 is configured to acquire a computing power usage state, which is used to represent a usage state of computing power of the intelligent computing center cloud platform in a process of training a large model.

[0115] A determination module 302 is configured to determine whether the computing power usage state matches an expected state.

[0116] An adjustment module 303 is configured to adjust a training task of the large model according to the computing power usage state in a case where the computing power usage state does not match the expected state.

[0117] Optionally, the computing power usage state comprises at least one of a computing card core utilization rate, a bandwidth of data exchanged between memory and a computing card, a video memory throughput, and a throughput between different computing cards.

[0118] Optionally, the computing power usage state comprises a computing card core utilization rate, and the adjustment module 303 comprises:

[0119] A first calculation sub-module is configured to determine that the computing power usage state does not match the expected state in a case where a fluctuation rate of the computing card core utilization rate in a preset period is greater than an expected fluctuation rate.

[0120] A first adjustment sub-module is configured to adjust a storage level of training data in the training task according to a fluctuation state of the computing card core utilization rate in a case where the computing power usage state does not match the expected state, the fluctuation state comprising the fluctuation rate.

[0121] Optionally, the computing power usage state comprises a throughput between different computing cards, and the adjustment module 303 comprises:

[0122] A second determination sub-module is configured to determine that the computing power usage state does not match the expected state in a case where the throughput between different computing cards is greater than a first throughput.

[0123] A second adjustment sub-module is configured to adjust a batch sample quantity of a single computing card according to the throughput between different computing cards in a case where the computing power usage state does not match the expected state.

[0124] Optionally, the computing power usage state comprises a video memory throughput, and the adjustment module 303 comprises:

[0125] a third determining sub-module, configured to determine that the computing power usage state does not match the expected state when the calculated GPU throughput is greater than the second throughput;

[0126] a third adjusting sub-module, configured to adjust the parallelism of the computing power according to the GPU throughput when the computing power usage state does not match the expected state.

[0127] Optionally, the computing power usage state comprises a bandwidth of data exchange between the memory and the computing card and a GPU occupancy rate, and the adjusting module 303 comprises:

[0128] a fourth determining sub-module, configured to determine that the computing power usage state does not match the expected state when the GPU occupancy rate does not reach a preset occupancy rate and the bandwidth of data exchange between the memory and the computing card reaches a preset bandwidth.

[0129] a fourth adjusting sub-module, configured to adjust a proportion of model weight data unloaded into the memory according to the bandwidth of data exchange between the memory and the computing card and the GPU occupancy rate.

[0130] The device 300 for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform provided by the application can perform each step in the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform, and thus has the same beneficial technical effects as the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform, which will not be described here in detail.

[0131] For reference Figure 4 The application further provides an electronic device 40 comprising a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor 41, which, when executed by the processor 41, implements each process shown in the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform and achieves the same technical effects, which will not be described here in detail to avoid repetition.

[0132] The application further provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements each process of the method for adjusting a large model training task based on a computing power usage state of an intelligent computing center cloud platform and achieves the same technical effects, which will not be described here in detail to avoid repetition. The computer readable storage medium may, for example, be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] The application further provides a computer program product comprising computer instructions which, when executed by a processor, implement the above method. Figure 1 The intelligent computing center cloud platform shown in the method for adjusting a large model training task based on a computing power use state adjusts each process of the method and can achieve the same technical effects, and thus, to avoid repetition, details are not described herein.

[0134] It should be noted that, in this document, the terms “comprises”, “comprising”, or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase “comprising a” does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0135] From the above description of the embodiments, those skilled in the art can clearly understand that the method provided by the application can be implemented by means of software and a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the various methods provided by the application.

[0136] The application has been described above in conjunction with the accompanying drawings, but the application is not limited to the specific embodiments described above, which are merely illustrative rather than limiting, and those of ordinary skill in the art can make many forms under the inspiration of the application without departing from the purpose of the application and the scope protected by the claims, all of which belong to the protection of the application.

Claims

1. A method for adjusting large model training tasks based on computing power usage status on an intelligent computing center cloud platform, characterized in that: include: Step S1: Obtain computing power usage status, where the computing power usage status is used to indicate the usage status of computing power in the intelligent computing center cloud platform by the training task during the training of the large model. Step S2: Determine whether the computing power usage status matches the expected status; Step S3: When the computing power usage status does not match the expected status, adjusting the training task of the large model according to the computing power usage status; The computing power usage status includes at least one of the following: computing card core utilization, bandwidth for exchanging data between memory and computing card, video memory occupancy, video memory throughput, and throughput between different computing cards; The computing power usage status includes computing card core utilization, and step S3 includes: Step S31: when it is calculated that the fluctuation rate of the computing card core utilization rate within a preset period is greater than the expected fluctuation rate, determining that the computing power usage state does not match the expected state; Step S32: When the computing power usage state does not match the expected state, the storage level of the training data in the training task is adjusted according to the fluctuation state of the computing card core utilization, where the fluctuation state includes the fluctuation rate.

2. The method according to claim 1, characterized in that The computing power usage status includes the throughput between different computing cards, and step S3 includes: Step S33: When the throughput between different computing cards is calculated to be greater than the first throughput, determining that the computing power usage state does not match the expected state; Step S34: When the computing power usage status does not match the expected status, the number of batch samples of a single computing card is adjusted according to the throughput between the different computing cards.

3. The method according to claim 1, characterized in that The computing power usage status includes video memory throughput, and step S3 includes: Step S35: when the calculated video memory throughput is greater than the second throughput, determining that the computing power usage state does not match the expected state; Step S36: When the computing power usage state does not match the expected state, adjust the parallelism of the computing power according to the video memory throughput.

4. The method according to claim 1, wherein The computing power usage status includes the bandwidth for exchanging data between the memory and the computing card and the video memory occupancy rate. Step S3 includes: Step S37: if the video memory occupancy rate does not reach the preset occupancy rate and the bandwidth for exchanging data between the memory and the computing card reaches the preset bandwidth, determining that the computing power usage state does not match the expected state; Step S38: adjusting the proportion of the model weight data unloaded into the memory according to the bandwidth of data exchange between the memory and the computing card and the memory occupancy rate.

5. A device for adjusting large model training tasks based on computing power usage status on an intelligent computing center cloud platform, characterized in that: include: An acquisition module is used to obtain the computing power usage status, where the computing power usage status is used to indicate the usage status of the computing power in the intelligent computing center cloud platform by the training task during the training of the large model; a determination module, configured to determine whether the computing power usage status matches the expected status; an adjustment module, configured to adjust the training task of the large model according to the computing power usage status when the computing power usage status does not match the expected status; The computing power usage status includes at least one of the following: computing card core utilization, bandwidth for exchanging data between memory and computing card, video memory occupancy, video memory throughput, and throughput between different computing cards; The computing power usage status includes the computing card core utilization, and the adjustment module includes: a first calculation submodule, configured to determine that the computing power usage state does not match the expected state when a fluctuation rate of the computing card core utilization rate within a preset period is calculated to be greater than an expected fluctuation rate; The first adjustment submodule is used to adjust the storage level of the training data in the training task according to the fluctuation state of the computing card core utilization when the computing power usage state does not match the expected state, and the fluctuation state includes the fluctuation rate.

6. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for adjusting large model training tasks based on computing power usage status of an intelligent computing center cloud platform as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for adjusting large model training tasks based on computing power usage status of an intelligent computing center cloud platform as described in any one of claims 1 to 4.

8. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the method for adjusting large model training tasks based on computing power usage status of an intelligent computing center cloud platform as described in any one of claims 1 to 4.

Images