Method and device for intelligent computing center cloud platform to adjust model training parameters according to computing power operation state

Abstract

The invention provides a method and device for an intelligent computing center cloud platform to adjust model training parameters according to computing power operation states, and relates to the technical field of intelligent computing centers, intelligent computing centers and computing power infrastructures. The first working component is a component deployed in a node in the intelligent computing center cloud platform, the target link comprises a data preparation link, a model training link, a model evaluation link and / or a model test link in the first working component, and the target link performs data processing or model training based on preset hyper-parameters; s2, calculating target correction parameters corresponding to the operation related parameters in a preset period based on a preset regulation and control model; and S3, adjusting the preset hyper-parameter based on the target correction parameter to obtain a target hyper-parameter. The computing power utilization rate of the intelligent computing center cloud platform can be greatly improved, and then the model training efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent computing centers, smart computing centers and computing power infrastructure, and specifically to a method for an intelligent computing center cloud platform to adjust model training parameters according to the computing power operation status. Background Art

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "intelligent computing centers" have emerged.

[0003] An "Intelligent Computing Center" is a facility that uses large-scale heterogeneous computing resources, including general-purpose and intelligent computing power, to provide the computing power, data, and algorithms required for AI applications (such as AI deep learning model development, model training, and model inference). The Intelligent Computing Center encompasses facilities, hardware, and software, and provides a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0004] “Intelligent Computing Center” includes but is not limited to “Smart Computing Center”.

[0005] "Intelligent Computing Center" refers to an artificial intelligence computing center. It is a type of computing power infrastructure that is based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services, and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing center" and "intelligent computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to perform certain computing needs. It is the computing power to achieve target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In intelligent computing center cloud platforms, computing power resources are typically provided by accelerator cards for model training. In existing technologies, model training typically involves multiple rounds of training. Each round requires training data preparation, which is then used for model training. The effectiveness of model training is then determined through model evaluation or testing. However, in existing technologies, many parameters in data preparation, model training, model evaluation, and model testing are manually pre-set. This can lead to low model training efficiency during actual model training due to improper parameter settings.

[0008] It can be seen that the existing technology has the problem of low training efficiency of the model of the intelligent computing center cloud platform. Summary of the Invention

[0009] The present invention provides a method and device for adjusting model training parameters of an intelligent computing center cloud platform according to the computing power operation status, so as to solve the problems in the prior art of low computing power resource utilization of the intelligent computing center cloud platform and low model training efficiency.

[0010] To solve the above problems, the present invention is achieved as follows:

[0011] In a first aspect, the present invention provides a method for adjusting model training parameters of an intelligent computing center cloud platform according to computing power operation status, comprising:

[0012] Step S1: monitoring operation-related parameters of a target link in a first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform, the first working component is used to perform model training on a first model, the target link includes a data preparation link, a model training link, a model evaluation link, and / or a model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters;

[0013] Step S2: calculating target correction parameters corresponding to the operation-related parameters within a preset period based on a preset control model;

[0014] Step S3: Adjust the preset hyperparameter based on the target correction parameter to obtain the target hyperparameter.

[0015] In one embodiment, the operation-related parameters include computing power resource-related parameters and / or time-related parameters, and the preset control model includes a computing power resource control sub-model and / or a time control sub-model;

[0016] The step S2 comprises:

[0017] Step S21: Calculate the computing power resource related parameters within the preset period based on the computing power resource control sub-model to obtain a first intermediate correction parameter, where the first intermediate correction parameter is used to improve computing power resource utilization; and / or

[0018] Step S22: calculating the time-related parameter within the preset period based on the time control sub-model to obtain a second intermediate correction parameter, where the second intermediate correction parameter is used to reduce the waiting time;

[0019] Step S23: Generate the target correction parameter based on the first intermediate correction parameter and / or the second intermediate correction parameter.

[0020] In one embodiment, when the operation-related parameters include the computing power resource-related parameters and the time-related parameters, and the preset control model includes the computing power resource control sub-model and the time control sub-model, step S23 includes:

[0021] Step S231: Weight the first intermediate correction parameter and the second intermediate correction parameter to obtain the target correction parameter.

[0022] In one embodiment, the time-related parameters include at least one of a first time parameter for reading the original data, a second time parameter for preprocessing the original data, a third time parameter for the preprocessed target data to wait in a data cache pool, or a fourth time parameter for backpropagation to read the target data.

[0023] In one embodiment, the operation-related parameters include training convergence-related parameters, the preset control model includes a model convergence control sub-model, and step S2 includes:

[0024] Step S24: Calculate the training convergence-related parameters within the preset period based on the model convergence control sub-model to obtain the target correction parameters, which are used to improve the model convergence speed.

[0025] In one embodiment, the training convergence related parameters include at least one of a learning rate, a training accuracy, a validation accuracy, a batch size, a gradient accumulation step number, a training loss, and a validation loss;

[0026] The target correction parameter includes at least one of a learning rate correction parameter, a batch size correction parameter, and a gradient accumulation step number correction parameter.

[0027] In one embodiment, when the target link is the data preparation link, the preset hyperparameter includes at least one of a data batch size, a number of data preparation processes, and a data cache pool size; and / or,

[0028] In the case where the target link is the model training link, the preset hyperparameter includes at least one of a data reading batch size and a learning rate; and / or,

[0029] In the case where the target link is the model evaluation link, the preset hyperparameter includes a model evaluation frequency; and / or,

[0030] When the target link is the model testing link, the preset hyperparameters include model testing frequency.

[0031] In one embodiment, the preset control model is obtained by:

[0032] Acquiring sample parameters, wherein the sample parameters include running sample parameters and correcting sample parameters;

[0033] An initial reinforcement learning algorithm is trained based on the sample parameters to obtain the preset control model.

[0034] In one embodiment, the first working component is a component included in a target node in the intelligent computing center cloud platform, and the target node includes multiple components. Before step S1, the method further includes:

[0035] Step S0: split the initial model into multiple sub-models, and assign the multiple sub-models to different components, wherein the multiple sub-models include the first model.

[0036] In a second aspect, the present invention further provides a device for adjusting model training parameters of an intelligent computing center cloud platform according to computing power operation status, comprising:

[0037] a monitoring module, configured to monitor operation-related parameters of a target link in a first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform, the first working component is configured to perform model training on a first model, the target link includes a data preparation link, a model training link, a model evaluation link, and / or a model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters;

[0038] a generating module, configured to generate a target correction parameter based on the operation-related parameter when the time-related parameter satisfies a preset condition;

[0039] An adjustment module is used to adjust the preset hyperparameter based on the target correction parameter to obtain the target hyperparameter.

[0040] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps in the method for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform as described in the first aspect above are implemented.

[0041] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method of adjusting model training parameters of the intelligent computing center cloud platform according to the computing power operation status as described in the first aspect above are implemented.

[0042] In a fifth aspect, the present invention also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the method of adjusting model training parameters of the intelligent computing center cloud platform according to the computing power operation status as described in the first aspect above.

[0043] In the present invention, the operation-related parameters of the target link in the first working component are monitored. The first working component is a component deployed in a node in the intelligent computing center cloud platform. The first working component is used to perform model training on the first model. The target link includes the data preparation link, model training link, model evaluation link and / or model testing link in the first working component. The target link performs data processing or model training based on preset hyperparameters; calculates the target correction parameters corresponding to the operation-related parameters within a preset period based on the preset control model; and adjusts the preset hyperparameters based on the target correction parameters to obtain target hyperparameters. In this way, the preset control model calculates the target correction parameters corresponding to the operation-related parameters within a preset period, and adjusts the preset hyperparameters through the target correction parameters to obtain target hyperparameters. The computing power resource utilization rate corresponding to the target hyperparameters is higher than the computing power resource utilization rate corresponding to the preset hyperparameters. The target link performs data processing or model training through the target hyperparameters, which can effectively improve the computing power utilization rate, thereby greatly improving the efficiency of model training. Furthermore, the preset control model can calculate the target hyperparameters from multiple aspects such as computing power resource utilization, waiting time and model convergence speed, thereby realizing the control of computing power resource utilization and model convergence speed, and thus improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 This is a flow chart of a method provided by the present invention for adjusting model training parameters of an intelligent computing center cloud platform according to computing power operation status;

[0046] Figure 2 is a schematic diagram of a first working assembly provided by the present invention;

[0047] Figure 3 is a schematic diagram of a target node provided by the present invention;

[0048] Figure 4 This is a structural diagram of a device for adjusting model training parameters according to the computing power operation status of an intelligent computing center cloud platform provided by the present invention;

[0049] Figure 5 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] The "computing power" mentioned in the present invention refers to: the ability of computer equipment or computing / data centers to process information, the ability of computer hardware and software to work together to execute certain computing requirements, and the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, and mainly provides services to society through computing power infrastructure.

[0052] The "computing power" (Computational Power, CP) mentioned in the present invention refers to: the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating-point operations performed per second (FLOPS, 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 .

[0053] The "carrying capacity" (Network Power, NP) mentioned in the present invention refers to: it is the performance of the data transmission capability of the computing power facility, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. It involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0054] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, security and reliability, and environmental friendliness. It is a comprehensive indicator for measuring a data center's data storage capacity, encompassing both external storage devices such as storage arrays and internal server storage. Storage capacity is commonly measured in exabytes (EB, 1EB = 2^60 bytes), while performance is commonly measured in IOPS / TB (Input / Output Operations Per Second / TB). Disaster recovery ratio is a key indicator of security and reliability.

[0055] The "computing power infrastructure" mentioned in the present invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, and can realize the centralized calculation, storage, transmission and application of information.

[0056] The "new information infrastructure" mentioned in the present invention refers to: mainly including network infrastructure such as 5G networks, fiber-optic broadband networks, backbone networks, international communication networks, satellite Internet, computing power infrastructure such as data centers, general computing power centers, intelligent computing centers, supercomputing centers, and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0057] The "computing power" mentioned in the present invention includes: general computing power, intelligent computing power and super computing power.

[0058] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0059] The "intelligent computing power" mentioned in this invention refers to: a computing platform based on specialized chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing and machine vision.

[0060] The "supercomputing power" mentioned in the present invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, genetic analysis, etc.

[0061] The "intelligent computing center" described in this article refers to a facility that provides the computing power, data, and algorithms required for artificial intelligence applications (such as AI deep learning model development, model training, and model inference) by utilizing large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center encompasses facilities, hardware, and software, and can provide a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0062] The "intelligent computing center cloud platform" mentioned in the present invention refers to: a cloud computing platform that provides comprehensive services based on the hardware resources and software resources of the intelligent computing center.

[0063] The "intelligent computing center" mentioned in the present invention includes but is not limited to the "intelligent computing center".

[0064] The "intelligent computing center" mentioned in the present invention is an artificial intelligence computing center, which is a type of computing power infrastructure based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services and algorithm services required for artificial intelligence applications.

[0065] The "computing power center" mentioned in the present invention refers to: a facility that is mainly composed of infrastructure such as wind, fire, water, electricity, and IT hardware and software equipment, and has computing power, transportation capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0066] The "supercomputing center" mentioned in the present invention refers to: a supercomputing data center, which is a data center based on a supercomputer or a large-scale computing cluster, which can provide large-scale computing, storage and network services and other functions, and is widely used in application scenarios such as aerospace, national defense, oil exploration, climate modeling and genome sequencing.

[0067] The "computing resources" mentioned in the present invention refer to: technologies and facilities with information computing, transmission, storage and application capabilities required for the development of a digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water and electricity.

[0068] The “model” mentioned in the present invention includes but is not limited to a “large language model” and a “large multimodal model”.

[0069] The "large language model" mentioned in the present invention refers to a large language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0070] The "Multimodal Large Models" mentioned in the present invention refer to models that combine multimodal information such as text, images, video, and audio for training, including but not limited to multimodal large language models.

[0071] See Figure 1 , Figure 1 This is a flow chart of a method provided by the present invention for adjusting model training parameters of an intelligent computing center cloud platform according to the computing power operation status. Figure 1 As shown, the following steps are included:

[0072] Step S1: Monitor the operation-related parameters of the target link in the first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform. The first working component is used to perform model training on the first model. The target link includes the data preparation link, model training link, model evaluation link and / or model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters.

[0073] The aforementioned run-related parameters are parameters used by different stages in data processing. These run-related parameters are the parameters displayed when the target stage processes data, such as GPU utilization and CPU utilization. It should be noted that when the target stage processes data based on different preset hyperparameter values, the corresponding run-related parameters will be different. The run-related parameters can be used to determine how to adjust the preset hyperparameters, thereby enabling adjustments to the preset hyperparameters.

[0074] The above preset hyperparameters are the parameters used as reference when processing data in the target stage. For example, the data preparation stage processes the original data according to the preset batch size, and each batch of the processed training data meets the preset batch size.

[0075] In some embodiments, as Figure 2 As shown, the first working component includes multiple steps, including data preparation, model training, model evaluation, and model testing. Data preparation is implemented through different workflows. During the monitoring process, each step of the first working component is monitored to obtain time-related and operation-related parameters.

[0076] Step S2: Calculate target correction parameters corresponding to the operation-related parameters within a preset period based on a preset control model.

[0077] The above-mentioned preset control model is used to modify the preset hyperparameters to reduce the waiting time and waste of computing resources during model training, thereby improving the utilization of computing resources. It should be noted that during the model training process, the target link performs data processing and model training based on the preset hyperparameters. However, there are several situations in which data processing and model training based on the preset hyperparameters result in low computing resource utilization: First, waiting occurs during the model training process, resulting in unused computing resources and low computing resource utilization; second, computing resources are not fully utilized during the model training process, and some computing resources are left unused, resulting in low computing resource utilization.

[0078] In the present invention, the preset hyperparameters are corrected through a preset control model, so that the target link can use more computing resources based on the corrected target hyperparameters, thereby improving the utilization rate of computing resources.

[0079] The target correction parameters are used to adjust preset hyperparameters. They are determined based on the values ​​of the runtime-related parameters. For example, if model training is experiencing wait times and the GPU utilization in the runtime-related parameters is low, you can increase the batch size to allow the model training phase to read more training data per batch, thus reducing wait times.

[0080] In some optional implementations, an adjustment strategy can be pre-set for specific waiting situations, and the target correction parameter can be determined based on the adjustment strategy. For example, for the waiting situation during model training, multiple parameter value ranges can be set for the operation-related parameters, with each parameter value range corresponding to a correction parameter. In this way, when the model training phase is waiting, the corresponding correction parameter is determined based on the value of the operation-related parameter.

[0081] The above preset period is the period for calculating the target correction parameters. It should be noted that model training is a long process, and the efficiency of model training is also affected by the training data. In order to continuously optimize the efficiency of model training, it is necessary to continuously calculate the target correction parameters. The target correction parameters are used to continuously adjust the preset hyperparameters, thereby effectively improving the efficiency of model training.

[0082] In some implementations, the preset period can be set according to the rounds of model training or according to the training time.

[0083] Step S3: Adjust the preset hyperparameter based on the target correction parameter to obtain the target hyperparameter.

[0084] It should be noted that processing data or training models based on target hyperparameters can reduce the waiting time of the target link compared to processing data or training models based on preset hyperparameters, avoiding the situation where the target link occupies the computing power resources of the intelligent computing center cloud platform for a long time without performing actual work, thereby effectively improving the utilization of computing power resources.

[0085] In the present invention, the operation-related parameters of the target link in the first working component are monitored. The first working component is a component deployed in a node in the intelligent computing center cloud platform. The first working component is used to perform model training on the first model. The target link includes the data preparation link, model training link, model evaluation link and / or model testing link in the first working component. The target link performs data processing or model training based on preset hyperparameters; calculates the target correction parameters corresponding to the operation-related parameters within a preset period based on the preset control model; and adjusts the preset hyperparameters based on the target correction parameters to obtain target hyperparameters. In this way, the preset control model calculates the target correction parameters corresponding to the operation-related parameters within a preset period, and adjusts the preset hyperparameters through the target correction parameters to obtain target hyperparameters. The computing power resource utilization rate corresponding to the target hyperparameters is higher than the computing power resource utilization rate corresponding to the preset hyperparameters. The target link performs data processing or model training through the target hyperparameters, which can effectively improve the computing power utilization rate, thereby greatly improving the efficiency of model training.

[0086] The operation-related parameters include computing power resource-related parameters and / or time-related parameters, and the preset control model includes a computing power resource control sub-model and / or a time control sub-model;

[0087] The step S2 comprises:

[0088] Step S21: Calculate the computing power resource related parameters within the preset period based on the computing power resource control sub-model to obtain a first intermediate correction parameter, where the first intermediate correction parameter is used to improve computing power resource utilization; and / or

[0089] Step S22: calculating the time-related parameter within the preset period based on the time control sub-model to obtain a second intermediate correction parameter, where the second intermediate correction parameter is used to reduce the waiting time;

[0090] Step S23: Generate the target correction parameter based on the first intermediate correction parameter and / or the second intermediate correction parameter.

[0091] It should be noted that during model training, there are two main situations that affect computing resource utilization. The first is that waiting occurs during model training, resulting in unused computing resources and low computing resource utilization. The second is that computing resources are not fully utilized during model training, with some computing resources remaining unused, resulting in low computing resource utilization.

[0092] To address the issue of underutilized computing resources during model training, computing resource-related parameters can be used to determine resource utilization. These parameters can include GPU utilization, GPU memory usage, GPU power consumption, GPU temperature, CPU utilization, system memory usage, and disk I / O rate. These parameters can be used to determine current computing resource usage and, in turn, determine how to modify preset hyperparameters.

[0093] Specifically, in the present invention, the computing power resource control sub-model calculates the computing power resource related parameters within a preset period to obtain a first intermediate correction parameter. The computing power resource control sub-model can perform calculations with the goal of improving the computing power resource utilization rate, so that the obtained first intermediate correction parameter can effectively improve the computing power resource utilization rate of the link for data processing or model training after correcting the preset hyperparameter.

[0094] At the same time, in order to solve the problem of waiting during model training, time-related parameters can be used to determine whether there is a waiting situation in a certain link. Among them, time-related parameters are used to reflect the time it takes for different links to process data. For example, when the target link is the data preparation link, the time-related parameters can be the time to read the original data, or the time to process the original data. It should be noted that the time-related parameters of the target link can be used to determine whether there is a link in the model training process that is waiting for other links to process data. When there is a link that is waiting for other links to process data, the link occupies the computing power resources of the intelligent computing center cloud platform but is not used. At this time, it is necessary to adjust the preset hyperparameters to improve the computing power utilization.

[0095] Specifically, in the present invention, the time-related parameters are calculated within a preset period based on the time control sub-model to obtain a second intermediate correction parameter. The time control sub-model can perform calculations with the goal of reducing the waiting time, so that the obtained second intermediate correction parameter can effectively reduce the waiting time for data processing or model training in the link after correcting the preset hyperparameter.

[0096] In some embodiments, the operation-related parameters only include computing power resource-related parameters or time-related parameters, and the preset control model only includes a computing power resource control sub-model or a time control sub-model, and the first intermediate correction parameter or the second intermediate correction parameter is directly set as the target correction parameter.

[0097] In one embodiment, when the operation-related parameters include the computing power resource-related parameters and the time-related parameters, and the preset control model includes the computing power resource control sub-model and the time control sub-model, step S23 includes:

[0098] Step S231: Weight the first intermediate correction parameter and the second intermediate correction parameter to obtain the target correction parameter.

[0099] In the present invention, the target correction parameter is obtained by weighting the first intermediate correction parameter and the second intermediate correction parameter, so that the target correction parameter can be optimized from two aspects of improving computing resource utilization and reducing waiting time, thereby further improving the efficiency of model training.

[0100] In one embodiment, the time-related parameters include at least one of a first time parameter for reading the original data, a second time parameter for preprocessing the original data, a third time parameter for the preprocessed target data to wait in a data cache pool, or a fourth time parameter for backpropagation to read the target data.

[0101] It should be noted that during the model training process of the first working component, the data preparation phase reads the original data, preprocesses the original data, and caches the preprocessed target data in the cache pool. The model training phase reads the target data in the cache pool for model training. The model evaluation phase reads the data in the cache pool for model evaluation. The model testing phase reads the data in the cache pool for model testing. Among them, waiting may occur when reading the original data, preprocessing the original data, storing the preprocessed target data in the data cache pool, and reading the target data by back propagation. Therefore, it is necessary to set corresponding time thresholds for the above processes to determine whether there is a situation of waiting too long.

[0102] Furthermore, even if both the reading and preprocessing of raw data meet the set time thresholds, there may still be waiting times. For example, if the time to read raw data is 0.1s, the time to preprocess raw data is 0.2s, and the set time thresholds are both 0.5s, neither exceeds the set time thresholds. However, the time to preprocess raw data is much longer than the time to read raw data, resulting in frequent waiting times when reading raw data. In this case, the preset hyperparameters need to be adjusted so that the time to read raw data (first time) matches the time to preprocess raw data (second time).

[0103] Similarly, even if both the preprocessed target data and the target data are read by backpropagation, there may still be a waiting period. For example, if the waiting time for the preprocessed target data in the data cache is 0.5s and the waiting time for the target data to be read by backpropagation is 0.3s, and both are set to 0.5s, then neither exceeds the set time threshold. However, the waiting time for the preprocessed target data in the data cache is much longer than the waiting time for the target data to be read by backpropagation, resulting in the preprocessed target data waiting too long in the data cache. In this case, the preset hyperparameters need to be adjusted so that the waiting time for the preprocessed target data in the data cache (the third time) matches the waiting time for the target data to be read by backpropagation (the fourth time).

[0104] In one embodiment, the operation-related parameters include training convergence-related parameters, the preset control model includes a model convergence control sub-model, and step S2 includes:

[0105] Step S24: Calculate the training convergence-related parameters within the preset period based on the model convergence control sub-model to obtain the target correction parameters, which are used to improve the model convergence speed.

[0106] It should be noted that the model training speed is not only affected by the utilization of computing resources, but also by the model convergence speed. In order to improve the efficiency of model training, the hyperparameters of model convergence are also regulated in the present invention.

[0107] Specifically, based on the model convergence control sub-model, the training convergence related parameters are calculated within the preset period to obtain the target correction parameters. The model convergence control sub-model can perform calculations with the goal of controlling the model convergence speed, so that the scheduled target correction parameters can effectively improve the model convergence speed.

[0108] In some embodiments, the training convergence-related parameters include at least one of a learning rate, training accuracy, validation accuracy, batch size, gradient accumulation steps, training loss, and validation loss;

[0109] The target correction parameter includes at least one of a learning rate correction parameter, a batch size correction parameter, and a gradient accumulation step number correction parameter.

[0110] By monitoring at least one of the learning rate, training accuracy, verification accuracy, batch size, gradient accumulation steps, training loss and verification loss, and inputting at least one of the learning rate, training accuracy, verification accuracy, batch size, gradient accumulation steps, training loss and verification loss into the model convergence control sub-model, the model convergence control sub-model can output at least one of the learning rate correction parameter, the batch size correction parameter and the gradient accumulation step correction parameter. At least one of the learning rate correction parameter, the batch size correction parameter and the gradient accumulation step correction parameter can be used to correct at least one of the learning rate, batch size and gradient accumulation steps, thereby improving the model convergence speed.

[0111] In one embodiment, when the target link is the data preparation link, the preset hyperparameter includes at least one of a data batch size and a data buffer pool size; and / or,

[0112] In the case where the target link is the model training link, the preset hyperparameter includes at least one of the number of data loading processes, the data reading batch size, and the learning rate; and / or,

[0113] In the case where the target link is the data evaluation link, the preset hyperparameter includes a model evaluation frequency; and / or,

[0114] When the target link is the data testing link, the preset hyperparameters include model testing frequency.

[0115] It should be noted that when different links perform data processing or model training, the preset hyperparameters they rely on are also different. When the time-related parameters of the target link meet the preset conditions, the preset hyperparameters of the target link need to be adjusted to reduce the waiting time of the target link and improve the utilization of computing resources.

[0116] Among them, when the target link is the data preparation link, the preset hyperparameters include at least one of the data batch size and the data cache pool size; and / or, when the target link is the model training link, the preset hyperparameters include at least one of the number of data loading processes, the data reading batch size and the learning rate; and / or, when the target link is the data evaluation link, the preset hyperparameters include the model evaluation frequency; and / or, when the target link is the data testing link, the preset hyperparameters include the model testing frequency.

[0117] In one embodiment, the preset control model is obtained by:

[0118] Acquiring sample parameters, wherein the sample parameters include running sample parameters and correcting sample parameters;

[0119] An initial reinforcement learning algorithm is trained based on the sample parameters to obtain the preset control model.

[0120] In the present invention, sample parameters are obtained, including operating sample parameters and correction sample parameters; an initial reinforcement learning algorithm is trained based on the sample parameters to obtain the preset control model. The trained preset adjustment model can predict target hyperparameters in a direction with high computing power utilization.

[0121] In some embodiments, the sample running parameters in the sample data include at least one of GPU utilization, GPU memory occupancy, GPU power consumption, GPU temperature, CPU utilization, system memory occupancy, disk I / O rate, disk I / O wait time, network I / O rate, each batch of data loading time, data loading process CPU occupancy, forward propagation time, backward propagation time, gradient update time, current batch size, current number of data loading processes, current data pre-read quantity, and current buffer pool size;

[0122] The modified sample parameters include at least one of batch size, gradient accumulation steps, data loading process number, data pre-read quantity, data cache pool size, learning rate, model saving frequency, and model evaluation frequency.

[0123] The items for running sample data correspond to items for running related parameters, and the items for correcting sample data correspond to items for target hyperparameters. In this way, by training the running sample data and correcting the sample data, a preset adjustment model is obtained, which enables the preset adjustment parameters to predict the input running related parameters and obtain the target hyperparameters. The target hyperparameters can effectively improve computing resource utilization.

[0124] In one embodiment, the first working component is a component included in a target node in the intelligent computing center cloud platform, and the target node includes multiple components. Before step S1, the method further includes:

[0125] Step S0: split the initial model into multiple sub-models, and assign the multiple sub-models to different components, wherein the multiple sub-models include the first model.

[0126] It is understandable that if Figure 3 As shown, the intelligent computing center cloud platform can have multiple nodes, and each node can have multiple work components, thereby achieving distributed training of models or training of different models. For example, the intelligent computing center cloud platform has nodes 1 and 2. Work components 1, 2, 3, and 4 are deployed in node 1, and work components 5, 6, 7, and 8 are deployed in node 2. Different work components can work together.

[0127] In the present invention, the initial model is split into multiple sub-models, and the multiple sub-models are assigned to different components. The multiple sub-models include the first model. In this way, the initial model can be distributedly trained through different working components, thereby greatly improving the efficiency of model training.

[0128] In some implementations, time-related parameters and operation-related parameters are monitored for each link in each working component in each node, thereby adjusting preset hyperparameters of different links.

[0129] See Figure 4 , Figure 4 This is a structural diagram of a device for adjusting model training parameters according to the computing power operation status of an intelligent computing center cloud platform provided by the present invention, such as Figure 4 As shown, the device 400 for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform includes:

[0130] Monitoring module 401 is configured to monitor operation-related parameters of a target link in a first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform, the first working component is configured to perform model training on a first model, the target link includes a data preparation link, a model training link, a model evaluation link, and / or a model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters;

[0131] A generating module 402 is configured to calculate target correction parameters corresponding to the operation-related parameters within a preset period based on a preset control model;

[0132] The adjustment module 403 is configured to adjust the preset hyperparameter based on the target correction parameter.

[0133] In one embodiment, the operation-related parameters include computing power resource-related parameters and / or time-related parameters, and the preset control model includes a computing power resource control sub-model and / or a time control sub-model;

[0134] The generating module 402 includes:

[0135] A first calculation unit is configured to calculate the computing power resource related parameters within the preset period based on the computing power resource control sub-model to obtain a first intermediate correction parameter, wherein the first intermediate correction parameter is used to improve the computing power resource utilization; and / or

[0136] a second calculation unit, configured to calculate the time-related parameter within the preset period based on the time control sub-model to obtain a second intermediate correction parameter, wherein the second intermediate correction parameter is used to reduce the waiting time;

[0137] A generating unit is configured to generate the target correction parameter based on the first intermediate correction parameter and / or the second intermediate correction parameter.

[0138] In one embodiment, when the operation-related parameters include the computing power resource-related parameters and the time-related parameters, and the preset control model includes the computing power resource control sub-model and the time control sub-model, the generating unit includes:

[0139] A generating subunit is configured to weight the first intermediate correction parameter and the second intermediate correction parameter to obtain the target correction parameter.

[0140] In one embodiment, the time-related parameters include at least one of a first time parameter for reading the original data, a second time parameter for preprocessing the original data, a third time parameter for the preprocessed target data to wait in a data cache pool, or a fourth time parameter for backpropagation to read the target data.

[0141] In one embodiment, the operation-related parameters include training convergence-related parameters, the preset control model includes a model convergence control sub-model, and the generation module 402 includes:

[0142] The third calculation unit is used to calculate the training convergence related parameters within the preset period based on the model convergence control sub-model to obtain the target correction parameters, and the target correction parameters are used to improve the model convergence speed.

[0143] In one embodiment, the training convergence related parameters include at least one of a learning rate, a training accuracy, a validation accuracy, a batch size, a gradient accumulation step number, a training loss, and a validation loss;

[0144] The target correction parameter includes at least one of a learning rate correction parameter, a batch size correction parameter, and a gradient accumulation step number correction parameter.

[0145] In one embodiment, when the target link is the data preparation link, the preset hyperparameter includes at least one of a data batch size, a number of data preparation processes, and a data cache pool size; and / or,

[0146] In the case where the target link is the model training link, the preset hyperparameter includes at least one of a data reading batch size and a learning rate; and / or,

[0147] In the case where the target link is the model evaluation link, the preset hyperparameter includes a model evaluation frequency; and / or,

[0148] When the target link is the model testing link, the preset hyperparameters include model testing frequency.

[0149] In one embodiment, the preset control model is obtained by:

[0150] Acquiring sample parameters, wherein the sample parameters include running sample parameters and correcting sample parameters;

[0151] An initial reinforcement learning algorithm is trained based on the sample parameters to obtain the preset control model.

[0152] In one embodiment, the device 400 for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform further includes:

[0153] An allocation module is used to split the initial model into multiple sub-models and allocate the multiple sub-models to different components, wherein the multiple sub-models include the first model.

[0154] The device provided by the present invention for adjusting the model training parameters of the intelligent computing center cloud platform according to the computing power operation status is capable of realizing the various processes of each embodiment of the method for adjusting the model training parameters of the intelligent computing center cloud platform according to the computing power operation status. The technical features correspond one to one and can achieve the same technical effects. To avoid repetition, they will not be described here.

[0155] It should be noted that the device for adjusting the model training parameters according to the computing power operation status of the intelligent computing center cloud platform in the present invention can be a device, or a component, integrated circuit, or chip in an electronic device.

[0156] The present invention also provides an electronic device, see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided by the present invention. The electronic device includes a memory 501, a processor 502, and a program or instruction stored in the memory 501 and running on the memory 501. When the program or instruction is executed by the processor 502, Figure 1 Any steps in the corresponding method embodiment of the intelligent computing center cloud platform adjusting the model training parameters according to the computing power operation status and achieving the same beneficial effects will not be repeated here.

[0157] The processor 502 may be a CPU, an ASIC, an FPGA, or a GPU.

[0158] Those skilled in the art will understand that all or part of the steps of the method embodiment for implementing the above-mentioned intelligent computing center cloud platform to adjust the model training parameters according to the computing power operation status can be completed through hardware related to program instructions, and the program can be stored in a readable medium.

[0159] The present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above Figure 1 The corresponding steps in the embodiment of the method for adjusting the model training parameters according to the computing power operation status of the intelligent computing center cloud platform can achieve the same technical effect. To avoid repetition, they are not repeated here. The storage medium is such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0160] The present invention also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the above Figure 1 The corresponding intelligent computing center cloud platform adjusts the model training parameters according to the computing power operation status of the method embodiment of each process, and can achieve the same technical effect, to avoid repetition, will not be repeated here.

[0161] The terms "first", "second" and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. In addition, the terms "comprise" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. In addition, "and / or" is used in this application to represent at least one of the connected objects, for example A and / or B and / or C, which means comprising seven situations including single A, single B, single C, and both A and B exist, both B and C exist, both A and C exist, and both A, B and C exist.

[0162] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of each embodiment of the present application.

[0164] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for adjusting model training parameters according to the computing power operation status of an intelligent computing center cloud platform, characterized in that: include: Step S1: monitoring operation-related parameters of a target link in a first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform, the first working component is used to perform model training on a first model, the target link includes a data preparation link, a model training link, a model evaluation link, and / or a model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters; Step S2: calculating target correction parameters corresponding to the operation-related parameters within a preset period based on a preset control model; Step S3: Adjust the preset hyperparameter based on the target correction parameter to obtain the target hyperparameter.

2. The method according to claim 1, wherein The operation-related parameters include computing power resource-related parameters and / or time-related parameters, and the preset control model includes a computing power resource control sub-model and / or a time control sub-model; The step S2 comprises: Step S21: Calculate the computing power resource related parameters within the preset period based on the computing power resource control sub-model to obtain a first intermediate correction parameter, where the first intermediate correction parameter is used to improve computing power resource utilization; and / or Step S22: calculating the time-related parameter within the preset period based on the time control sub-model to obtain a second intermediate correction parameter, where the second intermediate correction parameter is used to reduce the waiting time; Step S23: Generate the target correction parameter based on the first intermediate correction parameter and / or the second intermediate correction parameter.

3. The method according to claim 2, wherein In the case where the operation-related parameters include the computing power resource-related parameters and the time-related parameters, and the preset control model includes the computing power resource control sub-model and the time control sub-model, step S23 includes: Step S231: Weight the first intermediate correction parameter and the second intermediate correction parameter to obtain the target correction parameter.

4. The method according to claim 2, wherein The time-related parameters include at least one of a first time parameter for reading the original data, a second time parameter for preprocessing the original data, a third time parameter for the preprocessed target data to wait in the data cache pool, or a fourth time parameter for backpropagation to read the target data.

5. The method according to claim 1, wherein The operation-related parameters include training convergence-related parameters, the preset control model includes a model convergence control sub-model, and step S2 includes: Step S24: Calculate the training convergence-related parameters within the preset period based on the model convergence control sub-model to obtain the target correction parameters, which are used to improve the model convergence speed.

6. The method according to claim 5, wherein The training convergence related parameters include at least one of a learning rate, a training accuracy, a verification accuracy, a batch size, a gradient accumulation step number, a training loss, and a verification loss; The target correction parameter includes at least one of a learning rate correction parameter, a batch size correction parameter, and a gradient accumulation step number correction parameter.

7. The method according to any one of claims 1 to 6, characterized in that In a case where the target link is the data preparation link, the preset hyperparameter includes at least one of a data batch size, a number of data preparation processes, and a data buffer pool size; and / or, When the target link is the model training link, the preset hyperparameter includes at least one of a data reading batch size and a learning rate; and / or, In the case where the target link is the model evaluation link, the preset hyperparameter includes a model evaluation frequency; and / or, When the target link is the model testing link, the preset hyperparameters include model testing frequency.

8. The method according to any one of claims 1 to 6, characterized in that The preset control model is obtained in the following way: Acquiring sample parameters, wherein the sample parameters include running sample parameters and correcting sample parameters; An initial reinforcement learning algorithm is trained based on the sample parameters to obtain the preset control model.

9. The method according to any one of claims 1 to 6, characterized in that The first working component is a component included in the target node in the intelligent computing center cloud platform, and the target node includes multiple components. Before step S1, the method further includes: Step S0: split the initial model into multiple sub-models, and assign the multiple sub-models to different components, wherein the multiple sub-models include the first model.

10. A device for adjusting model training parameters according to the computing power operation status of an intelligent computing center cloud platform, characterized in that: include: a monitoring module, configured to monitor operation-related parameters of a target link in a first working component, where the first working component is a component deployed in a node in the intelligent computing center cloud platform, the first working component is configured to perform model training on a first model, the target link includes a data preparation link, a model training link, a model evaluation link, and / or a model testing link in the first working component, and the target link performs data processing or model training based on preset hyperparameters; A generation module, configured to calculate target correction parameters corresponding to the operation-related parameters within a preset period based on a preset control model; An adjustment module is used to adjust the preset hyperparameter based on the target correction parameter to obtain the target hyperparameter.

11. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform as described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform as described in any one of claims 1 to 9.

13. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the method for adjusting model training parameters according to the computing power operation status of the intelligent computing center cloud platform as described in any one of claims 1 to 9.

Images