Distributed self-adaptive intelligent computing resource scheduling management system

Abstract

The invention relates to a distributed self-adaptive intelligent computing resource scheduling management system, and relates to the field of AI intelligent model training, the system comprises a system interface module, a resource scheduling module, a monitoring module and a training module, the system interface display module is used for managing the system and sending a creation instruction of an AI training task, and the resource scheduling module is used for scheduling the AI training task; the resource scheduling module responds to a creation instruction, carries out initial resource allocation on a created AI training task, receives current resource information transmitted by the monitoring module and adjusts a current resource allocation scheme based on the current resource information, the monitoring module is used for monitoring and obtaining the current resource information, and the training module executes the AI training task and sends the current resource allocation scheme to the AI training task. And calculating a GOODPUT value of each pod in the current AI training task, and adjusting a batch size parameter and a learning rate parameter when the AI training task is trained based on the GOODPUT value. According to the method, the effect of optimizing the efficiency of the training task is achieved.

Description

Technical Field

[0001] The present application relates to the field of AI intelligent model training, and in particular to a distributed adaptive intelligent computing resource scheduling and management system. Background Art

[0002] In recent years, the rapid development of artificial intelligence, especially deep learning, has promoted distributed machine learning to become a research hotspot. With the increase in the number of training data sets and model parameters, single-machine training can no longer meet the needs of large-scale data processing. Distributed machine learning not only improves computing efficiency but also speeds up training by utilizing multi-node parallel processing.

[0003] The rapid development of large models in recent years has greatly increased the demand for graphics card resources. Although hardware is also developing rapidly, it still cannot meet the computing power requirements brought about by the development of artificial intelligence. The scale of distributed training clusters is getting larger and larger. A common challenge faced when performing distributed training in a cluster environment is how to allocate resources. The appropriate amount of resources is crucial to avoid resource queuing and inefficient utilization, while insufficient resource allocation may lead to extended training time and idle resources. Some existing solutions provide the ability to adjust task resources, but often ignore the specific needs of machine learning tasks. For example, the training efficiency of the same batch parameters and learning rate parameters under different GPU resources is different. Therefore, how to maximize the utilization of hardware resources in distributed clusters through the system, improve training efficiency, and maximize hardware production capacity is a major challenge in the current era of artificial intelligence. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a distributed adaptive intelligent computing resource scheduling and management system.

[0005] In the first aspect, the present application provides a distributed adaptive intelligent computing resource scheduling and management system, which adopts the following technical solutions: The adaptive intelligent computing resource scheduling management system based on distributed computing includes: a system interface module, a resource scheduling module, a monitoring module and a training module, wherein the resource scheduling module and the monitoring module are respectively connected to the system interface module, and the training module is connected to the resource scheduling module, wherein: System interface display module: used to manage the system and send a creation instruction for an AI training task, where the AI ​​training task consists of at least two pods; Resource scheduling module: responds to creation instructions, performs initial resource allocation for created AI training tasks, receives current resource information transmitted by the monitoring module, and adjusts the current resource allocation plan based on the current resource information; Monitoring module: used to monitor and obtain current resource information, including the GPU resources and topology of each computing node in the system, and the GOODPUT value of the current AI training task obtained from the training module. The GOODPUT value comprehensively reflects the statistical efficiency and throughput of the current AI training task. Training module: After the resource scheduling module completes the initial resource allocation, it executes the AI ​​training task and calculates the GOODPUT value of each pod in the current AI training task. Based on the GOODPUT value, it adjusts its own batch size parameters and learning rate parameters when training the AI ​​training task.

[0006] In one possible implementation, the system interface display module provides a UI interface, and the user configures relevant parameters of the AI ​​training task through the UI interface, including: computing resource type, initial number of GPUs, and mounting directory.

[0007] In one possible implementation, the monitoring module includes two parts: a monitoring agent and a monitoring module backend. The monitoring agent is installed on each computing node in the system. The monitoring agent collects current resource information of the system and transmits the collected current resource information to the monitoring module backend for analysis and storage.

[0008] In one possible implementation, the scheduling strategies provided by the resource scheduling module include basic scheduling, extended scheduling, and global optimal scheduling. The resource scheduling module performs initial resource allocation for the created AI training task, receives current resource information transmitted by the monitoring module, and adjusts the current resource allocation scheme based on the current resource information, including: Allocate the required GPU resources to the created AI training tasks through basic scheduling; The global optimal scheduling starts to obtain the current resource information recorded in the monitoring module at every preset time period; The extended scheduling determines the current resource allocation plan by combining the current resource information and the resource requirements required for the AI ​​training task.

[0009] In one possible implementation, the extended scheduling determines the current resource allocation scheme by combining the current resource information and the resource requirement size required by the AI ​​training task, including: Based on the topological structure in the current resource information, determine whether high-speed RDMA connections are used between different computing nodes in the current system and whether there is data in the GOOTPUT value in the current resource information; If the GOOTPUT value does not contain data and a high-speed RDMA connection is not used, all pods of the same AI training task are scheduled to the same computing node that can meet the resource requirements of all pods. If there is no computing node in the system that meets the resource requirements of the AI ​​training task, all pods are assigned to idle computing nodes in the form of one pod per computing node. If the GOOTPUT value does not contain data but a high-speed rdma connection is used, distributed scheduling is preferred to schedule pods of the same AI training task to different idle computing nodes with rdma connections; If the GOOTPUT value has data and high-speed RDMA connection is not used, it is determined that the current scheduling does not belong to the initial resource allocation. The current FITNESS value is calculated based on the current resource information, and the optimal solution of FITNESS and the current optimal resource allocation plan are automatically calculated. The current resource allocation is adjusted according to the current optimal resource allocation plan; If the GOOTPUT value contains data and a high-speed RDMA connection is used, it is determined that the current scheduling does not belong to the initial resource allocation. The current FITNESS value is calculated based on the current resource information, and the optimal solution of FITNESS and the current optimal resource allocation plan are automatically calculated. The current resource allocation is adjusted according to the current optimal resource allocation plan.

[0010] In a possible implementation, the current FITNESS value is calculated based on the current resource information, and the optimal solution of FITNESS and the current optimal resource allocation scheme are automatically calculated, including: The FITNESS value, the optimal solution of FITNESS, and the current optimal resource allocation scheme are calculated by maximizing the fitting function, where the maximization fitting function is as follows: ; The definition of SPEEDUP is as follows: ; Among them, J represents the number of AI training tasks, A represents an allocation matrix, and A j Indicates the number of GPU resources allocated to the pod participating in AI training task j, a f It represents the average number of GPU resources allocated to each pod in the AI ​​training task. p is the allocation strategy parameter that can be manually modified. The extended scheduling calculates and searches for the current optimal resource allocation solution corresponding to FITNESS based on the above maximum fitting function.

[0011] In a possible implementation, the training module includes an analyzer and a parameter adjuster, and the calculation of the GOODPUT value of each pod in the current AI training task and the adjustment of the batch size parameter and the learning rate parameter when training the AI ​​training task based on the GOODPUT value include: Calculate the GOODPUT value of the current AI training task through the analyzer; The parameter tuner adjusts the batch size and learning rate parameters when training AI training tasks based on the GOODPUT value.

[0012] In a possible implementation, calculating the GOODPUT value of each pod in the current AI training task by the analyzer includes: The analyzer calculates the GOODPUT value of each pod through the GOODPUT model, which is as follows: ; where a represents an allocation vector of size N, m represents the batch size per GPU, s represents the number of gradient accumulation steps, EFFICIENCY represents the training progress promoted by each training sample of batch size M relative to the initial batch size M0, and φ t represents the pre-processing gradient noise scale, g represents the true gradient, P is the pre-conditioning matrix of the adaptive SGD algorithm, Σ is the covariance matrix of the random gradient of each sample, THROUGHPUT(a,m,s) represents the system throughput, T iter (a, m, s) represents the total computation time, T grad (a,m) represents the local gradient time, T sync (a) represents the gradient synchronization time.

[0013] In one possible implementation, the parameter adjuster adjusts the batch size parameter and the learning rate parameter when training the AI ​​training task based on the GOODPUT value, including: The parameter adjuster determines whether the GOODPUT value reaches the optimal GOODPUT value; If it is not reached, continue to adjust the batch size parameters and learning rate parameters when training the AI ​​training task; If it is reached, no further adjustments will be made.

[0014] In a second aspect, the present application provides a computer-readable medium, which adopts the following technical solution: A computer-readable storage medium, characterized in that a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, a distributed adaptive intelligent computing resource scheduling and management system as described in any one of claims 1 to 9 is implemented.

[0015] In summary, this application includes the following beneficial technical effects: The adaptive intelligent computing resource scheduling and management system provided in this application first optimizes the scheduling of AI training tasks by perceiving the topological structure of each computing node in the overall system. After the scheduling is completed, it monitors the overall cluster resources based on the k8s scheduler extension framework, regularly obtains GOOTPUT and other data recorded in the resource description of each pod in the AI ​​training task, and evaluates the training process in real time. It dynamically adjusts its own batch size and learning rate parameters according to the evaluation results, and calculates the optimal resource allocation plan suitable for the current situation according to the maximum fitting function, thereby achieving the efficiency optimization effect of the training task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural diagram of a distributed adaptive intelligent computing resource scheduling and management system in an embodiment of the present application.

[0017] Figure 2 It is a structural diagram of the resource scheduling module in the embodiment of the present application.

[0018] Figure 3 It is a structural diagram of the training module in the embodiment of the present application. DETAILED DESCRIPTION

[0019] The present application is further described in detail below in conjunction with the accompanying drawings.

[0020] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0023] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0024] First, the key terms and abbreviations in this application are explained: Statistical efficiency: In deep learning, it reflects the degree to which the model effectively utilizes resources during the training process. By using efficient training algorithms, applying early stopping to avoid overfitting, and using parallel computing resources such as GPUs, statistical efficiency can be improved, the model update speed can be accelerated, and the accuracy of the model can be improved within a given time.

[0025] Learning rate: refers to the hyperparameter used by deep learning models to update weights during gradient descent during training. Using a low learning rate ensures that we do not miss any local minima and achieve better training results, but it also means a longer convergence time. On the contrary, an excessively large learning rate will make it difficult for the model to converge. A reasonable learning rate will allow the model to converge to the minimum point rather than the optimal point, thereby obtaining the best learning results.

[0026] Throughput: refers to the amount of data that a deep learning model can process per unit time. It directly affects the speed and efficiency of model training. Generally, higher throughput means that the model can process more data in the same amount of time, thereby improving the efficiency of the training process.

[0027] Batch size: The batch size parameter of training can affect the throughput of the system. A larger batch parameter means more computing resources and higher utilization, and a higher throughput. However, a larger batch will lead to a decrease in statistical efficiency.

[0028] NUMA: Non-uniform memory access (NUMA) is a memory architecture designed for multi-processor computers. It separates CPU resources and groups them into computing nodes. Each NUMA node has its own unique CPU, memory, GPU and other resources.

[0029] Gradient Noise Scale: During deep learning training, we calculate the gradient of the loss function with respect to the model parameters to guide the direction and size of parameter updates. However, due to the randomness of the data and the complexity of the model, the gradient calculated for each batch of data is often not completely accurate, which leads to the so-called "gradient noise". The size of the gradient noise, that is, the variance of the gradient, is called the gradient noise scale.

[0030] AI training task: refers to the use of machine learning or deep learning algorithms to train models so that the models can perform specific tasks or solve specific problems. In order to accelerate the completion of training tasks and generate training models, AI training tasks are generally composed of 2 or more pods to jointly complete a training task. In this application, each pod has a corresponding GOODPUT value.

[0031] The present application embodiment provides a distributed adaptive intelligent computing resource scheduling management system, such as Figure 1 As shown, the system includes a system interface module, a resource scheduling module, a monitoring module and a training module, wherein the resource scheduling module and the monitoring module are respectively connected to the system interface module, and the training module and the monitoring module are respectively connected to the resource scheduling module. Specifically, the functions of each module are as follows: 1. System interface module The system interface module provides an intuitive and easy-to-use graphical interface for user interaction. Through the UI interface, users can easily manage the system and create AI training tasks. In addition, users can also configure relevant parameters of AI training tasks through the UI interface, such as the required number of GPU resources, mount directory, etc.

[0032] When the system interface module detects the user's creation operation, it means that the current user has completed the creation of the AI ​​training task that needs to be trained and submitted it to the system. At this time, the system interface module will send the resource scheduling instructions of the AI ​​training task to the resource scheduling module.

[0033] 2. Resource Scheduling Module: The resource scheduling module is used to respond to resource scheduling instructions, perform initial resource allocation for the created AI training tasks, receive current resource information transmitted by the monitoring module, and adjust the current resource allocation plan based on the current resource information. Specifically, Figure 2 As shown in the figure, the scheduling strategies provided by the resource scheduling module include basic scheduling, extended scheduling, and global optimal scheduling. When the status of the AI ​​training task is different, the processing method of the resource scheduling module is also different, as follows: 2.1. For the newly created AI training task, since the scheduling strategy of the global optimal scheduling is the GOODPUT value in the current resource information obtained every preset time period, but there is no GOODPUT value data in the newly created AI training task, the basic GPU resource requirements of the AI ​​training task are first met through basic scheduling, and then the extended scheduling determines whether the various computing nodes in the current system are connected by high-speed RDMA through the topological structure in the current resource information, and then determines the corresponding scheduling strategy based on the judgment result, as follows: (1) When the judgment result is that high-speed RDMA connection is not used, the network speed between computing nodes will become the biggest bottleneck of the training task. At this time, the extended scheduling will give priority to scheduling all pods of the same AI training task to the same computing node that can meet the resource requirements of all pods, so as to reduce the delay caused by cross-node communication; and when there is no computing node that meets the resource requirements of all pods of the AI ​​training task, the extended scheduling will allocate all pods to idle computing nodes in the form of one pod per computing node. Specifically, when allocating each pod to an idle computing node, the computing node to be allocated is first determined from the idle computing nodes based on the GPU resources corresponding to each computing node, and then the GPU resources corresponding to each pod are determined from the multiple GPU resources of the computing node to be allocated, thereby completing the resource scheduling process.

[0034] (2) When the judgment result is to use a high-speed RDMA connection, the extended scheduling will give priority to the distributed scheduling method to schedule the pods of the same AI training task to different idle computing nodes with RDMA connections. Specifically, when scheduling the pods of the same AI training task to different idle computing nodes with RDMA connections, the computing nodes to be allocated will be determined from the different idle computing nodes with RDMA connections based on the GPU resources corresponding to each computing node. Then, the GPU resources corresponding to each pod will be determined from all the GPU resources of the computing nodes to be allocated, thereby completing the resource scheduling process.

[0035] 2.2. For AI training tasks that have undergone partial training, the global optimal scheduling contains data in the GOOTPUT value contained in the current resource information obtained every preset time period. At this time, the scheduling module will calculate the current FITNESS value based on the current resource information, and automatically calculate the optimal solution of FITNESS and the current optimal resource allocation plan, and adjust the current resource allocation according to the current optimal resource allocation plan. Specifically, in the embodiment of the present application, the current FITNESS value, the optimal solution of FITNESS and the current optimal resource allocation plan are calculated by the maximum fitting function, as follows: ; Among them, SPEEDUP is defined as follows: ; m represents the batch data size allocated to each GPU in each iteration of the training task, s represents the number of gradient accumulation steps, J represents the number of AI training tasks, and A represents an allocation matrix. j Indicates the number of GUP resources allocated to the participating pods of AI training task j, a fIt represents the average number of GPU resources allocated to each pod in the AI ​​training task (assuming there are n GPUs in the cluster, each computing node is allocated 1 / n of the cluster's resources), p is a manually modifiable allocation strategy parameter (p is 1, which means that the most resources will be allocated to tasks with high SPPEDUP, that is, the most efficient tasks will be allocated as many resources as possible to achieve the local optimal computing speed, and when p is negative infinity, it tends to the global FITNESS optimal allocation). The extended scheduling calculates and searches for the current optimal resource allocation plan corresponding to FITNESS based on the above maximum fitting function.

[0036] 3. Monitoring Module The monitoring module consists of two parts: the monitoring agent and the monitoring module backend. The monitoring agent is installed on each computing node in the system. The monitoring agent collects the current resource information of the system, and transmits the collected GPU resources, topology structure of each computing node, and the GOODPUT value of the current AI training task obtained from the training module to the monitoring module backend. The monitoring module backend analyzes and stores the information, thereby realizing the hardware resource identification and information collection of each node, providing monitoring display data for the system interface module, and obtaining relevant data for the resource scheduling module, including but not limited to the graphics card network topology and the current GOODPUT of the AI ​​training task.

[0037] 4. Training Module The training module is used to execute the AI ​​training task after the resource scheduling module completes the initial resource allocation. In the embodiment of the present application, the training module re-encapsulates the corresponding training interface based on the deep learning framework pytorch, and expands the functions of the existing interface to realize the evaluation of the training effect of the AI ​​training task. At the same time, the training module will dynamically adjust its batch size parameters and learning rate parameters according to the evaluation results during operation. Specifically, Figure 3 As shown, the training module of this application adds two parts, an analyzer and a parameter tuner, based on the original deep learning framework pytorch. The analyzer is used to calculate the GOODPUT value of the pod that has been trained in the current AI training task. The parameter tuner adjusts the batch size parameters and learning rate parameters for subsequent training of other pods in the AI ​​training task based on the calculated GOODPUT value. The specific implementation process is as follows: The GOODPUT model is defined in this application: the GOODPUT of an AI training task at iteration t is the product of its system throughput and statistical efficiency, as follows: ; , where SUM(a) represents the total number of GPUs used by the training task.

[0038] ; where a represents an allocation vector of size N, m represents the batch size per GPU, s represents the number of gradient accumulation steps, EFFICIENCY represents the training progress promoted by each training sample of batch size M relative to the initial batch size M0, and φ t represents the pre-processing gradient noise scale, tr() represents the trace of the matrix in the brackets, g represents the true gradient, P is the pre-conditioning matrix of the adaptive SGD algorithm, Σ is the covariance matrix of the random gradient of each sample, THROUGHPUT(a,m,s) represents the system throughput, T iter (a, m, s) represents the total computation time, T grad (a,m) represents the local gradient time, T sync (a) represents the gradient synchronization time, and the superscript r is the learning parameter of the model.

[0039] Based on the above algorithm, the interface is extended on the basis of training pytorch to realize that the training program can obtain batch parameters, batch training time and other training information by itself during training. The GOODPUT value is calculated according to the above algorithm, and the data is put into the resource description of pods to facilitate the global optimal scheduling reading in the subsequent resource scheduling module. In addition, the training module code will adjust its own parameters according to the calculated GOODPUT value during operation to obtain the best GOODPUT result.

[0040] To facilitate understanding of the functions of each module in the process of training AI training tasks, the following provides the specific implementation process of each module in training AI training tasks, as follows: S1, the user configures the relevant parameters of the AI ​​training task and creates the AI ​​training task through the UI interface provided by the system interface module, and triggers the creation instruction at the same time; S2, the resource scheduling module responds to the creation instruction and records the relevant parameters configured by the user (for example, the number of GPU resources required for the training task); S3, the monitoring module obtains the current resource information in real time, wherein the current resource information includes: the GPU resources and topology of each computing node and the GOODPUT value of the currently running AI training task; S4, the resource scheduling module obtains the current resource information from the monitoring module (since the AI ​​training task has just been created at this time, the GOODPUT value obtained at this time has no data), and determines whether the various computing nodes in the current system use high-speed RDMA connections based on the topological structure in the current resource information: (1) If not, all pods of the AI ​​training task are preferentially scheduled to the same computing node. If there is no computing node in the system that meets the resource requirements of the AI ​​training task, all pods are allocated to idle computing nodes in the form of one pod per computing node; (2) If it is adopted, the distributed scheduling method is preferentially adopted to evenly schedule the pods of the same AI training task to different idle computing nodes that have RDMA connections with each other; S5, after the preset time period, the resource scheduling module obtains the current resource information from the monitoring module again (since the AI ​​training task has been partially trained at this time, the GOODPUT value obtained at this time has data), calculates the current FITNESS value based on the current resource information, and automatically calculates the optimal solution of FITNESS and the current optimal resource allocation plan, and adjusts the resource allocation plan determined in the previous preset time period according to the current optimal resource allocation plan; S6: Due to the change in resource allocation plan, the scheduling system will reschedule the pod and allocate resources. The re-running training module will calculate the current GOODPUT value based on the current resource information, and adjust its own batch size parameters and learning rate parameters based on the GOODPUT value. S5 and S6 will be executed in a loop to dynamically adjust the resource allocation of the AI ​​training task, and find the optimal resource allocation plan, batch size, and learning rate based on the GOODPUT of the AI ​​training task to maximize the overall resource utilization of the cluster.

[0041] The adaptive intelligent computing resource scheduling and management system provided in this application first optimizes the scheduling of AI training tasks by perceiving the topological structure of each computing node in the overall system. After the scheduling is completed, it monitors the overall cluster resources based on the k8s scheduler extension framework, regularly obtains GOOTPUT and other data recorded in the resource description of each pod in the AI ​​training task, and evaluates the training process in real time. It dynamically adjusts its own batch size and learning rate parameters according to the evaluation results, and calculates the optimal resource allocation plan suitable for the current situation according to the maximum fitting function, thereby achieving the efficiency optimization effect of the training task.

[0042] An embodiment of the present application provides a computer-readable medium having a computer program stored thereon. When the computer-readable medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0043] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0044] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A distributed adaptive intelligent resource scheduling and management system, characterized in that: include: System interface module, resource scheduling module, monitoring module and training module, wherein the resource scheduling module and the monitoring module are connected to the system interface module respectively, and the training module is connected to the resource scheduling module, wherein System interface display module: used to manage the system and send a creation instruction for an AI training task, where the AI ​​training task consists of at least two pods; Resource scheduling module: responds to creation instructions, performs initial resource allocation for created AI training tasks, receives current resource information transmitted by the monitoring module, and adjusts the current resource allocation plan based on the current resource information; Monitoring module: used to monitor and obtain current resource information, including the GPU resources and topology of each computing node in the system, and the GOODPUT value of the current AI training task obtained from the training module. The GOODPUT value comprehensively reflects the statistical efficiency and throughput of the current AI training task. Training module: After the resource scheduling module completes the initial resource allocation, it executes the AI ​​training task and calculates the GOODPUT value of each pod in the current AI training task. Based on the GOODPUT value, it adjusts its own batch size parameters and learning rate parameters when training the AI ​​training task.

2. The distributed adaptive intelligent computing resource scheduling and management system according to claim 1 is characterized in that: The system interface display module provides a UI interface, through which the user configures the relevant parameters of the AI ​​training task, including: computing resource type, initial number of GPUs, and mounting directory.

3. The distributed adaptive intelligent computing resource scheduling and management system according to claim 1 is characterized in that: The monitoring module includes a monitoring agent and a monitoring module backend. The monitoring agent is installed on each computing node in the system. The monitoring agent collects the current resource information of the system and transmits the collected current resource information to the monitoring module backend for analysis and storage.

4. The distributed adaptive intelligent computing resource scheduling and management system according to claim 1 is characterized in that: The scheduling strategies provided by the resource scheduling module include basic scheduling, extended scheduling, and global optimal scheduling. The resource scheduling module performs initial resource allocation for the created AI training task, receives the current resource information transmitted by the monitoring module, and adjusts the current resource allocation plan based on the current resource information, including: Allocate the required GPU resources to the created AI training tasks through basic scheduling; The global optimal scheduling starts to obtain the current resource information recorded in the monitoring module at every preset time period; The extended scheduling determines the current resource allocation plan by combining the current resource information and the resource requirements required for the AI ​​training task.

5. The distributed adaptive intelligent computing resource scheduling and management system according to claim 4 is characterized in that: The extended scheduling determines the current resource allocation scheme by combining the current resource information and the resource requirement size required by the AI ​​training task, including: Based on the topological structure in the current resource information, determine whether high-speed RDMA connections are used between different computing nodes in the current system and whether there is data in the GOOTPUT value in the current resource information; If the GOOTPUT value does not contain data and a high-speed RDMA connection is not used, all pods of the same AI training task are scheduled to the same computing node that can meet the resource requirements of all pods. If there is no computing node in the system that meets the resource requirements of the AI ​​training task, all pods are assigned to idle computing nodes in the form of one pod per computing node. If the GOOTPUT value does not contain data but a high-speed rdma connection is used, distributed scheduling is preferred to schedule pods of the same AI training task to different idle computing nodes with rdma connections; If the GOOTPUT value has data and high-speed RDMA connection is not used, it is determined that the current scheduling does not belong to the initial resource allocation. The current FITNESS value is calculated based on the current resource information, and the optimal solution of FITNESS and the current optimal resource allocation plan are automatically calculated. The current resource allocation is adjusted according to the current optimal resource allocation plan; If the GOOTPUT value contains data and a high-speed RDMA connection is used, it is determined that the current scheduling does not belong to the initial resource allocation. The current FITNESS value is calculated based on the current resource information, and the optimal solution of FITNESS and the current optimal resource allocation plan are automatically calculated. The current resource allocation is adjusted according to the current optimal resource allocation plan.

6. The distributed adaptive intelligent computing resource scheduling and management system according to claim 5 is characterized in that: The method of calculating the current FITNESS value based on the current resource information and automatically calculating the optimal solution of FITNESS and the current optimal resource allocation scheme includes: The FITNESS value, the optimal solution of FITNESS, and the current optimal resource allocation scheme are calculated by maximizing the fitting function, where the maximization fitting function is as follows: ； The definition of SPEEDUP is as follows: ； Among them, J represents the number of AI training tasks, A represents an allocation matrix, and A j Indicates the number of GPU resources allocated to the pod participating in AI training task j, a f It represents the average number of GPU resources allocated to each pod in the AI ​​training task. p is the allocation strategy parameter that can be manually modified. The extended scheduling calculates and searches for the current optimal resource allocation solution corresponding to FITNESS based on the above maximum fitting function.

7. The distributed adaptive intelligent computing resource scheduling and management system according to claim 1 is characterized in that: The training module includes an analyzer and a parameter adjuster. The GOODPUT value of each pod in the current AI training task is calculated, and the batch size parameter and learning rate parameter of the training AI training task are adjusted based on the GOODPUT value, including: Calculate the GOODPUT value of the current AI training task through the analyzer; The parameter tuner adjusts the batch size and learning rate parameters when training AI training tasks based on the GOODPUT value.

8. The distributed adaptive intelligent computing resource scheduling and management system according to claim 7 is characterized in that: The GOODPUT value of each pod in the current AI training task is calculated by the analyzer, including: The analyzer calculates the GOODPUT value of each pod through the GOODPUT model, which is as follows: ； where a represents an allocation vector of size N, m represents the batch size per GPU, s represents the number of gradient accumulation steps, EFFICIENCY represents the training progress promoted by each training sample of batch size M relative to the initial batch size M0, and φ t represents the pre-processing gradient noise scale, g represents the true gradient, P is the pre-conditioning matrix of the adaptive SGD algorithm, Σ is the covariance matrix of the random gradient of each sample, THROUGHPUT(a,m,s) represents the system throughput, T iter (a, m, s) represents the total computation time, T grad (a,m) represents the local gradient time, T sync (a) represents the gradient synchronization time.

9. The distributed adaptive intelligent computing resource scheduling and management system according to claim 7 is characterized in that: The parameter adjuster adjusts the batch size parameter and the learning rate parameter when training the AI ​​training task based on the GOODPUT value, including: The parameter adjuster determines whether the GOODPUT value reaches the optimal GOODPUT value; If it is not reached, continue to adjust the batch size parameters and learning rate parameters when training the AI ​​training task; If it is reached, no further adjustments will be made.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the distributed adaptive intelligent computing resource scheduling and management system as described in any one of claims 1 to 9 is implemented.

Images