Deep learning training job resource placement system and method based on reinforcement learning
Through deep learning training of job resource placement method based on reinforcement learning, the DRL neural network model is trained, cluster failures and job characteristics are analyzed, and the job placement location is optimized, which solves the problem of low resource utilization in the existing technology and achieves more efficient resources Improve the quality of use and user service.
Patent Information
- Application Number
- CN202110874519.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-07-30
AI Technical Summary
The prior art fails to effectively consider the error characteristics of different GPUs in the cluster in the resource scheduling of multi-machine and multi-card deep learning training jobs, resulting in frequent job restarts, low resource utilization, and load-aware methods fail to make full use of cluster resources.
The deep learning training job resource placement method based on reinforcement learning is adopted. By training the DRL neural network model, cluster failures and job characteristics are adaptively analyzed, the job placement position is optimized, the maximum completion time is reduced, and resource utilization is improved.
Through the adaptive placement strategy, the maximum completion time of large distributed deep learning jobs is reduced, resource utilization is improved, job restart overhead is reduced, and user service quality is improved.
Smart Images

Figure CN113535365B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computing resource scheduling, and in particular to a system and method for placing deep learning training job resources based on reinforcement learning. Background Art
[0002] Deep Learning Training (DLT) jobs are usually computationally intensive tasks that require powerful and expensive computing resources, such as GPU devices. In order to handle the ever-growing amount of training data, most mainstream IT companies or enterprises currently run DLT jobs through GPU server clusters and perform distributed deep learning (DDL) training to utilize multiple GPUs in parallel, thereby reducing the load on a single GPU and speeding up the training rate of the model.
[0003] The multi-machine and multi-card training mode is the main feature of large-scale distributed DLT jobs. As the system complexity increases, the probability of job errors will increase year-on-year. In addition, the DLT job training time is generally long, and long-term operation will also increase the probability of job errors. In addition, frequent submissions in multi-tenant and multi-job scenarios usually also lead to an increase in the probability of job errors. DLT job errors are one of the important reasons for the decline in system resource utilization. The time overhead caused by job errors cannot be ignored. The more errors there are, the greater the job restart overhead and resource recovery overhead caused by the errors, and the lower the resource utilization.
[0004] In order to better place DLT jobs in the scenario of cluster resource errors, the prior art proposes a method based on cluster capacity perception and a method based on load interference perception. Among them, the method based on cluster capacity perception does not take into account the error characteristics of different GPUs in the cluster. For example, when a GPU device with a low error probability is in a relatively high load state for a long time, the scheduling strategy is likely to frequently place multi-card DLT large jobs on GPU devices with a high error probability, which will cause the job to restart multiple times, resulting in a decrease in resource utilization. Although the method based on load interference perception largely avoids the training performance degradation and resource utilization caused by interference between DLT jobs, it still does not take into account the error characteristics of each GPU device in the cluster. For example, if the GPUs with a high error probability in the cluster are dispersed, when the distributed multi-card DLT jobs with a large degree of interference are placed separately, they are likely to be placed on GPUs with a high error probability, resulting in frequent job restarts, bringing more serious training performance and resource utilization. Decrease.
[0005] Reinforcement Learning (RL) is similar to traditional deep learning methods and is also a self-learning method. However, deep learning is a static learning algorithm that predicts unknown data by learning features from existing data. RL, on the other hand, is a dynamic learning algorithm that continuously explores unknown environments, builds decision-making models, and learns the optimal strategy. Therefore, to some extent, RL is more in line with human thinking and learning processes, especially RL that incorporates deep learning technology, namely Deep Reinforcement Learning (DRL), which is recognized as the paradigm closest to true artificial intelligence.
[0006] Therefore, how to apply the DRL algorithm to resource scheduling, that is, the decision-making problem of job placement, and reasonably place DLT jobs in the scenario of cluster resource errors to maximize resource utilization and improve user service quality has become a problem that needs to be solved. Summary of the invention
[0007] One of the objectives of the present invention is to provide a deep learning training job resource placement method based on reinforcement learning, which can adaptively place DLT jobs in resource error scenarios.
[0008] In order to solve the above technical problems, this application provides the following technical solutions:
[0009] The method for placing resources for deep learning training jobs based on reinforcement learning includes the following steps:
[0010] Initialization step: Randomly initialize the parameters of the DRL neural network model;
[0011] State vector generation step: generating the state vector of the batch job;
[0012] Reasoning step: Send the state vector to the DRL neural network model to infer the placement information of the batch job, and place the job according to the placement information to obtain the maximum completion time of the batch job operation, which is recorded as T_RL;
[0013] Random generation step: randomly generate several placement location information, and place the job according to the randomly generated placement location information to obtain several maximum completion times of the batch job, and obtain the smallest maximum completion time among them and record it as T_Random;
[0014] Reward calculation steps: Calculate rewards based on the maximum completion time T_RL and the maximum completion time T_Random;
[0015] Parameter updating step: Backward gradient updates the parameters of the DRL neural network model.
[0016] The basic scheme principles and beneficial effects are as follows:
[0017] In this solution, the DRL neural network model is trained to infer the job placement location. Compared with the traditional heuristic algorithm, the DRL neural network model can analyze and extract more effective and accurate features of cluster failures and DLT jobs by itself without manually selecting certain parameters as features, thereby reducing the impact caused by incorrect human selection of features.
[0018] The reward of the DRL neural network model is calculated based on the minimum value T_random of the completion time of multiple randomly scheduled batch jobs. Using randomness to obtain a larger reward range can improve the learning ability of the DRL neural network model.
[0019] The training process of this scheme can use the simulator for pre-training or full training to save time and economic costs. It can also use the historical data of the real cluster system for training in order to obtain a scheduling strategy that is more suitable for the system. It can also be trained online directly on the prototype system to obtain a more accurate scheduling strategy.
[0020] In summary, this solution aims to solve the placement decision problem of DLT jobs in the event of cluster errors. It trains the DRL neural network model to perform adaptive placement in the resource error scenario of DLT jobs, thereby reducing the maximum completion time of large-scale distributed DLT jobs in batches and improving resource utilization.
[0021] Furthermore, it also includes an experience replay step: sampling the four-tuple samples generated during the DRL neural network model training process for experience replay.
[0022] Through experience replay, on the one hand, the correlation between samples can be eliminated to meet the basic requirements of neural network training. On the other hand, dynamic experience replay can maximize the scope of replay and ensure the effectiveness of experience replay.
[0023] Furthermore, in the state vector generation step, a state vector is generated based on the DLT job information and cluster information, and is recorded as Among them, N is the number of computing units required for the current job; T is the estimated running time of the current job without errors; S is the usage status of each computing unit in the current cluster.
[0024] In this preferred solution, DLT job information and cluster information are obtained, and after processing, a state vector is generated as a feature input to the DRL neural network model for training. The maximum completion time of batch jobs obtained by using the optimal random scheduling solution and the scheduling solution inferred by the current DRL neural network model are combined as evaluation criteria to guide the neural network to adaptively make placement decisions for DLT jobs, reduce the maximum completion time of batch large-scale distributed DLT jobs, and improve resource utilization.
[0025] Furthermore, the reasoning steps specifically include:
[0026] A1: Input the state vector into the value network of the DRL neural network model to obtain the long-term measurement index V;
[0027] A2: Input the state vector into the policy network of the DRL neural network model to obtain the selection probability P of N computing units i , where i = 1, 2, ..., N;
[0028] A3: The probability P corresponding to the occupied computing unit and the failed computing unit j Set to zero and get P′ i ;
[0029] A4: Select the kth computing unit as one of the computing units to be placed for the job, where P k =max(P′ i );
[0030] A5: If the number of computing units to be placed for the job is equal to the number of computing units required for the job, the placement location information reasoning for the job is completed, and the location information of the next job is inferred. Otherwise, jump to step A1.
[0031] Furthermore, the experience playback step specifically includes:
[0032] B1: Create a playback buffer pool stack;
[0033] B2: Push the four-tuple samples generated during the training process into the revisit buffer pool stack;
[0034] B3: If the revisit buffer pool is full, the earliest four-tuple sample pushed into the stack will overflow;
[0035] B4: Select X four-tuple samples as a batch for the next training, where X is the total number of four-tuple samples in the current revisit buffer pool stack.
[0036] In the DRL neural network model, after a series of reasoning actions are made, a number of quadruple samples will be generated. These samples have strong correlation, which does not meet the requirements of independent and identically distributed training samples for deep neural networks, and the sample sequence generated within an interval cannot represent the global experience. The forgetfulness of the neural network will make it easy to fall into local optimality during training. This preferred solution uses experience replay to solve this problem. Considering that the uncertainty at the moment of resource error is very strong, the number of samples generated in different scheduling intervals may vary greatly, so dynamic batches are used for sampling. Each time the DRL neural network model is trained, the size of the randomly sampled batch is equal to the number of samples inferred in the current scheduling cycle. The batch is input into the DRL neural network model for training, which can give full play to the effect of experience replay, thereby greatly reducing the correlation between samples.
[0037] Furthermore, in the reward calculation step, the reward calculation formula is:
[0038]
[0039] The smaller the expected T_RL is, the better. However, T_RL is relative, that is, it is related to the actual operation time. If the actual operation time is long, T_RL cannot be too small. Therefore, in this solution, T_random is used as a relative value and compared with T_RL.
[0040] Furthermore, it also includes a training judgment step: judging whether the DRL neural network model has been trained, if not, returning to the state vector generation step, otherwise the training ends.
[0041] Furthermore, the method also includes a use step: using the trained DRL neural network model to infer the placement position of each job in the batch job.
[0042] Further, the use steps specifically include:
[0043] C1: Get the job information and cluster information of the batch job;
[0044] C2: Generate a state vector based on the information collected in step C1;
[0045] C3: Input the state vector in step C2 into the policy network of the DRL neural network model to obtain the placement position information output by the policy network;
[0046] C4: If the number of computing units currently inferred is less than the number of computing units required for the current job, repeat step C3; otherwise, jump to step C5;
[0047] C5: Place the corresponding job according to the placement location information inferred in step C3.
[0048] The second object of the present invention is to provide a deep learning training job resource placement system based on reinforcement learning, including a DRL neural network model and a job scheduling module; the job scheduling module uses the steps of the above method to train the DRL neural network model. The placement location information is obtained from the trained DRL neural network model, and the corresponding job is placed according to the placement location information.
[0049] This solution uses the DRL neural network model to schedule computing units. It periodically obtains cluster information and currently submitted job information, processes them and uses them as feature input for DRL neural network model training. It combines the maximum completion time of batch jobs obtained by the optimal random scheduling method with the scheduling method inferred by the current DRL neural network model as an evaluation criterion to guide the DRL neural network model to adaptively make placement decisions for DLT jobs, thereby reducing the maximum completion time of batch large-scale distributed DLT jobs, thereby improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 This is a schematic diagram of the cluster job life cycle;
[0051] Figure 2 Flowchart for DRL neural network model training;
[0052] Figure 3 Schematic diagram of the DRL neural network model structure design;
[0053] Figure 4 This is a schematic diagram of experience playback;
[0054] Figure 5 Schematic diagram of the DRL neural network model reasoning process. DETAILED DESCRIPTION
[0055] The following is further described in detail through specific implementation methods:
[0056] Example
[0057] like Figure 1 As shown, the method of this embodiment is applied to the job scheduling process of the cluster, and the goal is to determine which nodes and computing resources in the cluster the job should be placed on. This embodiment takes the common computing unit - GPU as an example to introduce a method for placing deep learning training job resources based on reinforcement learning, including the following steps:
[0058] Use reinforcement learning to train neural network model parameters, such as Figure 2 As shown, specifically including:
[0059] Initialization step: Randomly initialize the parameters of the DRL neural network model.
[0060] State vector generation step: Generate the state vector of workloads (batch jobs) based on DLT job information and cluster information. The state vector is recorded as The specific expressions are as follows:
[0061] N: The number of GPUs required for the current job.
[0062] T: The estimated running time of the current job under normal circumstances without errors.
[0063] S: The usage status of each GPU in the current cluster. For example, if there are 4 GPU devices in the current cluster, the first two GPUs are available, and the last two GPUs are unavailable due to errors or being occupied, then S = [0, 0, 1, 1].
[0064] In this embodiment, DLT job information and cluster information are obtained periodically.
[0065] Reasoning step: transform the state vector The DRL neural network model is sent to infer the placement information of a workload, and the job is placed according to the placement information to obtain the maximum completion time T_RL of the workload.
[0066] The DRL neural network model of the reasoning step is as follows Figure 3 As shown, specifically including:
[0067] A1: The state vector The value network (Value Network) of the DRL neural network model is input, and after passing through 5 layers of fully connected layers with the number of neurons being 256, 196, 196, 128, and 1 respectively, a long-term measurement indicator V is obtained.
[0068] A2: The state vector The policy network of the DRL neural network model is input, and then passes through 5 layers of fully connected layers with 256, 196, 196, 128, and N neurons respectively, and then passes through the softmax layer to obtain the selection probability P of N GPUs. i ,i=1,2,...,N。
[0069] A3: The probability P corresponding to the occupied GPU and the faulty GPU j Set to zero and get P′ i .
[0070] A4: Select the kth GPU as one of the GPUs to be placed for this job, where P k =max(P′ i ).
[0071] A5: If the number of GPUs to be placed for the job is equal to the number of GPUs required for the job, the placement location information reasoning for the job is completed, and the location information of the next job is inferred. Otherwise, jump to step A1.
[0072] Random generation step: Randomly generate a series of placement location information, and place jobs according to the placement location information to obtain a series of maximum completion times for the workloads. Finally, the minimum maximum completion time is obtained and recorded as T_Random.
[0073] Reward calculation steps: Calculate the reward based on the maximum completion time T_RL and the maximum completion time T_Random. The calculation formula is:
[0074]
[0075] Experience replay step: Sample the four-tuple samples (s, a, r, s′) generated during the DRL training process to apply experience replay. In the four-tuple sample (s, a, r, s′), s is the environment state, a is the Actor, that is, an action selected based on the current strategy, s' is the next environment state after executing action a in the environment state s, and r is the reward of the environment feedback, that is, reward.
[0076] The steps of experience playback are as follows Figure 4 As shown, specifically including:
[0077] B1: Create a Replay Buffer stack.
[0078] B2: Push the four-tuple sample (s, a, r, s′) generated during the training process into the revisit buffer pool stack.
[0079] B3: If the buffer pool is full, the earliest data pushed into the stack will overflow.
[0080] B4: Select x four-tuple samples as a batch and wait for the next training, where the number of X is the total number of four-tuple samples in the current revisit buffer pool stack.
[0081] Parameter update step: Reverse gradient update of the parameters of the DRL neural network model. In this embodiment, the reward is used for reverse gradient update, which is a prior art and will not be described here.
[0082] Training judgment step: if the training is not completed, return to step S12 to continue execution, otherwise the training ends. In this embodiment, a planned training number is preset, and when the actual training number is equal to the planned training number, the training is considered to be completed.
[0083] Usage steps: Use the trained DRL neural network model to infer the placement of each job in the workloads. Figure 5 As shown, specifically including:
[0084] C1: Get workload job information and cluster GPU information.
[0085] C2: Generate the state vector using the information collected in step C1
[0086] C3: Input the state vector in step C2 into the policy network to obtain the placement strategy output by the policy network, that is, the placement location information.
[0087] C4: If the number of GPUs currently inferred is less than the number of GPUs required for the current job, repeat step C3.
[0088] C5: Place the corresponding job according to the placement location information inferred in step C3.
[0089] In this embodiment, the GPU is the smallest scheduling unit, that is, a GPU cannot be allocated to multiple jobs. In other embodiments, the computing unit may also use a TPU (tensor processing unit) and an MLU (machine learning processor).
[0090] According to the method for placing deep learning training job resources based on reinforcement learning, this embodiment also provides a system for placing deep learning training job resources based on reinforcement learning, including a DRL neural network model and a job scheduling module. The job scheduling module uses the steps of the above method to train the DRL neural network model. The job scheduling module also uses the steps of the above method to obtain placement location information from the trained DRL neural network model, and places the corresponding job according to the placement location information.
[0091] The above are only embodiments of the present invention. The invention is not limited to the fields involved in this implementation case. The common sense such as the known specific structures and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to explain the content of the claims.
Claims
1. A method for placing deep learning training job resources based on reinforcement learning. It is characterized in that The steps include: Initialization step: Randomly initialize the parameters of the DRL neural network model; State vector generation step: generating the state vector of the batch job; Reasoning step: Send the state vector to the DRL neural network model to infer the placement information of the batch job, and place the job according to the placement information to obtain the maximum completion time of the batch job operation, which is recorded as T_RL; Random generation step: randomly generate several placement location information, and place the job according to the randomly generated placement location information to obtain several maximum completion times of the batch job, and obtain the smallest maximum completion time among them and record it as T_Random; Reward calculation steps: Calculate the reward based on the maximum completion time T_RL and the maximum completion time T_Random; Parameter updating step: Backward gradient updates the parameters of the DRL neural network model.
2. According to the method for placing deep learning training job resources based on reinforcement learning in claim 1, Features: It also includes an experience replay step: sampling the four-tuple samples generated during the DRL neural network model training process for experience replay.
3. According to the method for placing deep learning training job resources based on reinforcement learning in claim 1, Features: In the state vector generation step, a state vector is generated based on the DLT job information and cluster information, and is denoted as Among them, N is the number of computing units required for the current job; T is the estimated running time of the current job without errors; S is the usage status of each computing unit in the current cluster.
4. The method for placing deep learning training job resources based on reinforcement learning according to claim 3, Features: The reasoning steps specifically include: A1: Input the state vector into the value network of the DRL neural network model to obtain the long-term measurement index V; A2: Input the state vector into the policy network of the DRL neural network model to obtain the selection probability P of N computing units i , where i = 1, 2, ..., N; A3: The probability P corresponding to the occupied computing unit and the failed computing unit j Set to zero and get P′ i ; A4: Select the kth computing unit as one of the computing units to be placed for the job, where P k =max(P′ i ); A5: If the number of computing units to be placed for the job is equal to the number of computing units required for the job, the placement location information reasoning for the job is completed, and the location information of the next job is inferred. Otherwise, jump to step A1.
5. The method for placing deep learning training job resources based on reinforcement learning according to claim 2, Features: The experience replay step specifically includes: B1: Create a playback buffer pool stack; B2: Push the four-tuple samples generated during the training process into the revisit buffer pool stack; B3: If the revisit buffer pool is full, the earliest four-tuple sample pushed into the stack will overflow; B4: Select X four-tuple samples as a batch for the next training, where X is the total number of four-tuple samples in the current revisit buffer pool stack.
6. The method for placing deep learning training job resources based on reinforcement learning according to claim 1, Features: In the reward calculation step, the reward calculation formula is:
7. The method for placing deep learning training job resources based on reinforcement learning according to claim 3, Features: It also includes a training judgment step: judging whether the DRL neural network model has been trained, if not, returning to the state vector generation step, otherwise the training ends.
8. The method for placing deep learning training job resources based on reinforcement learning according to claim 7, Features: The method also includes the steps of using the trained DRL neural network model to infer the placement position of each job in the batch job.
9. The method for placing deep learning training job resources based on reinforcement learning according to claim 8, Features: The use steps specifically include: C1: Get the job information and cluster information of the batch job; C2: Generate a state vector based on the information collected in step C1; C3: Input the state vector in step C2 into the policy network of the DRL neural network model to obtain the placement position information output by the policy network; C4: If the number of computing units currently inferred is less than the number of computing units required for the current job, repeat step C3; otherwise, jump to step C5; C5: Place the corresponding job according to the placement location information inferred in step C3.
10. A deep learning training job resource placement system based on reinforcement learning, including a DRL neural network model and a job scheduling module; It is characterized in that The job scheduling module uses the steps of the method described in any one of claims 1-8 to train the DRL neural network model, obtains placement location information from the trained DRL neural network model, and places corresponding jobs according to the placement location information.
Citation Information
Patent Citations
Method and apparatus for task scheduling based on deep reinforcement learning, and device
US20210081787A1
Reinforcement learning using meta-learned intrinsic rewards
US20210089910A1
