Distributed heterogeneous resource scheduling method based on recursive reinforcement learning

Abstract

The invention discloses a distributed heterogeneous resource scheduling method based on recursive reinforcement learning, and the method comprises the following steps: firstly, defining a target and a performance standard of a distributed system, and then converting the target and the performance standard into a reinforcement learning problem; the method comprises the following steps of: scheduling resources of each node by utilizing a deep Q network (DQN) to ensure that the performance of a distributed system is always kept above a certain standard; the DQN uses a deep neural network to approximate a Q function, and the Q function Q (s, a) represents an expected accumulated reward which can be obtained after an action a is adopted in a state s, namely, the Q function Q (s, a) is used for estimating an expected reward obtained after a certain action is adopted in a given state; an RNN network is used to approximately estimate a Q function. According to the distributed heterogeneous resource scheduling method based on recursive reinforcement learning, scheduling optimization can be achieved in real time, resource allocation can be dynamically adjusted, the method adapts to the change requirements of tasks and the change conditions of resources, and the overall performance and the response speed of a system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed system resource scheduling, and particularly to a distributed heterogeneous resource scheduling method based on recursive reinforcement learning. Background Art

[0002] With the rapid development of information technology, distributed systems have been widely applied in many fields, such as cloud computing, big data processing, distributed storage, etc. In these complex distributed systems, the efficiency of resource scheduling directly determines the overall performance of the system and the user experience. Distributed systems often involve thousands of tasks and resources, and there are significant differences in the task characteristics and resource configurations of each node, showing a high degree of heterogeneity, which poses many severe challenges to resource scheduling.

[0003] Traditional resource scheduling methods are unable to cope when faced with the huge state space of distributed systems. Since the state space covers a vast amount of information such as the hardware resources of nodes, load conditions, queue lengths, resource utilization rates, task execution durations, and historical state sequences, traditional algorithms are difficult to effectively learn and represent these complex relationships, resulting in a lack of accuracy and comprehensiveness in scheduling decisions. For example, in a large-scale data center, if a traditional simple rule-based scheduling algorithm is adopted, tasks may be allocated only based on the idle resources of the current node, ignoring the historical load fluctuations of the node and the correlations between tasks, thus causing resource waste or low task execution efficiency.

[0004] In a distributed system, scheduling decisions do not solely depend on the current state. Past factors such as task execution history and long-term resource utilization also have an important impact on decisions. However, traditional scheduling methods usually cannot capture these complex temporal relationships. Taking a distributed computing cluster as an example, when a new task arrives, traditional methods may not be able to reasonably allocate resources based on the execution efficiency of past similar tasks on different nodes, resulting in an extended task execution time and affecting the throughput of the entire system.

[0005] In addition, the dynamic nature of distributed systems is also a major problem faced by resource scheduling. The resource allocation, demands, and loads of each node may change at any time, and traditional scheduling methods lack sufficient adaptability. Under conditions of incomplete knowledge, traditional methods are difficult to gradually explore the optimal scheduling strategy through the observation and feedback of the system state. For example, in a cloud service platform, the business demands of users may fluctuate significantly in a short period, and traditional scheduling methods cannot adjust resource allocation in a timely manner, resulting in a decline in the service quality of some users. Summary of the Invention

[0006] In view of the above deficiencies in the prior art, the present invention provides a distributed heterogeneous resource scheduling method based on recursive reinforcement learning, aiming to solve problems such as complex state space, many factors dependent on scheduling decisions, and poor dynamic adaptability faced by resource scheduling in distributed systems, and to improve the overall performance and response speed of the system.

[0007] To achieve the above object of the invention, the technical solution adopted by the present invention is as follows:

[0008] A distributed heterogeneous resource scheduling method based on recursive reinforcement learning, comprising the following steps:

[0009] S1: First, define the goals and performance criteria of the distributed system, and then convert them into a reinforcement learning problem;

[0010] S2: Use a Deep Q Network (DQN) to schedule the resources of each node to ensure that the performance of the distributed system always remains above a certain standard;

[0011] S3: DQN uses a deep neural network to approximate the Q function. The Q function Q(s,a) represents the expected cumulative reward that can be obtained after taking action a in state s, that is, it is used to estimate the expected reward obtained after taking a certain action in a given state;

[0012] S4: Use an RNN network to approximately estimate the Q function.

[0013] Further, the goals and performance criteria of the distributed system include:

[0014] State space S t : Includes various types of information of the distributed system at time t;

[0015] Action space A t : The actions taken by the distributed system at time t.

[0016] Further, the various types of information of the distributed system at time t include the hardware resources of the nodes, the load conditions, the queue length or load distribution of the current system, the node resource utilization rate, the average task execution duration, and the historical state sequence.

[0017] Further, the actions taken by the distributed system at time t include the task allocation strategy of each node, the allocation of resources, the scheduling priority of tasks, and various system scheduling strategies.

[0018] Further, the expected cumulative reward reward function R t+1 :

[0019] The standard for evaluating the impact of each action on the system performance:

[0020] R t+1(s t , a t ) = -(ω1·C t + ω2·L t + ω3·RU t )

[0021] Where:

[0022] C t : CompletionTime, the completion time of executing a t in the state;

[0023] L t : LoadImbalance, the degree of load imbalance when executing a t in the state s t ;

[0024] RU t : ResourceUtilization, the degree of resource utilization when executing a t in the state s t ;

[0025] ω1, ω2, ω3: Hyperparameters that control the impact of each indicator on the reward.

[0026] Furthermore, the RNN network structure:

[0027] Input: State sequence - the states S of the past k steps t , S t-1 , …, S t-k ;

[0028] Hidden layer: Composed of RNN units;

[0029] Output layer: Outputs the Q values of each possible action;

[0030] Q θ (s t , a t ) = RNN(s t , s t-1 , s t-2 , …, s t-k , a t )

[0031] θ is the parameter of the current RNN;

[0032] Q-learning update rule:

[0033] Q θ (s t , a t ) ← Q θ (s t , at ) + α[r t+1 + γ·max a' Q(s t + 1, a′) - Q θ (s t , a t )]

[0034] Where:

[0035] Q θ (s t , a t ) is the Q - value for taking action a t in the current state s t ;

[0036] r t+1 is the immediate reward obtained by executing action a t ;

[0037] 0 < γ < 1 is the discount factor, usually set to 0.9 - 0.99;

[0038] max a’ Q(s t+1 , a’) is the maximum Q - value when executing the optimal action a’ in the next state s t+1 , representing the maximum possible future return, that is, the ideal resource scheduling scenario that the system will reach in the future;

[0039] α is the learning rate.

[0040] Furthermore, when using the RNN network, the following steps are executed:

[0041] A. Initialize Q θ (s, a), the main network parameter θ, and the target network parameter θ' = θ;

[0042] B. At each time step t, execute:

[0043] a. Calculate the Q - value from the current state s t and the historical state, and select an action a t , and the action selection is based on the ∈ - greedy policy, that is:

[0044]

[0045] b. Execute action a t , obtain the immediate reward r t+1 and the next state s t+1

[0046]

[0047] According to what is mentioned in the previous text:

[0048] r t+1 (s t ,a t ) = -(ω1·C t +ω2·L t +ω3·RU t )

[0049] c. Calculate the target value:

[0050] target t = r t+1 + γ·max a’ Q θ (s t+1 ,a’)

[0051] d. Calculate the Q - value loss:

[0052] Use the mean - squared error (MSE) loss function to calculate the error between the current Q - value and the target value:

[0053] L(θ) = (Q θ (s t ,a t ) - traget t ) 2

[0054] e. Update the network parameters:

[0055]

[0056] Where:

[0057] α is the learning rate, which controls the step size of each update;

[0058] is the gradient of the loss function with respect to the network parameter θ;

[0059] C. Regularly update the main network parameter θ' to the target network to improve training stability;

[0060] D. Continuously execute in each training cycle: Select an action → Execute the action and observe the reward → Calculate the target → Calculate the loss → Backpropagation and update the parameters → Synchronize the target network until the stop condition is reached.

[0061] The beneficial effects of the present invention are:

[0062] A distributed heterogeneous resource scheduling method based on recursive reinforcement learning of the present invention can achieve real - time scheduling optimization, dynamically adjust resource allocation, adapt to the changing requirements of tasks and the changing situation of resources, and improve the overall performance and response speed of the system.

[0063] Distributed systems usually involve thousands of tasks and resources, and the tasks and resources on different nodes are different. The state space is extremely large. Neural networks can be used to effectively learn and represent these complex relationships.

[0064] In a distributed system, scheduling decisions not only depend on the current system state but are also closely related to factors such as the past task execution history and resource utilization. An RNN can capture these complex temporal relationships to help optimize scheduling decisions.

[0065] In a distributed system, the resource allocation, demands, and loads of each node may change at any time. Through the observation of the system state and the feedback of the reward mechanism, DQN can gradually find the optimal scheduling strategy under the condition of incomplete knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flowchart of the distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to the present invention;

[0067] Figure 2 It is a flowchart executed during the training cycle of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0068] The technical solutions in the embodiments of the present invention will be described clearly and completely below.

[0069] As Figures 1 to 2 shown, a distributed heterogeneous resource scheduling method based on recursive reinforcement learning includes the following steps:

[0070] First, define the goals and performance criteria of the distributed system, and then convert them into a reinforcement learning problem.

[0071] Use a Deep Q Network (DQN) to schedule the resources of each node to ensure that the performance of the distributed system always remains above a certain standard.

[0072] DQN uses a deep neural network to approximate the Q function. The Q function Q(s,a) represents the expected cumulative reward that can be obtained after taking action a in state s, that is, it is used to estimate the expected reward obtained after taking a certain action in a given state.

[0073] Use an RNN network to approximately estimate the Q function.

[0074] State space S t : It contains various types of information of the distributed system at time t. Such as the hardware resources of the nodes, the load situation, the queue length or load distribution of the current system, the node resource utilization rate, the average task execution duration, the historical state sequence (the RNN needs to make decisions based on historical information), etc.

[0075] Action space A t : Actions that the distributed system can take at time t. Such as the task allocation strategy of each node (allocating a certain task to a certain node for execution), resource allocation (assigning different tasks to nodes with different resources for execution), task scheduling priority, various system scheduling strategies, etc.

[0076] Reward function R t+1 : A criterion for evaluating the impact of each action on system performance.

[0077] R t+1 = -(ω1·C t + ω2·L t + ω3·RU t )

[0078] Where:

[0079] C t : CompletionTime, the completion time of executing a t under state s t ;

[0080] L t : LoadImbalance, the degree of load imbalance when executing a t under state s t ;

[0081] RU t : ResourceUtilization, the degree of resource utilization when executing a t under state s t ;

[0082] ω1, ω2, ω3: Hyperparameters that control the impact of each indicator on the reward.

[0083] RNN network structure:

[0084] Input: State sequence - the states S of the past k steps t , S t-1 , …, S t-k ;

[0085] Hidden layer: Composed of RNN units;

[0086] Output layer: Outputs the Q value of each possible action.

[0087] Q θ (s t , a t ) = RNN(s t , s t-1 , s t-2 , …, s t-k,a t )

[0088] θ is the parameter of the current RNN.

[0089] Q-learning update rule:

[0090] Q θ (s t ,a t ) ← Q θ (s t ,a t ) + α[r t+1 + γ·max a' Q(s t + 1,a') - Q θ (s t ,a t )]

[0091] Where:

[0092] Q θ (s t ,a t ) is the Q-value of taking action a t under the current state s; t

[0093] r t+1 is the immediate reward obtained by executing action a; t

[0094] 0 < γ < 1 is the discount factor, usually set to 0.9 - 0.99;

[0095] max a’ Q(s t+1 ,a’) is the maximum Q-value when executing the optimal action a’ under the next state s, indicating the maximum possible future return, that is, the ideal resource scheduling scenario that the system will reach in the future; t+1

[0096] α is the learning rate.

[0097] When using the RNN network, the following steps are executed:

[0098] Initialize Q θ (s,a), the parameters θ of the main network, and the parameters θ' of the target network = θ;

[0099] Execute at each time step t:

[0100] (1). Calculate the Q-value from the current state s t and the historical state, and select an action a t . The action selection is based on the ∈-greedy policy, that is:

[0101]

[0102] (2). Execute action a t , obtain immediate reward r t+1 and next state s t+1

[0103]

[0104] According to what was mentioned previously:

[0105] r t+1 (s t ,a t ) = -(ω1·C t +ω2·L t +ω3·RU t )

[0106] (3). Calculate the target value:

[0107] target t = r t+1 +γ·max a’ Q θ (s t+1 ,a’)

[0108] (4). Calculate the Q-value loss:

[0109] Use the mean squared error (MSE) loss function to calculate the error between the current Q-value and the target value:

[0110] L(θ) = (Q θ (s t ,a t )-traget t ) 2

[0111] (5) Update network parameters:

[0112]

[0113] Where:

[0114] α is the learning rate, controlling the step size of each update;

[0115] is the gradient of the loss function with respect to the network parameter θ.

[0116] Regularly update the main network parameter θ' to the target network to improve training stability

[0117] Continuously execute in each training cycle: select an action → execute the action and observe the reward (i.e., the system performance) → calculate the target (expected computing performance) → calculate the loss (the difference between the actual expected situation and the actual situation of resources) → backpropagation and update parameters → synchronize the target network. Until the stop condition is reached (such as the number of training steps, system performance convergence, etc.).

[0118] A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to the present invention can achieve real-time scheduling optimization, dynamically adjust resource allocation, adapt to the changing requirements of tasks and the changing conditions of resources, and improve the overall performance and response speed of the system.

[0119] Distributed systems usually involve thousands of tasks and resources, and the tasks and resources on different nodes are different. The state space is very large. Neural networks can effectively learn and represent these complex relationships.

[0120] In a distributed system, the scheduling decision not only depends on the current system state, but is also closely related to factors such as the past task execution history and resource utilization. An RNN can capture these complex temporal relationships and help optimize the scheduling decision.

[0121] In a distributed system, the resource allocation, requirements, and load of each node may change at any time. Through the observation of the system state and the feedback of the reward mechanism, DQN can gradually find the optimal scheduling strategy under the condition of incomplete knowledge.

[0122] The above are only the preferred embodiments of the present invention for the patent, and are not intended to limit the present invention for the patent. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention for the patent shall be included within the protection scope of the present invention for the patent.

Claims

1. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning, characterized in that, It includes the following steps: S1: First, define the goals and performance criteria of the distributed system, and then convert them into a reinforcement learning problem; S2: Use a Deep Q Network (DQN) to schedule the resources of each node to ensure that the performance of the distributed system always remains above a certain standard; S3: DQN uses a deep neural network to approximate the Q function. The Q function Q(s,a) represents the expected cumulative reward that can be obtained after taking action a in state s, that is, it is used to estimate the expected reward obtained after taking a certain action in a given state; S4: Use an RNN network to approximately estimate the Q function.

2. The distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 1, characterized in that: The goals and performance criteria of the distributed system include: State space s t : It contains various types of information of the distributed system at time t; Action space A t : The action taken by the distributed system at time t.

3. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 2, characterized in that: All types of information of the distributed system at time t include the hardware resources of the nodes, the load situation, the current system queue length or load distribution, the node resource utilization rate, the average task execution duration, and the historical state sequence.

4. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 2, characterized in that: The actions taken by the distributed system at time t include the task allocation strategy of each node, the allocation of resources, the scheduling priority of tasks, and various system scheduling strategies.

5. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 1, characterized in that: Expected cumulative reward reward function R t+1 : The criteria for evaluating the impact of each action on system performance: R t+1 (s t ,a t ) = -(ω1·C t + ω2·L t + ω3·RU t ) Among them: C t : CompletionTime, the completion time of executing a under the state t ; L t : LoadImbalance, state s t Under this condition, execute a t Degree of load imbalance at this time; RU t : ResourceUtilization, status s t Under this condition, execute a t Degree of resource utilization at this time; ω1, ω2, ω3: are hyperparameters that control the impact of each indicator on the reward.

6. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 1, characterized in that: RNN network structure: Input: State sequence - states S in the past k steps t , S t-1 , …, S t-k ; Hidden layer: Consists of RNN units; Output layer: Outputs the Q value of each possible action; Q θ (s t ,a t ) = RNN(s t ,s t-1 ,s t-2 ,…,s t-k ,a t ) θ is the parameter of the current RNN; Q-learning update rule: Q θ (s t ,a t ) ← Q θ (s t ,a t ) + α[r t+1 + γ·max a' Q(s t + 1,a′) - Q θ (s t ,a t )] Among them: Q θ (s t ,a t ) is the Q-value of the current state s t when taking action a t ; r t+1 is the immediate reward obtained t when performing action a; 0 < γ < 1 is the discount factor, usually set to 0.9 - 0.99; max a’ Q(s t+1 , a’) is the next state s t+1 The maximum Q value when performing the optimal action a’ next, representing the maximum possible future reward, that is, the ideal resource scheduling scenario that the system will reach in the future; α is the learning rate.

7. A distributed heterogeneous resource scheduling method based on recursive reinforcement learning according to claim 1, characterized in that: Execute the following steps when using the RNN network: A. Initialize Q θ (s,,), the main network parameter θ, and the target network parameter θ' = θ; B. Execute at each time step t: .Calculate the Q-value from the current state s t and the historical states, and select an action a t , and the action selection is based on the ∈-greedy policy, that is: b. Execute action a t , obtain immediate reward r t+1 and next state s t+1 According to what was mentioned above: r 1+1 (s t ,a t )=-(ω1·C t +ω2·L t +ω3·RU t ) c. Calculate the target value: target t = r t+1 + γ·max a’ Q θ (s t+1 , a’) d. Calculate the Q value loss: Use the mean squared error (MSE) loss function to calculate the error between the current Q value and the target value: L(θ) = (Q θ (s t , a t ) - target t ) 2 e. Update the network parameters: Among them: α is the learning rate that controls the step size of each update; is the gradient of the loss function with respect to the network parameter θ; C. Regularly update the main network parameter θ' to the target network to improve training stability; D. Continuously execute in each training cycle: Select an action → Execute the action and observe the reward → Calculate the target → Calculate the loss → Backpropagation and update the parameters → Synchronize the target network until the stop condition is reached.

Images