Container cloud task scheduling method and device based on deep reinforcement learning

Abstract

The application provides a container cloud task scheduling method based on deep reinforcement learning, comprising the following steps: defining a container cloud task scheduling event; modeling a Markov decision model, comprising: taking the task demand of each task and the current resource state of each available node as a state set, and taking the set of available nodes as an action set; an agent schedules a task to a certain available node according to the task demand of the task, the current resource state of the available node and a task scheduling strategy; defining a reward function for calculating a reward after the agent schedules the task to the available node; and optimizing the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy. The application also provides a container cloud task scheduling device based on deep reinforcement learning, a storage medium and an electronic device. Therefore, the application can automatically learn an expected scheduling strategy from the running information of the system, and automatically adapt to a changing environment without relying on expert knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence technology and container resource scheduling, and particularly relates to a container cloud task scheduling method and device based on deep reinforcement learning, a storage medium and an electronic device. BACKGROUND

[0002] With the development of cloud computing and container ecology, cloud native has become a new computing paradigm. Developers build customized applications following the microservice architecture, encapsulate them as containers, and submit them to cloud servers for running. Under this paradigm, developers can focus on developing and delivering microservices without worrying about the underlying infrastructure. Since the microservice architecture greatly increases the number of microservices, how to efficiently manage these microservices becomes an important research problem. Usually, these microservices are managed and scheduled by a container orchestration framework.

[0003] Resource scheduling is a key problem in managing large clusters. A good scheduling algorithm can help enterprises to reasonably schedule resources for each microservice, improve the resource utilization of the cluster, and thus reduce the cost of enterprises. In the industry and academia, the resource scheduling problem is considered as an NP (Nondeterministic Polynomial) hard problem. In different scenarios, due to different workloads of clusters, different application characteristics, and different scheduling goals, it is difficult to design a universal scheduling algorithm. For the resource scheduling problem, the traditional solution is to design a heuristic algorithm. However, heuristic algorithms usually use fixed parameters and scheduling strategies, and cannot adapt to changing environments, so they cannot achieve ideal performance. Some researches propose meta-heuristic algorithms to improve heuristic algorithms, which combine heuristic algorithms with random algorithms and local search algorithms, such as ant colony algorithm, simulated annealing and genetic algorithm, etc. However, it is complex and tedious to design these meta-heuristic algorithms.

[0004] In summary, the prior art has obvious inconvenience and defects in actual use, so it is necessary to improve. SUMMARY

[0005] In view of the above defects, the purpose of the present application is to provide a container cloud task scheduling method and device based on deep reinforcement learning, a storage medium and an electronic device, which can automatically learn the expected scheduling strategy from the running information of the system, and automatically adapt to changing environments without relying on expert knowledge.

[0006] To solve the above technical problems, the present application is implemented as follows:

[0007] In a first aspect, the embodiments of the present application provide a container cloud task scheduling method based on deep reinforcement learning, comprising:

[0008] The task scheduling definition steps define container cloud task scheduling items, which include task attributes, node attributes, and task scheduling strategies.

[0009] The MDP modeling steps involve modeling the container cloud task scheduling event as a Markov decision model, including: using the task requirements of each task and the current resource status of each available node as a state set, and the set of available nodes as an action set; the agent schedules at least one task to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes, and the task scheduling strategy; and defining a reward function to calculate the reward after the agent schedules the task to the available node.

[0010] The reinforcement learning optimization step involves optimizing the Markov decision model using a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0011] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the task attributes include the task type, task requirements and / or priority of the task; the node attributes include the computing power, resource status, storage capacity and / or network bandwidth of each available node in the computing cluster; the task scheduling strategy includes determining how to allocate tasks and scheduling targets according to task requirements and node resources.

[0012] The task scheduling definition steps include:

[0013] After receiving the task request, the container cloud creates a corresponding task for the task request;

[0014] The intelligent agent generates the corresponding task scheduling strategy based on the task requirements and the current resource status of the available nodes;

[0015] Define the set of available nodes in the computing cluster as Q = {q1, q2, ..., q}. m}, where m = |Q| represents the number of available nodes, and each available node has different node attributes;

[0016] Define a set of input tasks Θ = {θ1, θ2, ..., θ} n}, where n = |Θ| represents the number of tasks, and each task has different task attributes;

[0017] Describe the task θ i The scheduling is ξ i ={θ i q jwhere j is the index of the selected available node, and the agent needs to perform task replacement to find another available node suitable for task allocation if the scheduling fails due to violation of resource constraints;

[0018] The total response time of the task is defined as

[0019]

[0020] wherein, represents the task queuing time, represents the task execution time;

[0021] The task scheduling strategy is defined, and the task scheduling goal is defined as minimizing the total response time of all tasks and reducing the occurrence of task replacement caused by scheduling failure, that is, wherein T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, wherein is the response time of the i-th task, and n is the total number of tasks.

[0022] According to the container cloud task scheduling method based on deep reinforcement learning, the Markov decision model is represented as (S, A, P, R, γ), wherein S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0, 1] is a discount factor, which represents the preference for future rewards;

[0023] According to the container cloud task scheduling method based on deep reinforcement learning, the reinforcement learning optimization step comprises:

[0024] The deep reinforcement learning model is established, which combines deep Q network and Rainbow method;

[0025] The minimization loss function of the deep reinforcement learning model is set, and a deep neural network is used to approximate the action value function Q(s, a; θ), and the target expression is the minimization loss function:

[0026] L(θ)=E[(r+γmaxQ(s′,a′;θ-)-Q(s,a;θ)) 2 ]

[0027] wherein (r+γmaxQ(s′,a′;θ - ) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, θ - represents the parameters of the target network, and the target network is regularly updated to stabilize the learning process;

[0028] Train the deep reinforcement learning model;

[0029] The task sequence to be processed is input into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

[0030] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the reinforcement learning optimization step includes:

[0031] In discrete time step t, the deep reinforcement learning model observes state s. t From the task scheduling strategy π(a) t |s t Choose an action a from the list. t Execute and transition to the new state. t +1, and receive reward r t The objective of the deep reinforcement learning model is to maximize the expected reward:

[0032] V π (s t ) = E π [Σ t γ t r t ], t∈T

[0033] Where V n (s t ) is the agent from state s t The sum of the discount rewards obtained by following the task scheduling strategy π, E π It is the expectation operator under the task scheduling strategy π. Summing is performed on time step t, γ t It is the discount factor, r t It is the immediate reward obtained at time step t, where t∈T means that time step t belongs to a set of time steps T.

[0034] Secondly, embodiments of the present invention provide a container cloud task scheduling device based on deep reinforcement learning constructed according to any one of the above methods, comprising:

[0035] The task scheduling definition module is used to define container cloud task scheduling items, which include task attributes, node attributes, and task scheduling strategies.

[0036] The MDP modeling module is configured to model the container cloud task scheduling event as a Markov decision model, including: taking the task demand of each task and the current resource state of each available node as a state set, taking the set of available nodes as an action set; an agent schedules at least one task to a certain available node according to the task demand of the at least one task, the current resource state of all available nodes and the task scheduling strategy; and defining a reward function for calculating a reward after the agent schedules the task to the available node.

[0037] The reinforcement learning optimization module is configured to optimize the Markov decision model by a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0038] In a third aspect, an embodiment of the present application provides a storage medium for storing a computer program for executing the deep reinforcement learning-based container cloud task scheduling method.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device including a storage medium, a processor and a computer program stored in the storage medium and executable on the processor, wherein the processor implements the deep reinforcement learning-based container cloud task scheduling method when executing the computer program.

[0040] In the embodiment of the present application, the present application provides a deep reinforcement learning-based container cloud task scheduling method, including: defining a container cloud task scheduling event; modeling as a Markov decision model, including: taking the task demand of each task and the current resource state of each available node as a state set, taking the set of available nodes as an action set; an agent schedules at least one task to a certain available node according to the task demand of the at least one task, the current resource state of all available nodes and the task scheduling strategy; defining a reward function for calculating a reward after the agent schedules the task to the available node; and optimizing the Markov decision model by a deep reinforcement learning model to obtain an optimized task scheduling strategy. The present application also provides a deep reinforcement learning-based container cloud task scheduling device, a storage medium and an electronic device. Therefore, the present application can automatically learn an expected scheduling strategy from the running information of the system, and automatically adapt to the changing environment without relying on expert knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 FIG. 1 is a flowchart of a deep reinforcement learning-based container cloud task scheduling method provided by an embodiment of the present application;

[0042] Figure 2 FIG. 1 is a structural diagram of a deep reinforcement learning-based container cloud task scheduling device provided by an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of a container cloud task scheduling device based on deep reinforcement learning provided by Embodiment Two of the present application;

[0044] Figure 4 is a hardware structure schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0046] It should be noted that the description of "one embodiment", "embodiment", "example embodiment" and the like in the specification means that the described embodiment can include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not mean the same embodiment. Further, when a specific feature, structure or characteristic is described in combination with an embodiment, it is indicated that such feature, structure or characteristic is combined into other embodiments within the knowledge of those skilled in the art, whether or not it is explicitly described.

[0047] In addition, some words are used in the specification and subsequent claims to refer to specific components or parts, and those skilled in the art should understand that manufacturers can use different names or terms to refer to the same component or part. The present specification and subsequent claims do not distinguish components or parts by name, but by functional differences. In the entire specification and subsequent claims, "including" and "containing" are open-ended terms, which should be interpreted as "including but not limited to". In addition, the word "connected" herein includes any direct and indirect electrical connection means. Indirect electrical connection means includes connection through other devices.

[0048] The container cloud task scheduling method based on deep reinforcement learning provided by the embodiments of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios.

[0049] In recent years, due to the excellent performance of reinforcement learning in sequence decision, it is applied to solve the resource scheduling problem of computing cluster. The advantages of reinforcement learning are introduced into container cloud scheduling, and a task scheduling method based on deep reinforcement learning is proposed. The DQN (Deep Q-Network, deep Q network) architecture is adopted, and the Rainbow DQN method is combined, which combines the advantages of multiple DQN algorithms, including double DQN, priority experience replay, multi-step learning, distributed RL, noise network, aiming at optimizing task scheduling in container cloud environment. The application can effectively adapt to the dynamics and heterogeneity of computing cluster, and through continuous interaction with the computing environment, it can adaptively learn the optimal task scheduling.

[0050] The container task scheduling problem is modeled as a Markov decision process. The DRS (Dynamic Resource Scheduler) takes the resource usage of each node and the resource demand of the pod (container group) as the state, and takes the schedulable node as the action space. After performing the action, the scheduler calculates the reward. Through this model, the scheduler can automatically learn the expected scheduling strategy from the running information of the system without relying on expert knowledge.

[0051] The container cloud task scheduling method based on deep reinforcement learning provided by the embodiment one of the application comprises the following steps:

[0052] Step S101, task scheduling definition step, defining container cloud task scheduling items, container cloud task scheduling items including task attributes, node attributes and task scheduling strategy.

[0053] Step S102, MDP modeling step, modeling the container cloud task scheduling items into a Markov decision model, including: taking the task demand of each task and the current resource state of each available node as the state set, taking the set of available nodes as the action set. The agent schedules at least one task to a certain available node according to the task demand of at least one task, the current resource state of all available nodes and the task scheduling strategy. Define a reward function for calculating the reward after the agent schedules the task to the available node.

[0054] Step S103, reinforcement learning optimization step, optimizing the Markov decision model through the deep reinforcement learning model to obtain the optimized task scheduling strategy.

[0055] Deep Reinforcement Learning (DRL) utilizes deep neural networks to handle decision-making problems with high-dimensional states, which can be formalized as Markov Decision Processes (MDPs). A Markov Decision Process is denoted as (S, A, P, R, γ), where S represents the state set, A represents the action set. P defines the transition probability, R is the reward function, and γ ∈ [0, 1] is the discount factor, representing the preference for future rewards.

[0056] Preferably, the task attributes include the task type, task requirements, and / or priority of the task. The node attributes include the computing power, resource status, storage capacity, and / or network bandwidth of each available node in the computing cluster. The task scheduling strategy includes deciding how to allocate tasks according to task requirements and node resources, as well as scheduling objectives.

[0057] Figure 1 is a flowchart provided by Embodiment Two of the present application, and the method includes the following steps:

[0058] Step S201, task scheduling definition step, defines container cloud task scheduling items, which include task attributes, node attributes, and task scheduling strategies.

[0059] Preferably, the task scheduling definition step includes:

[0060] After the container cloud receives a task request, it creates a corresponding task for the task request. Specifically, a user sends a task request from a local device to the container cloud, and a corresponding task is created for each incoming task request in the cloud.

[0061] Nodes (Nodes): Define the set of available nodes Q = {q1, q2,..., q m} in the computing cluster, where m = |Q| represents the number of available nodes, and each available node has different node attributes. Assume that each node has a separate quantum processing unit (QPU) that reflects the current state of the available computer. Each node has different attributes, including CPU utilization, memory utilization (Mem), network interface receive and transmit rates (Recv, Tran), and disk read and write rates (Read, Write).

[0062] Tasks (Tasks): Define a set of incoming tasks Θ = {θ1, θ2,..., θ n} where n = |Θ| represents the number of tasks, and each task has different task attributes.

[0063] The scheduling of task θ i is denoted as ξ i = {θ i , qj} where j is the index of the selected available node. If the scheduling fails due to the violation of the resource constraints, for example, placing a task to a node without enough resources, the agent needs to perform task replacement (or rescheduling) to find another available node that is suitable for assigning the task.

[0064] Define the total response time (or total makespan) as the total waiting time from the request submission to the completion, including the execution time, queuing time, as follows:

[0065]

[0066] where, represents the task queuing time (from the arrival time to the start time of execution), represents the task execution time.

[0067] Therefore, the present task can be described as designing a task scheduling strategy in a computing cluster with nodes and continuous incoming tasks, selecting the most suitable node for each incoming task, thereby minimizing the total response time of all tasks and reducing the frequency of task replacement due to constraint violations.

[0068] Define the task scheduling strategy, define the task scheduling goal as minimizing the total response time of all tasks, and reducing the occurrence of task replacement due to scheduling failure, that is, where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks.

[0069] Step S102, MDP modeling step, modeling the container cloud task scheduling matter as a Markov decision model, including: taking the task requirements of each task and the current resource state of each available node as the state set, taking the set of available nodes as the action set. The agent schedules at least one task to a certain available node according to the task requirements of at least one task, the current resource state of all available nodes, and the task scheduling strategy. Define a reward function for calculating the reward after the agent schedules a task to an available node.

[0070] Preferably, the Markov decision model is represented as (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ ∈ [0, 1] is the discount factor, representing the preference for future rewards.

[0071] Deep Reinforcement Learning (DRL) utilizes deep neural networks to handle decision problems with high-dimensional states, which can be formalized as Markov Decision Processes (MDPs). A Markov Decision Process is represented as (S, A, P, R, γ), where S represents the set of states, A represents the set of actions, P defines the transition probabilities, R is the reward function, and γ∈[0,1] is a discount factor representing the preference for future rewards.

[0072] Preferably, the step of using the task requirements of each task and the current resource status of each available node as a state set in the MDP modeling step includes:

[0073] The agent observes the state from the cloud container environment, including the task requirements for each task and the current resource state of each available node.

[0074] Assuming there are m available nodes in the current computing cluster Q, and each available node has o features, then at time step t, it can be represented as:

[0075]

[0076] Where i is the index of the available nodes, and z is the index of the node characteristics.

[0077] Suppose that task θ is in the current task set Θ. j If there are p features, then at time step t, it can be represented as:

[0078]

[0079] in, It is a set containing p elements, each element being... k is the index of the current task feature.

[0080] Information about each task is sent to the agent only as part of the state after the task is completed; therefore, the state set S is defined as follows:

[0081]

[0082] Where each element s in the state set S t ,Depend on Composition, encompassing all time The state s below t .

[0083] Preferably, the MDP modeling step includes using the set of available nodes as an action set, which includes:

[0084] (1) Define the action as placing the task on an available node in the computing cluster, action a tdenotes the action of placing task θ j at available node q i at time step t, the action set A is determined as:

[0085] a t ={q i , θ j}

[0086] The action set A is equivalent to the set of available nodes in the cluster, i.e., A = Q.

[0087] The total response time of a task is defined as where represents the task queuing time, represents the task execution time.

[0088] The task scheduling strategy is designed, and the scheduling goal is defined as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, i.e., where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks.

[0089] Preferably, the main goal of the entire algorithm is to minimize the total response time of all tasks. In addition, it also reduces the occurrence of task replacement due to scheduling failure (or maximizes the success rate of task scheduling).

[0090] Preferably, the reward function is defined in the MDP modeling step, and the reward step after the agent schedules a task to an available node includes:

[0091] First, the resource utilization of available node i at time t is represented as .

[0092]

[0093] where K represents the number of resource types, including processor (CPU), memory (Mem), reception rate (Recv), transmission rate (Tran), read rate (Read), and write rate (Write). represents the sum of all K resource types.

[0094] Then the reward r t at time t can be defined as follows:

[0095]

[0096] where, is the task θi total response time, a is a penalty factor, and k is the number of replacements.

[0097] If the task scheduling is successful (done = 1), the inverse of its total response time is assigned as the reward to encourage the policy to find better scheduling to achieve shorter total response time and higher reward r t .

[0098] On the contrary, if the task scheduling fails, a larger predetermined value Δ is used as the penalty, and the task scheduling policy is suggested to avoid similar actions in the future. In addition, the number of replacements (or the number of rescheduling) of the task θi up to k is considered, and a penalty factor a is defined when assigning the reward to reduce the number of replacements. The penalty factor acts as an additional discount factor when the task needs to be scheduled multiple times to be successful, and the predetermined value Δ of the penalty is amplified if the task fails multiple times. Therefore, the reward function set by the present application can be used to achieve the goal of minimizing the total response time and reducing the number of task replacements.

[0099] In step S203, a deep reinforcement learning model is established, which combines a deep Q network and a Rainbow method.

[0100] In step S204, a minimization loss function of the deep reinforcement learning model is set, and a deep neural network is used to approximate the action value function Q(s, a; θ). The target expression is to minimize the loss function:

[0101] L(θ) = E[(r + γmaxQ(s', a'; θ - ) - Q(s, a; θ)) 2 ]

[0102] where (r + γmaxQ(s, a; θ - ) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, θ - represents the parameters of the target network, which is updated regularly to stabilize the learning process.

[0103] In step S205, the deep reinforcement learning model is trained.

[0104] In step S206, the task sequence to be processed is input into the trained deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0105] Preferably, in a discrete time step t, the deep reinforcement learning model observes a state s t , selects an action a t from the task scheduling policy π(a t |s t ) to execute, and moves to a new state st +1, while receiving a reward r t The goal of a deep reinforcement learning model is to maximize the expected return as:

[0106] V π (s t )=E π [∑ t γ t r t ],t∈T

[0107] where V π (s t ) is the sum of discounted rewards that the agent would receive starting from state s t following a task scheduling policy π, E π is the expected operator under the task scheduling policy π, is summed over time steps t, γ t is a discount factor, r t is the immediate reward obtained at time step t, and t∈T indicates that time step t belongs to a set of time steps T.

[0108] It should be noted that the container cloud task scheduling method based on deep reinforcement learning provided in the embodiments of the present application can be executed by an electronic device, an apparatus, or a control module in the apparatus for executing the container cloud task scheduling method based on deep reinforcement learning. In the embodiments of the present application, the container cloud task scheduling method based on deep reinforcement learning is executed by a container cloud task scheduling apparatus based on deep reinforcement learning, and the container cloud task scheduling apparatus based on deep reinforcement learning provided in the embodiments of the present application is described.

[0109] Figure 2 Fig. 1 is a structural schematic diagram of a container cloud task scheduling apparatus based on deep reinforcement learning provided in Embodiment I of the present application. The apparatus 100 comprises:

[0110] A task scheduling definition module 10 is configured to define a container cloud task scheduling matter, wherein the container cloud task scheduling matter comprises a task attribute, a node attribute, and a task scheduling policy.

[0111] An MDP modeling module 20 is configured to model the container cloud task scheduling matter as a Markov decision process, comprising: taking the task demand of each task and the current resource state of each available node as a state set, and taking the set of available nodes as an action set. An agent schedules at least one task to a certain available node according to the task demand of at least one task, the current resource state of all available nodes, and the task scheduling policy. A reward function is defined to calculate a reward after the agent schedules the task to the available node.

[0112] The reinforcement learning optimization module 30 is configured to optimize the Markov decision model by a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0113] Figure 3 is a structural schematic diagram of the container cloud task scheduling device based on deep reinforcement learning provided in Embodiment Two of the present application. The device 100 comprises:

[0114] The task scheduling definition module 10 is configured to define a container cloud task scheduling event, wherein the container cloud task scheduling event comprises task attributes, node attributes and a task scheduling strategy.

[0115] Preferably, the task attributes comprise a task type, a task demand and / or a priority of the task. The node attributes comprise a computing capability, a resource state, a storage capacity and / or a network bandwidth of each of the available nodes in the computing cluster. The task scheduling strategy comprises a decision on how to allocate tasks according to the task demand and the node resource and a scheduling target.

[0116] Preferably, the task scheduling definition module 10 is configured to:

[0117] After the container cloud receives a task request, a corresponding task is created for the task request.

[0118] The agent generates a corresponding task scheduling strategy according to the task demand of the task and the current resource state of the available nodes.

[0119] The available node set Q = {q1, q2,..., q m} in the computing cluster is defined, wherein m = |Q| represents the number of available nodes, and each available node has different node attributes.

[0120] A group of incoming tasks Θ = {θ1, θ2,..., θ n} is defined, wherein n = |Θ| represents the number of tasks, and each task has different task attributes.

[0121] The scheduling of the task θ i is described as ξ i = {θ i , q j}, wherein j is the index of the selected available node. If the scheduling fails due to violation of the resource constraint, the agent needs to perform task replacement to find another available node suitable for allocating the task.

[0122] The total response time of the task is defined as

[0123]

[0124] wherein, This represents the queuing time for the task. This represents the task execution time.

[0125] The task scheduling strategy is defined, with the task scheduling objective defined as minimizing the total response time of all tasks and reducing the occurrence of task replacement due to scheduling failures. Where T(Ξ) represents the minimization of the total response time for all tasks. This represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks.

[0126] The MDP modeling module 20 is used to model the container cloud task scheduling event as a Markov decision model, including: using the task requirements of each task and the current resource status of each available node as a state set, and the set of available nodes as an action set. The agent schedules at least one task to a specific available node based on the task requirements of at least one task, the current resource status of all available nodes, and the task scheduling strategy. A reward function is defined to calculate the reward after the agent schedules the task to the available node.

[0127] Preferably, the Markov decision model is represented as (S,A,P,R,γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0,1] is a discount factor representing the preference for future rewards.

[0128] The MDP modeling module 20 is used for:

[0129] The agent observes the state from the cloud container environment, including the task requirements for each task and the current resource state of each available node.

[0130] Assuming there are m available nodes in the current computing cluster Q, and each available node has o features, then at time step t, it can be represented as:

[0131]

[0132] Where i is the index of the available nodes, and z is the index of the node characteristics.

[0133] Suppose that task θ is in the current task set Θ. j If there are p features, then at time step t, it can be represented as:

[0134]

[0135] in, It is a set containing p elements, each element being... k is the index of the current task feature.

[0136] The information of each task is only sent to the agent as part of the state after the task arrives, so the state set S is defined as:

[0137]

[0138] where each element s in the state set S t , consists of all the states s at time

[0139] t .

[0140] Preferably, the MDP modeling module 20 is further configured to:

[0141] define the action as placing the task on the available nodes of the computing cluster, the action a t represents the action of placing the task θ j to the available node q i at time step t, determine the action set A as:

[0142] a t = {q i , θ j}

[0143] The action set A is equivalent to the set of available nodes Q in the computing cluster, i.e. A = Q.

[0144] define the total response time of the task as where represents the task queuing time, represents the task execution time.

[0145] design the task scheduling strategy, define the scheduling goal as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, i.e. where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks.

[0146] Preferably, the MDP modeling module 20 is further configured to:

[0147] firstly represent the resource utilization of the available node i at time t as .

[0148] ​

[0149] where K represents the number of resource types, including processor (CPU), memory (Mem), receiving rate (Recv), transmission rate (Tran), reading rate (Read) and writing rate (Write). represents the summation of all K resource types.

[0150] The reward r t is defined as follows:

[0151]

[0152] where, is the total response time of task θ i , α is the penalty factor, and κ is the number of replacements.

[0153] If the task scheduling is successful, the reciprocal of its total response time is assigned as the reward to encourage the policy to find a better scheduling to obtain a shorter total response time and a higher reward r t .

[0154] If the task scheduling fails, a predetermined value Δ is used as a penalty to suggest the task scheduling policy to avoid similar actions in the future. The penalty factor acts as an additional discount factor when the task needs to be scheduled multiple times to be successful, and the predetermined value Δ of the penalty is amplified if the task fails multiple times.

[0155] The reinforcement learning optimization module 30 is configured to optimize the Markov decision model by a deep reinforcement learning model to obtain an optimized task scheduling policy.

[0156] Preferably, the reinforcement learning optimization module 30 further comprises:

[0157] A model establishing submodule 31 is configured to establish the deep reinforcement learning model, which combines a deep Q network and a Rainbow method.

[0158] A function setting submodule 32 is configured to set a minimization loss function of the deep reinforcement learning model, and a deep neural network is used to approximate an action value function Q(s, a; θ), whose target expression is the minimization loss function:

[0159] L(θ) = E[(r + γmaxQ(s′, a′; θ 2 - Q(s, a; θ))

[0160] where (r + γmaxQ(s′, a′; θ -The objective value of the state-action pair (s,a) is defined, where r is the reward, γ is the discount factor, s' is the subsequent state, and θ is the target value. - This represents the parameters of the target network, which is updated periodically to stabilize the learning process.

[0161] The model training submodule 33 is used to train the deep reinforcement learning model.

[0162] The strategy optimization submodule 34 is used to input the task sequence to be processed into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

[0163] Preferably, the reinforcement learning optimization step includes:

[0164] In discrete time step t, the deep reinforcement learning model observes state s. t From the task scheduling strategy π(a) t |s t Choose an action a from the list. t Execute and transition to the new state. t +1, and receive a reward r t The objective of the deep reinforcement learning model is to maximize the expected reward:

[0165] V π (st)=E π [∑ t γ t r t ], t∈T

[0166] Where V π (s t ) is the agent from state s t The sum of the discount rewards obtained by starting to follow the task scheduling strategy π, E π It is the expectation operator under the task scheduling strategy π. Summing is performed on time step t, γ t It is the discount factor, r t It is the immediate reward obtained at time step t, where t∈T means that time step t belongs to a set of time steps T.

[0167] The container cloud task scheduling device based on deep reinforcement learning provided in this embodiment of the invention can achieve... Figure 1 The various processes implemented in the embodiment of the container cloud task scheduling method based on deep reinforcement learning shown are not described in detail here to avoid repetition.

[0168] The container cloud task scheduling device based on deep reinforcement learning provided by the embodiment of the application comprises: a task scheduling definition module configured to define a container cloud task scheduling event; an MDP modeling module configured to model as a Markov decision model, comprising: taking the task demand of each task and the current resource state of each available node as a state set, and taking the set of available nodes as an action set; an agent schedules the tasks to certain available nodes according to the task demand of the tasks, the current resource state of the available nodes and a task scheduling strategy; a reward function is defined, configured to calculate a reward after the agent schedules the tasks to the available nodes; and a reinforcement learning optimization module configured to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy. The application further provides a container cloud task scheduling device based on deep reinforcement learning, a storage medium and an electronic device. Therefore, the application can automatically learn an expected scheduling strategy from the running information of the system, and automatically adapt to the changing environment without relying on expert knowledge.

[0169] The application further provides a storage medium configured to store the computer program. Figure 1 The computer program can be computer program instructions, which, when executed by a computer, can call or provide the method and / or technical solutions according to the application through the operation of the computer, and can achieve the same technical effects. To avoid repetition, details are not described here. The program instructions of the method of the application can be stored in a fixed or removable storage medium, and / or transmitted and / or stored in a storage medium of an electronic device running according to the program instructions through a data stream in a broadcast or other signal bearing medium.

[0170] According to one embodiment of the application, the application further provides a computer program product comprising a computer program. Figure 4The electronic device 400 shown can optionally include a storage medium 200 for storing a computer program and a processor 300 for executing the computer program, wherein when the computer program is executed by the processor 300, any one of the above container cloud task scheduling methods based on deep reinforcement learning is implemented, triggering the electronic device 400 to execute the method and / or technical solutions in the foregoing embodiments, and the same technical effects can be achieved. To avoid repetition, they will not be described here. It should be noted that the electronic device in the embodiments of the present application includes mobile electronic devices and non-mobile electronic devices. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer, a netbook, or a personal digital assistant, etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the embodiments of the present application are not limited specifically.

[0171] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented by using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer readable recording medium, such as a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit cooperating with the processor to perform the respective steps or functions.

[0172] The present application can be implemented on a computer as a computer implemented method, or in dedicated hardware, or in a combination of both. Executable code for a method according to the present application, or parts thereof, can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Alternatively, the computer program product comprises non-transitory program code means stored on a computer readable medium for performing a method according to the present application when the program product is executed on a computer.

[0173] In an alternative embodiment, the computer program comprises computer program code means adapted to perform all the steps of a method according to the present application when the computer program is run on a computer. Alternatively, the computer program is embodied on a computer readable medium.

[0174] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Exclusionary language, such as "only", does not exclude the addition of other elements, unless the context clearly indicates otherwise. Furthermore, it is to be understood that the terminology "first", "second", and / or "third", and / or the like, is used in the description and claims merely for purposes of differentiation, and is not intended to limit the scope of the application to the described order or number of steps. Other than in the examples, or where otherwise indicated, all numbers expressing quantities of materials, quantities of steps, reaction conditions, and so forth used in the specification and claims are to be understood as being modified in all instances by the term "about". Accordingly, unless indicated to the contrary, the numerical parameters are approximations. As used herein, the expression "consisting essentially of limits the scope of a composition or method to the elements recited and those that do not materially affect the basic and novel characteristic(s) of the claimed application. Thus, for example, a composition consisting essentially of the elements as defined herein would not exclude trace amounts of other elements, provided that the trace amounts do not materially affect the basic and novel characteristics of the composition. In this specification, unless defined otherwise, all technical and scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Nothing herein is to be construed as limiting the scope of the application to the specific examples given. Specific examples are included to more clearly demonstrate and describe the application and the best mode of making and using the same. It will be appreciated that those skilled in the art will be able to devise numerous alternative examples without departing from the scope of the application.

[0175] It will be appreciated that those skilled in the art will be able to devise numerous alternative examples without departing from the scope of the application.

Claims

1. A container cloud task scheduling method based on deep reinforcement learning, characterized in that, The method comprises the following steps: a task scheduling definition step of defining a container cloud task scheduling event, the container cloud task scheduling event comprising a task attribute, a node attribute and a task scheduling strategy; an MDP modeling step of modeling the container cloud task scheduling event into a Markov decision model, comprising: taking the task demand of each task and the current resource state of each available node as a state set, taking the set of available nodes as an action set; an agent scheduling at least one task to a certain available node according to the task demand of at least one task, the current resource state of all available nodes and the task scheduling strategy; and defining a reward function for calculating a reward after the agent schedules the task to the available node; a reinforcement learning optimization step of optimizing the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy; the Markov decision model is represented as (S, A, P, R, γ), wherein S represents the state set, A represents the action set, P represents a transition probability, R represents the reward function, and γ∈[0, 1] is a discount factor representing a preference for future rewards; the step of taking the task demand of each task and the current resource state of each available node as the state set in the MDP modeling step comprises: the agent observes the state from the cloud container environment, including the task demand of each task and the current resource state of each available node; assuming that there are m available nodes in the current computing cluster Q, and each available node has o characteristics, at time step t, it can be represented as: wherein i is the index of the available node, and z is the index of the node characteristics; Assume a task θ in the current task set Θ j With p features, this can be represented at time step t as: wherein, is a set containing p elements, each element being k is an index of the current task feature; the information of each task is only sent to the agent as part of the state after the task arrives, so the state set S is defined as: wherein each element s in the state set S t , consisting of all times t. state s t ; the step of defining a reward function for calculating a reward after the agent schedules the task to the available node in the MDP modeling step comprises: Let us first denote the resource utilization of the available node i at time t by Ri(t). where K represents the number of resource types, including processor (CPU), memory (Mem), receive rate (Recv), transmit rate (Tran), read rate (Read), and write rate (Write); denotes the summation over all K resource types; So the reward r at time t can be written as t is defined as follows: wherein, is the total response time of the task θ i , α is a penalty factor, and κ is the number of substitutions; If the task scheduling is successful, the inverse of its total response time is assigned as the reward to encourage the policy to find better scheduling to achieve shorter total response time and higher reward r t ; if the task scheduling fails, a predetermined value Δ is adopted as a penalty, and the task scheduling strategy is suggested to avoid similar actions in the future, the penalty factor acts as an additional discount factor when the task needs to be scheduled multiple times to be successful, and the predetermined value Δ of the penalty is amplified if the task fails multiple times; the reinforcement learning optimization step comprises: establishing the deep reinforcement learning model, which combines a deep Q network and a Rainbow method; setting a minimization loss function of the deep reinforcement learning model, and using a deep neural network to approximate an action value function Q(s, a; θ), the target expression of which is the minimization loss function: L(0) = E[(r + γ max Q(s', a'; 0) - Q(s, a; 0)) - 2 ]​ where (r + γmaxQ(s', a'; θ - ) defines the target value for state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, θ - denotes the parameters of the target network, which is periodically updated to stabilize the learning process; training the deep reinforcement learning model; inputting a task sequence to be processed into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

2. The container cloud task scheduling method based on deep reinforcement learning according to claim 1, characterized in that, the task attribute comprises a task type, a task demand and / or a priority of the task; and the node attribute comprises a computing capability, a resource state, a storage capacity and / or a network bandwidth of each available node in the computing cluster; the task scheduling strategy comprises a decision on how to allocate tasks according to task demands and node resources and a scheduling target. The task scheduling defining step comprises: After the container cloud receives a task request, a corresponding task is created for the task request; The agent generates a corresponding task scheduling strategy according to the task demand of the task and the current resource state of the available nodes; Define the set of available nodes in the computing cluster as Q = {q1, q2, ..., q}. m }, where m = |Q| represents the number of available nodes, and each available node has different node attributes; Definition of a set of incoming tasks Θ = {θ1, θ2,..., θn} where n = |Θ| denotes the number of tasks, and each task has different said task attributes; n} where n = |Θ| denotes the number of tasks, and each task has different said task attributes; Description of the task θ i The schedule for ξ i = {θ i , q j}, where j is the index of the selected available node, if the schedule fails due to violation of resource constraints, the agent needs to perform task replacement to find another available node suitable for assigning the task; The total response time of the task is defined as wherein, represents the task queuing time, represents the task execution time; defining the task scheduling policy, defining a task scheduling objective as minimizing the total response time of all tasks, and reducing the occurrence of task replacement due to scheduling failure, i.e. where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks. 3.The container cloud task scheduling method based on deep reinforcement learning according to claim 1, characterized in that, In the MDP modeling step, the set of available nodes is taken as the action set, which comprises: Define an action as placing a task on an available node of a computing cluster, action a t denote the action of placing a task θ j at time step t, to an available node q i The set of actions A is determined as: a t = {q i , θ j} The action set A is equivalent to the set of available nodes Q in the computing cluster, that is, A = Q; The total response time of the task is defined as wherein represents the task queuing time, represents the task execution time; The task scheduling strategy is designed, and the scheduling goal is defined as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, that is where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the i-th task, and n is the total number of tasks. 4.The container cloud task scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning optimization step comprises: In a discrete time step t, the deep reinforcement learning model observes a state s t from the task scheduling policy π(a t |s t ) and selects an action a t to perform and transitions to a new state s t +1 with a reward r t , the goal of the deep reinforcement learning model is to maximize the expected return as: V π (s t )=E π [∑ t γ t r t ],t∈T Where V π (s t ) is the agent from state s t The sum of the discount rewards obtained by starting to follow the task scheduling strategy π, E π It is the expectation operator under the task scheduling strategy π. Summing is performed on time step t, γ t It is the discount factor, r t It is the immediate reward obtained at time step t, where t∈T means that time step t belongs to a set of time steps T.

5. A container cloud task scheduling apparatus based on deep reinforcement learning constructed based on the method of any one of claims 1-4. It comprises: A task scheduling defining module is configured to define container cloud task scheduling items, wherein the container cloud task scheduling items comprise task attributes, node attributes and task scheduling strategies; An MDP modeling module is configured to model the container cloud task scheduling items into a Markov decision model, which comprises: taking the task demand of each task and the current resource state of each available node as a state set, taking the set of available nodes as an action set; an agent schedules at least one task to a certain available node according to the task demand of the at least one task, the current resource state of all available nodes and the task scheduling strategy; a reward function is defined to calculate the reward after the agent schedules the task to the available node; A reinforcement learning optimization module is configured to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

6. A storage medium, characterized by A computer program for executing the deep reinforcement learning-based container cloud task scheduling method according to any one of claims 1-4.

7. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the computer program to implement the deep reinforcement learning-based container cloud task scheduling method according to any one of claims 1-4.

Images