Container cloud task scheduling method and device based on deep reinforcement learning

Abstract

The invention provides a container cloud task scheduling method based on deep reinforcement learning. The method comprises the steps that container cloud task scheduling items are defined; modeling as a Markov decision model, which comprises the following steps: taking a task demand of each task and a current resource state of each available node as a state set, and taking a set of available nodes as an action set; the intelligent agent schedules the task to a certain available node according to the task requirement of the task, the current resource state of the available node and the task scheduling strategy; defining a reward function for calculating a reward after the agent schedules the task to an available node; and optimizing the Markov decision model through the deep reinforcement learning model to obtain an optimized task scheduling strategy. The invention further provides a container cloud task scheduling device based on deep reinforcement learning, a storage medium and electronic equipment. Therefore, according to the method, the expected scheduling strategy can be automatically learned from the operation information of the system, and the method automatically adapts to the changing environment without depending on expert knowledge.

Description

Technical Field

[0001] The present invention relates to the technical field of the combination of artificial intelligence technology and container resource scheduling, and in particular to a container cloud task scheduling method, device, storage medium and electronic device based on deep reinforcement learning. Background Art

[0002] With the development of cloud computing and container ecology, cloud native has become a new computing paradigm. Developers follow the microservice architecture to build customized applications, encapsulate them into containers, and submit them to cloud servers for operation. Under this paradigm, developers can focus on developing and delivering microservices without having to pay attention to the underlying infrastructure. Since the microservice architecture greatly increases the number of microservices, how to efficiently manage these microservices has become an important research issue. Usually, these microservices are managed and scheduled by the container orchestration framework.

[0003] Resource scheduling is a key issue in managing large clusters. A good scheduling algorithm can help enterprises reasonably schedule resources for each microservice, improve the resource utilization of the cluster, and thus reduce enterprise costs. In industry and academia, the resource scheduling problem is considered to be an NP (Nondeterministic Polynomial) hard problem. In different scenarios, due to different workloads of clusters, different application characteristics, and different scheduling goals, it is difficult to design a general scheduling algorithm. For the resource scheduling problem, the traditional solution is to design heuristic algorithms. However, heuristic algorithms usually use fixed parameters and scheduling strategies, which cannot adapt to changing environments and thus cannot achieve ideal performance. Some studies have proposed meta-heuristic algorithms to improve heuristic algorithms. They combine heuristic algorithms with random algorithms and local search algorithms, such as ant colony algorithms, simulated annealing, and genetic algorithms. However, designing these meta-heuristic algorithms is complex and cumbersome.

[0004] In summary, the prior art obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the invention

[0005] In view of the above-mentioned defects, the purpose of the present invention is to provide a container cloud task scheduling method, device, storage medium and electronic device based on deep reinforcement learning, which can automatically learn the expected scheduling strategy from the system operation information and automatically adapt to the changing environment without relying on expert knowledge.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] In a first aspect, an embodiment of the present invention provides a container cloud task scheduling method based on deep reinforcement learning, comprising:

[0008] A task scheduling definition step is to define container cloud task scheduling matters, wherein the container cloud task scheduling matters include task attributes, node attributes, and task scheduling strategies;

[0009] The MDP modeling step is to model the container cloud task scheduling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy; defining a reward function for calculating a reward after the intelligent agent schedules the task to the available node;

[0010] The reinforcement learning optimization step optimizes the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0011] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the task attributes include the task type, task requirements and / or priority of the task; the node attributes include the computing power, resource status, storage capacity and / or network bandwidth of each available node in the computing cluster; the task scheduling strategy includes determining how to allocate tasks and scheduling targets according to task requirements and node resources;

[0012] The task scheduling definition step includes:

[0013] After receiving the task request, the container cloud creates a corresponding task for the task request;

[0014] The intelligent agent generates the corresponding task scheduling strategy according to the task requirements of the task and the current resource status of the available nodes;

[0015] Define the available node set Q in the computing cluster = {q1, q2, ..., q m}, wherein m=|Q| represents the number of available nodes, and each available node has different node attributes;

[0016] Define a set of incoming tasks Θ = {θ1,θ2,...,θ n}, wherein n=|Θ| represents the number of tasks, and each task has different task attributes;

[0017] Describe the task θ i The scheduling is ξ i ={θ i ,q j}, where j is the index of the selected available node. If the scheduling fails due to violation of resource constraints, the agent needs to perform task replacement to find another available node suitable for assigning tasks;

[0018] The total response time of the task is defined as

[0019]

[0020] in, Represents the task queue time, Represents the task execution time;

[0021] The task scheduling strategy is defined, and the task scheduling goal is defined as minimizing the total response time of all tasks and reducing the occurrence of task replacement caused by scheduling failure, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

[0022] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the Markov decision model is expressed as (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0,1] is a discount factor, which represents a preference for future rewards;

[0023] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the reinforcement learning optimization step includes:

[0024] Establishing the deep reinforcement learning model, wherein the deep reinforcement learning model combines the deep Q network and the Rainbow method;

[0025] The minimization loss function of the deep reinforcement learning model is set, and the action value function Q(s, a; θ) is approximated by a deep neural network, and its goal is expressed as the minimization loss function:

[0026] L(θ)=E[(r+γγmaxQ(s′,a′;θ-)-Q(s,a;θ)) 2 ]

[0027] Among them, (r+γmaxQ(s′,a′;θ - ) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, and θ - represents the parameters of the target network, which is updated regularly to stabilize the learning process;

[0028] Training the deep reinforcement learning model;

[0029] The task sequence to be processed is input into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

[0030] According to the container cloud task scheduling method based on deep reinforcement learning of the present invention, the reinforcement learning optimization step includes:

[0031] At discrete time step t, the deep reinforcement learning model observes the state s t , from the task scheduling strategy π(a t |s t ) Select an action a t Execute and transfer to the new state s t +1, and get reward r t , the goal of the deep reinforcement learning model is to maximize the expected return as:

[0032] V π (s t )=E π [Σ t γ t r t ], t∈T

[0033] Where V n (s t ) is the agent from state s t The sum of discounted rewards obtained by starting to follow the task scheduling strategy π, E π is the expected operator under the task scheduling strategy π, is the sum over time steps t, γ t is the discount factor, r t is the immediate reward obtained at time step t, and t∈T means that time step t belongs to a time step set T.

[0034] In a second aspect, an embodiment of the present invention provides a container cloud task scheduling device based on deep reinforcement learning constructed based on any of the above methods, including:

[0035] A task scheduling definition module is used to define container cloud task scheduling matters, wherein the container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies;

[0036] The MDP modeling module is used to model the container cloud task scheduling items as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy; defining a reward function for calculating the reward after the intelligent agent schedules the task to the available node;

[0037] The reinforcement learning optimization module is used to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0038] In a third aspect, an embodiment of the present invention provides a storage medium for storing a computer program for executing any of the above-mentioned container cloud task scheduling methods based on deep reinforcement learning.

[0039] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a storage medium, a processor, and a computer program stored in the storage and executable on the processor, wherein the processor implements any of the above-mentioned methods for scheduling container cloud tasks based on deep reinforcement learning when executing the computer program.

[0040] In an embodiment of the present invention, the present invention provides a container cloud task scheduling method based on deep reinforcement learning, including: defining container cloud task scheduling matters; modeling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules the task to a certain available node according to the task requirements of the task, the current resource status of the available nodes and the task scheduling strategy; defining a reward function for calculating the reward after the intelligent agent schedules the task to the available node; optimizing the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy. The present invention also provides a container cloud task scheduling device, storage medium and electronic device based on deep reinforcement learning. Thereby, the present invention can automatically learn the expected scheduling strategy from the system's operating information and automatically adapt to the changing environment without relying on expert knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a container cloud task scheduling method based on deep reinforcement learning provided in Embodiment 2 of the present invention;

[0042] Figure 2 It is a structural diagram of a container cloud task scheduling device based on deep reinforcement learning provided in Example 1 of the present invention;

[0043] Figure 3 It is a structural diagram of a container cloud task scheduling device based on deep reinforcement learning provided in Embodiment 2 of the present invention;

[0044] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment described, which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it has been shown that it is within the knowledge of those skilled in the art to combine such features, structures or characteristics into other embodiments.

[0047] In addition, certain words are used in the specification and subsequent claims to refer to specific components or parts. Those with ordinary knowledge in the relevant field should understand that manufacturers can use different nouns or terms to refer to the same component or part. This specification and subsequent claims do not use differences in names as a way to distinguish components or parts, but use differences in the functions of components or parts as the criteria for distinction. "Including" and "including" mentioned throughout the specification and subsequent claims are open-ended terms and should be interpreted as "including but not limited to". In addition, the word "connected" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connection through other devices.

[0048] In the following, in conjunction with the accompanying drawings, the container cloud task scheduling method based on deep reinforcement learning provided by the embodiment of the present invention is described in detail through specific embodiments and their application scenarios.

[0049] In recent years, due to the outstanding performance of reinforcement learning in sequential decision-making, it has been applied to solve the resource scheduling problem of computing clusters. The present invention introduces the advantages of reinforcement learning into container cloud scheduling and proposes a task scheduling method based on deep reinforcement learning. The DQN (Deep Q-Network) architecture is adopted, and the Rainbow DQN method is preferably combined, which integrates the advantages of various DQN algorithms, including dual DQN, priority experience replay, multi-step learning, distributed RL, and noise network, aiming to optimize task scheduling in container cloud environments. The present invention can effectively adapt to the dynamics and heterogeneity of computing clusters, and adaptively learn the optimal task scheduling through continuous interaction with the computing environment.

[0050] The container task scheduling problem is modeled as a Markov decision process. DRS (Dynamic Resource Scheduler) uses the resource usage of each node and the resource requirements of the pod (container group) as the state, and the schedulable nodes as the action space. After executing the action, the scheduler calculates the reward. Through this model, the scheduler can automatically learn the expected scheduling strategy from the system's operating information without relying on expert knowledge.

[0051] The container cloud task scheduling method based on deep reinforcement learning provided in the first embodiment of the present invention includes the following steps:

[0052] Step S101, task scheduling definition step, defines container cloud task scheduling matters, and container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies.

[0053] Step S102, MDP modeling step, models the container cloud task scheduling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as the state set, and taking the set of available nodes as the action set. The agent schedules at least one task to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy. Define a reward function for calculating the reward after the agent schedules the task to the available node.

[0054] Step S103, a reinforcement learning optimization step, optimizes the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0055] Deep reinforcement learning (DRL) uses deep neural networks to handle decision-making problems with high-dimensional states, which can be formalized as a Markov decision process (MDP). A Markov decision process is represented as (S, A, P, R, γ), where S represents the state set and A represents the action set. P defines the transition probability, R is the reward function, and γ∈[0,1] is the discount factor, which represents the preference for future rewards.

[0056] Preferably, the task attributes include the task type, task requirements and / or priority of the task. Node attributes include the computing power, resource status, storage capacity and / or network bandwidth of each available node in the computing cluster. Task scheduling strategy includes deciding how to allocate tasks and scheduling targets based on task requirements and node resources.

[0057] Figure 1 : is a flow chart of the second embodiment of the present invention, wherein the method comprises the following steps:

[0058] Step S201, task scheduling definition step, defines container cloud task scheduling matters, and container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies.

[0059] Preferably, the task scheduling definition step includes:

[0060] After receiving the task request, the container cloud creates a corresponding task for the task request. Specifically, the user sends a task request from the local device to the container cloud, and the cloud creates a corresponding task for each incoming task request.

[0061] Nodes: Defines the set of available nodes in the computing cluster Q = {q1,q2,...,q m}, where m = |Q| represents the number of available nodes, and each available node has different node attributes. It is assumed that each node has a separate quantum processing unit (QPU) that reflects the state of the currently available computer. Each node has different attributes, including CPU utilization, memory utilization (Mem), network interface receive and send rates (Recv, Tran), and disk read and write rates (Read, Write).

[0062] Tasks: Define a set of incoming tasks Θ = {θ1, θ2, ..., θ n}, where n = |Θ| represents the number of tasks, and each task has different task attributes.

[0063] Describe the task θ i The scheduling is ξ i ={θ i ,qj}, where j is the index of the selected available node. If scheduling fails due to violation of resource constraints, such as placing a task on a node without sufficient resources, the agent needs to perform task replacement (or rescheduling) to find another available node suitable for assigning the task.

[0064] Definition of total response time: (or total duration) represents the total waiting time from request submission to completion, including execution time and queue time, as follows:

[0065]

[0066] in, represents the task queuing time (from arrival time to execution start time), Represents the task execution time.

[0067] Therefore, this task can be described as designing a task scheduling strategy in a computing cluster with nodes and continuously incoming tasks to select the most suitable node for each incoming task, thereby minimizing the total response time of all tasks and reducing the frequency of task replacement due to constraint violations.

[0068] Define the task scheduling strategy and define the task scheduling goal as minimizing the total response time of all tasks and reducing the occurrence of task replacement caused by scheduling failure, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

[0069] Step S102, MDP modeling step, models the container cloud task scheduling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as the state set, and taking the set of available nodes as the action set. The agent schedules at least one task to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy. Define a reward function for calculating the reward after the agent schedules the task to the available node.

[0070] Preferably, the Markov decision model is expressed as (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0,1] is a discount factor, which represents the preference for future rewards.

[0071] Deep reinforcement learning (DRL) uses deep neural networks to handle decision-making problems with high-dimensional states, which can be formalized as a Markov decision process (MDP). A Markov decision process is represented as (S, A, P, R, γ), where S represents the state set and A represents the action set. P defines the transition probability, R is the reward function, and γ∈[0,1] is the discount factor, which represents the preference for future rewards.

[0072] Preferably, the step of taking the task requirements of each task and the current resource state of each available node as the state set in the MDP modeling step includes:

[0073] The agent observes the status from the cloud container environment, including the task requirements of each task and the current resource status of each available node.

[0074] Assuming that there are m available nodes in the current computing cluster Q, and each available node has o features, then at time step t, it can be expressed as:

[0075]

[0076] Where i is the index of an available node and z is the index of a node feature.

[0077] Assume that task θ in the current task set Θ j There are p features, then at time step t it can be expressed as:

[0078]

[0079] in, is a set containing p elements, each of which is k is the index of the current task feature.

[0080] The information of each task is sent to the agent as part of the state only after the task arrives, so the state set S is defined as:

[0081]

[0082] Among them, each element s in the state set S t ,Depend on Composition, including all time The state of t .

[0083] Preferably, in the MDP modeling step, taking the set of available nodes as an action set comprises:

[0084] (1) Define the action as placing the task on an available node in the computing cluster. Action a tIndicates that at time step t, the task θ j Place it on available node q i action, determine the action set A as:

[0085] a t = {q i ,θ j}

[0086] The action set A is equivalent to the set of available nodes Q in the computing cluster, that is, A=Q.

[0087] The total response time of the task is defined as in Represents the task queue time, Represents the task execution time.

[0088] Design a task scheduling strategy and define the scheduling goal as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

[0089] Preferably, the main goal of the entire algorithm is to minimize the total response time of all tasks. In addition, it is also necessary to reduce the occurrence of task replacement caused by scheduling failure (or maximize the success rate of task scheduling).

[0090] Preferably, the reward function is defined in the MDP modeling step, and the step of calculating the reward after the agent schedules the task to the available node includes:

[0091] First use To represent the resource utilization of available node i at time t.

[0092]

[0093] Wherein, K represents the number of resource types, including processor (CPU), memory (Mem), receiving rate (Recv), transmission rate (Tran), reading rate (Read) and writing rate (Write). Indicates the sum of all K resource types.

[0094] Then the reward r at time t can be t The definition is as follows:

[0095]

[0096] in, is the task θi is the total response time, α is the penalty factor, and κ is the number of replacements.

[0097] If the task is successfully scheduled (done = 1), the inverse of its total response time is assigned as a reward to encourage the strategy to find a better schedule to obtain a shorter total response time and a higher reward r t .

[0098] On the contrary, if the task scheduling fails, a larger predetermined value Δ is used as a penalty, and the task scheduling strategy is recommended to avoid similar actions in the future. In addition, the number of replacements (or rescheduling times) of the task θi that has reached κ is also considered, and a penalty factor α is defined when allocating rewards to reduce the number of replacements. The penalty factor acts as an additional discount factor when the task requires multiple scheduling to succeed, and if the task fails multiple times, the predetermined value Δ of the penalty is amplified. Therefore, the reward function set by the present invention can be used to achieve the goal of minimizing the total response time and reducing the number of task replacements.

[0099] Step S203: establishing a deep reinforcement learning model, which combines the deep Q network and the Rainbow method.

[0100] Step S204, setting the minimization loss function of the deep reinforcement learning model, using a deep neural network to approximate the action value function Q(s, a; θ), and its goal is expressed as minimizing the loss function:

[0101] L(θ)=E[(r+γmaxQ(s′,a′;θ - )-Q(s,a;θ)) 2 ]

[0102] Among them, (r+γmaxQ(s,a;θ - ) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, and θ - represents the parameters of the target network, which is updated regularly to stabilize the learning process.

[0103] Step S205, training a deep reinforcement learning model.

[0104] Step S206: input the task sequence to be processed into the trained deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0105] Preferably, at discrete time step t, the deep reinforcement learning model observes the state s t , from the task scheduling strategy π(a t |s t ) Select an action a t Execute and transfer to the new state st +1, and get reward r t , the goal of the deep reinforcement learning model is to maximize the expected return as:

[0106] V π (s t )=E π [∑ t γ t r t ], t∈T

[0107] Where V π (s t ) is the agent from state s t The sum of discounted rewards obtained by starting to follow the task scheduling strategy π, E π is the expected operator under the task scheduling strategy π, is the sum over time steps t, γ t is the discount factor, r t is the immediate reward obtained at time step t, and t∈T means that time step t belongs to a time step set T.

[0108] It should be noted that the container cloud task scheduling method based on deep reinforcement learning provided in the embodiment of the present invention can be executed by an electronic device, a device, or a control module in the device for executing the container cloud task scheduling method based on deep reinforcement learning. In the embodiment of the present invention, the container cloud task scheduling method based on deep reinforcement learning is executed by a container cloud task scheduling device based on deep reinforcement learning as an example to illustrate the container cloud task scheduling device based on deep reinforcement learning provided in the embodiment of the present invention.

[0109] Figure 2 1 is a schematic diagram of the structure of a container cloud task scheduling device based on deep reinforcement learning provided in Embodiment 1 of the present invention. The device 100 includes:

[0110] The task scheduling definition module 10 is used to define container cloud task scheduling matters, and the container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies.

[0111] The MDP modeling module 20 is used to model the container cloud task scheduling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set. The intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy. A reward function is defined to calculate the reward after the intelligent agent schedules the task to the available node.

[0112] The reinforcement learning optimization module 30 is used to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0113] Figure 3 1 is a schematic diagram of the structure of a container cloud task scheduling device based on deep reinforcement learning provided in Embodiment 2 of the present invention. The device 100 includes:

[0114] The task scheduling definition module 10 is used to define container cloud task scheduling matters, and the container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies.

[0115] Preferably, the task attributes include the task type, task requirements and / or priority of the task. The node attributes include the computing power, resource status, storage capacity and / or network bandwidth of each of the available nodes in the computing cluster. The task scheduling strategy includes determining how to allocate tasks and scheduling targets based on task requirements and node resources.

[0116] Preferably, the task scheduling definition module 10 is used to:

[0117] After receiving the task request, the container cloud creates a corresponding task for the task request.

[0118] The intelligent agent generates the corresponding task scheduling strategy according to the task requirements of the task and the current resource status of the available nodes.

[0119] Define the available node set Q in the computing cluster = {q1, q2, ..., q m}, wherein m=|Q| represents the number of available nodes, and each available node has different node attributes.

[0120] Define a set of incoming tasks Θ = {θ1,θ2,...,θ n}, where n=|Θ| represents the number of tasks, and each task has different task attributes.

[0121] Describe the task θ i The scheduling is ξ i ={θ i ,q j}, where j is the index of the selected available node. If the scheduling fails due to violation of resource constraints, the agent needs to perform task replacement to find another available node suitable for assigning the task.

[0122] The total response time of the task is defined as

[0123]

[0124] in, Represents the task queue time, Represents the task execution time.

[0125] The task scheduling strategy is defined, and the task scheduling goal is defined as minimizing the total response time of all tasks and reducing the occurrence of task replacement caused by scheduling failure, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

[0126] The MDP modeling module 20 is used to model the container cloud task scheduling items as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set. The intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy. Define a reward function for calculating the reward after the intelligent agent schedules the task to the available node.

[0127] Preferably, the Markov decision model is expressed as (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0,1] is a discount factor, indicating a preference for future rewards.

[0128] The MDP modeling module 20 is used to:

[0129] The agent observes the status from the cloud container environment, including the task requirements of each task and the current resource status of each available node.

[0130] Assuming that there are m available nodes in the current computing cluster Q, and each available node has o features, then at time step t, it can be expressed as:

[0131]

[0132] Where i is the index of an available node and z is the index of a node feature.

[0133] Assume that task θ in the current task set Θ j There are p features, then at time step t it can be expressed as:

[0134]

[0135] in, is a set containing p elements, each of which is k is the index of the current task feature.

[0136] The information of each task is sent to the agent as part of the state only after the task arrives, so the state set S is defined as:

[0137]

[0138] Among them, each element s in the state set S t ,Depend on Composition, including all time

[0139] The state of t .

[0140] Preferably, the MDP modeling module 20 is further used for:

[0141] Define the action as placing the task on the available nodes of the computing cluster. Action a t Indicates that at time step t, the task θ j Place it on available node q i action, determine the action set A as:

[0142] a t = {q i ,θ j}

[0143] The action set A is equivalent to the set of available nodes Q in the computing cluster, that is, A=Q.

[0144] The total response time of the task is defined as in Represents the task queue time, Represents the task execution time.

[0145] Design the task scheduling strategy and define the scheduling goal as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

[0146] Preferably, the MDP modeling module 20 is further used for:

[0147] First use To represent the resource utilization of available node i at time t.

[0148]

[0149] Wherein, K represents the number of resource types, and the resource types include processor (CPU), memory (Mem), receiving rate (Recv), transmission rate (Tran), reading rate (Read) and writing rate (Write). Indicates the sum of all K resource types.

[0150] Then the reward r at time t can be t The definition is as follows:

[0151]

[0152] in, is the task θ i is the total response time, α is the penalty factor, and κ is the number of replacements.

[0153] If the task is successfully scheduled, the inverse of its total response time is assigned as a reward to encourage the strategy to find a better schedule to obtain a shorter total response time and a higher reward r t .

[0154] If the task scheduling fails, a predetermined value Δ is used as a penalty, and the task scheduling strategy is recommended to avoid similar actions in the future. The penalty factor acts as an additional discount factor when the task requires multiple scheduling to succeed, and if the task fails multiple times, the predetermined value Δ of the penalty is amplified.

[0155] The reinforcement learning optimization module 30 is used to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

[0156] Preferably, the reinforcement learning optimization module 30 further comprises:

[0157] The model building submodule 31 is used to build the deep reinforcement learning model, which combines the deep Q network and the Rainbow method.

[0158] The function setting submodule 32 is used to set the minimization loss function of the deep reinforcement learning model, and use a deep neural network to approximate the action value function Q(s, a; θ), and its goal is expressed as the minimization loss function:

[0159] L(θ)=E[(r+γmaxQ(s′,a′;θ-)-Q(s,a;θ)) 2 ]

[0160] Among them, (r+γmaxQ(s′,a′;θ -) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, and θ - represents the parameters of the target network, which is updated regularly to stabilize the learning process.

[0161] The model training submodule 33 is used to train the deep reinforcement learning model.

[0162] The strategy optimization submodule 34 is used to input the task sequence to be processed into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

[0163] Preferably, the reinforcement learning optimization step comprises:

[0164] At discrete time step t, the deep reinforcement learning model observes the state s t , from the task scheduling strategy π(a t |s t ) Select an action a t Execute and transfer to the new state s t +1, and get reward r t , the goal of the deep reinforcement learning model is to maximize the expected return as:

[0165] V π (st)=E π [∑ t γ t r t ], t∈T

[0166] Where V π (s t ) is the agent from state s t The sum of discounted rewards obtained by starting to follow the task scheduling strategy π, E π is the expected operator under the task scheduling strategy π, is the sum over time steps t, γ t is the discount factor, r t is the immediate reward obtained at time step t, and t∈T means that time step t belongs to a time step set T.

[0167] The container cloud task scheduling device based on deep reinforcement learning provided by the embodiment of the present invention can achieve Figure 1 The various processes implemented in the embodiment of the container cloud task scheduling method based on deep reinforcement learning are not described here to avoid repetition.

[0168] The container cloud task scheduling device based on deep reinforcement learning provided by the embodiment of the present invention includes: a task scheduling definition module, which is used to define container cloud task scheduling matters; an MDP modeling module, which is used to model as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules the task to a certain available node according to the task requirements of the task, the current resource status of the available nodes and the task scheduling strategy; defining a reward function, which is used to calculate the reward after the intelligent agent schedules the task to the available node; a reinforcement learning optimization module, which is used to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy. The present invention also provides a container cloud task scheduling device, a storage medium and an electronic device based on deep reinforcement learning. By this, the present invention can automatically learn the expected scheduling strategy from the operation information of the system and automatically adapt to the changing environment without relying on expert knowledge.

[0169] The present invention also provides a storage medium for storing Figure 1 A computer program of any of the container cloud task scheduling methods based on deep reinforcement learning. For example, a computer program instruction, when it is executed by a computer, through the operation of the computer, can call or provide the method and / or technical solution according to the present invention, and can achieve the same technical effect. To avoid repetition, it will not be repeated here. The program instructions for calling the method of the present invention may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium and / or stored in a storage medium of an electronic device that operates according to the program instructions.

[0170] According to one embodiment of the present invention, the present invention also provides a Figure 4The electronic device 400 shown, the electronic device 400 optionally includes a storage medium 200 for storing a computer program and a processor 300 for executing a computer program, wherein when the computer program is executed by the processor 300, any of the above-mentioned container cloud task scheduling methods based on deep reinforcement learning is implemented, triggering the electronic device 400 to execute the methods and / or technical solutions based on the aforementioned multiple embodiments, and can achieve the same technical effect, to avoid repetition, it is not repeated here. It should be noted that the electronic devices in the embodiments of the present invention include mobile electronic devices and non-mobile electronic devices. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present invention.

[0171] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0172] The present invention can be implemented on a computer as a computer-implemented method, or implemented in dedicated hardware, or implemented in a combination of the two. The executable code or part thereof for the method according to the present invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes a non-temporary program code component stored on a computer-readable medium so as to perform the method according to the present invention when the program product is executed on a computer.

[0173] In an alternative embodiment, the computer program comprises computer program code means adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer readable medium.

[0174] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0175] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A container cloud task scheduling method based on deep reinforcement learning, characterized in that: include: A task scheduling definition step is to define container cloud task scheduling matters, wherein the container cloud task scheduling matters include task attributes, node attributes, and task scheduling strategies; The MDP modeling step is to model the container cloud task scheduling as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy; defining a reward function for calculating a reward after the intelligent agent schedules the task to the available node; The reinforcement learning optimization step optimizes the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

2. The container cloud task scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The task attributes include the task type, task requirements and / or priority of the task; the node attributes include the computing power, resource status, storage capacity and / or network bandwidth of each of the available nodes in the computing cluster; The task scheduling strategy includes determining how to allocate tasks and scheduling targets based on task requirements and node resources; The task scheduling definition step includes: After receiving the task request, the container cloud creates a corresponding task for the task request; The intelligent agent generates the corresponding task scheduling strategy according to the task requirements of the task and the current resource status of the available nodes; Define the available node set Q in the computing cluster = {q1, q2, ..., q m }, wherein m=|Q| represents the number of available nodes, and each available node has different node attributes; Define a set of incoming tasks Θ = {θ1,θ2,...,θ n }, wherein n=|Θ| represents the number of tasks, and each task has different task attributes; Describe the task θ i The scheduling is ξ i ={θ i ,q j }, where j is the index of the selected available node. If the scheduling fails due to violation of resource constraints, the agent needs to perform task replacement to find another available node suitable for assigning tasks; The total response time of the task is defined as in, Represents the task queue time, Represents the task execution time; The task scheduling strategy is defined, and the task scheduling goal is defined as minimizing the total response time of all tasks and reducing the occurrence of task replacement caused by scheduling failure, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

3. The container cloud task scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: The Markov decision model is expressed as (S, A, P, R, γ), where S represents the state set, A represents the action set, P represents the transition probability, R represents the reward function, and γ∈[0,1] is a discount factor, which represents the preference for future rewards; The MDP modeling step includes the following steps: taking the task requirements of each task and the current resource status of each available node as the state set: The agent observes the status from the cloud container environment, including the task requirements of each task and the current resource status of each available node; Assuming that there are m available nodes in the current computing cluster Q, and each available node has o features, then at time step t, it can be expressed as: Among them, i is the index of the available node, z is the index of the node feature; Assume that task θ in the current task set Θ j There are p features, then at time step t it can be expressed as: in, is a set containing p elements, each of which is k is the index of the current task feature; The information of each task is sent to the agent as part of the state only after the task arrives, so the state set S is defined as: Among them, each element s in the state set S t ,Depend on Composition, including all time The state of t .

4. The method for scheduling container cloud tasks based on deep reinforcement learning according to claim 3 is characterized in that: In the MDP modeling step, taking the set of available nodes as an action set includes: Define the action as placing the task on the available nodes of the computing cluster. Action a t Indicates that at time step t, the task θ j Place it on available node q i action, determine the action set A as: a t ={q i ,the j } The action set A is equal to the set of available nodes Q in the computing cluster, that is, A=Q; The total response time of the task is defined as in Represents the task queue time, Represents the task execution time; Design the task scheduling strategy and define the scheduling goal as minimizing the total response time of all tasks and reducing the frequency of task replacement due to violation of resource constraints, that is, Where T(Ξ) represents the minimization of the total response time of all tasks, represents the sum of the response times of all tasks, where is the response time of the ith task, and n is the total number of tasks.

5. The method for scheduling container cloud tasks based on deep reinforcement learning according to claim 4 is characterized in that: The step of defining a reward function in the MDP modeling step for calculating a reward after the agent schedules the task to the available node includes: First use To represent the resource utilization of available node i at time t; Wherein, K represents the number of resource types, and the resource types include processor (CPU), memory (Mem), receiving rate (Recv), transmission rate (Tran), reading rate (Read) and writing rate (Write); Indicates the sum of all K resource types; Then the reward r at time t can be t The definition is as follows: in, is the task θ i The total response time, α is the penalty factor, and κ is the number of replacements; If the task is successfully scheduled, the inverse of its total response time is assigned as a reward to encourage the strategy to find a better schedule to obtain a shorter total response time and a higher reward r t ; If the task scheduling fails, a predetermined value Δ is used as a penalty, and the task scheduling strategy is recommended to avoid similar actions in the future. The penalty factor acts as an additional discount factor when the task requires multiple scheduling to succeed, and if the task fails multiple times, the predetermined value Δ of the penalty is amplified.

6. The method for scheduling container cloud tasks based on deep reinforcement learning according to claim 1, characterized in that: The reinforcement learning optimization step includes: Establishing the deep reinforcement learning model, wherein the deep reinforcement learning model combines the deep Q network and the Rainbow method; The minimization loss function of the deep reinforcement learning model is set, and the action value function Q(s, a; θ) is approximated by a deep neural network, and its goal is expressed as the minimization loss function: L(θ)=E[(r+γmaxQ(s′,a;θ - )-Q(s,a;θ)) 2 ] Among them, (r+γmaxQ(s′,a′;θ - ) defines the target value of the state-action pair (s, a), r is the reward, γ is the discount factor, s' is the subsequent state, and θ - represents the parameters of the target network, which is updated regularly to stabilize the learning process; Training the deep reinforcement learning model; The task sequence to be processed is input into the trained deep reinforcement learning model to obtain the optimized task scheduling strategy.

7. The method for scheduling container cloud tasks based on deep reinforcement learning according to claim 6, characterized in that: The reinforcement learning optimization step includes: At discrete time step t, the deep reinforcement learning model observes the state s t , from the task scheduling strategy π(a t |s t ) Select an action a t Execute and transfer to the new state s t +1, and get reward r t , the goal of the deep reinforcement learning model is to maximize the expected return as: V π (s t )=E π [∑ t γ t r t ],t∈T Where V π (s t ) is the agent from state s t The sum of discounted rewards obtained by starting to follow the task scheduling strategy π, E π is the expected operator under the task scheduling strategy π, is the sum over time steps t, γ t is the discount factor, r t is the immediate reward obtained at time step t, and t∈T means that time step t belongs to a time step set T.

8. A container cloud task scheduling device based on deep reinforcement learning constructed based on the method described in any one of claims 1 to 7, characterized in that: include: A task scheduling definition module is used to define container cloud task scheduling matters, wherein the container cloud task scheduling matters include task attributes, node attributes and task scheduling strategies; The MDP modeling module is used to model the container cloud task scheduling items as a Markov decision model, including: taking the task requirements of each task and the current resource status of each available node as a state set, and taking the set of available nodes as an action set; the intelligent agent schedules at least one of the tasks to a certain available node according to the task requirements of at least one task, the current resource status of all available nodes and the task scheduling strategy; defining a reward function for calculating the reward after the intelligent agent schedules the task to the available node; The reinforcement learning optimization module is used to optimize the Markov decision model through a deep reinforcement learning model to obtain an optimized task scheduling strategy.

9. A storage medium, characterized in that: Used to store a computer program for executing the container cloud task scheduling method based on deep reinforcement learning as described in any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the container cloud task scheduling method based on deep reinforcement learning described in any one of claims 1 to 7 is implemented.

Images