A method and device for optimizing and scheduling flow system resources based on DQN algorithm

Abstract

The present invention relates to the field of distributed stream computing technology, and in particular to a stream system resource optimization scheduling method and device based on a DQN algorithm. The method includes: based on a stream processing model, modeling the application topology as a directed acyclic graph, based on the starting point of the directed acyclic graph, receiving the original data stream from the external environment, and passing the data stream to the downstream data processing component for processing and analysis; establishing a stream application model, including the task completion time and resource consumption of the system; establishing a resource model, and designing resource constraints based on the resource model, and then establishing an environmental model, forming the scheduling process into a model data change; optimizing resource allocation and task scheduling in the stream processing system based on an improved reinforcement learning model DQN; and updating the resource allocation strategy based on a gradient strategy algorithm. By adopting the present invention, the scheduling strategy and the reward function can be optimized, and the resource consumption of the system can be further reduced.

Description

Technical Field

[0001] The present invention relates to the field of distributed stream computing technology, and in particular to a stream system resource optimization scheduling method and device based on a DQN algorithm. Background Art

[0002] The current status of research on dynamic resource adjustment algorithms for streaming data task computing includes two aspects: responsive resource adjustment and proactive resource adjustment. Since the Storm system does not provide a mechanism for elastic resource scaling, it can only statically configure topology resources. Therefore, in response to this problem, many literatures have proposed cluster resource dynamic adjustment algorithms to support dynamic resource scaling of Storm running on clusters, thereby improving system elasticity and achieving excellent system performance.

[0003] At present, the dynamic adjustment of cluster resources on most platforms adopts a relatively simple responsive strategy, which adjusts resources according to the real-time status of the system to cope with sudden loads or resource bottlenecks. That is, when resources or certain indicators reach the threshold, the system will respond and adjust resources, such as adding or removing virtual machines to change node configurations. The Stela stream processing system, which has been integrated into Storm, mainly adds / reduces instances to nodes for operations with the maximum / minimum effective throughput percentage defined by itself, thereby optimizing throughput. It also supports the migration of operations to other nodes.

[0004] Active strategies allocate resources based on forecasts or historical data to prevent potential performance issues. The implementation of these strategies enables stream computing systems to respond to various complex scenarios more flexibly, improving the elasticity and performance of the system. Active algorithms have always been the focus and difficulty of research. Generally speaking, they include time series analysis and reinforcement learning. "Active" is reflected in making resource adjustment decisions in advance based on load forecasts. The prediction model is based on the analysis of recorded historical data and obtains the predicted value of the load in the future. In Orient Stream, it is based on incremental learning, and the prediction model is constructed to realize the dynamic adjustment of Storm resources. At the same time, various workloads are used as the initial training set to extract query features and use a set regression model to predict resource usage.

[0005] In the 21st century, artificial intelligence technology has become the mainstream of science and technology. In the application of artificial intelligence, many application problems require algorithms to make decisions and execute actions in a timely manner at every moment. For autonomous driving algorithms, it is necessary to determine the current driving strategy based on the current road conditions to ensure safe arrival at the destination; for Go algorithms, it is necessary to accurately calculate the position of the chess piece on the chessboard at each step to have the greatest probability of defeating the opponent. The common feature of these algorithms is that they make decisions and actions based on current conditions to achieve the desired goal. Machine algorithms that solve such problems are collectively called reinforcement learning (RL). The principle process of reinforcement learning algorithms can usually be described using Markov decision processes

[18] (Markov decision process, MDP). Markov decision processes are classic formulas for sequential decision making and are suitable for solving a series of problems in which optimal decisions are made to efficiently and reasonably utilize effective resources. In order to solve MDP problems, reinforcement learning provides a number of solutions, such as dynamic programming, Monte Carlo method, and time difference.

[0006] The typical representative of deep reinforcement learning based on value function is DQN, which was proposed by Deep Mind in 2013. This method uses convolutional neural network to fit the value function, generally the Q function. The input of the network is the data of the original scene, and the output is the maximum value of the Q function that can be obtained when performing various actions in this scene.

[0007] Van HV et al. first proposed the DQN algorithm, which is the first real work of deep reinforcement learning. In this article, the author tested the algorithm with Atari games. In the DQN algorithm, a deep convolutional neural network is used to fit the Q function. This network is called a Q network. In the DQN algorithm, there is no need to manually extract states and features. It integrates deep learning and reinforcement learning, and can solve decision-making problems in complex environments. After the DQN algorithm, a large number of improved algorithms have emerged. The specific aspects of the improvement include the overall structure of the system, the construction of training samples, and the structure of the neural network. Schaul T proposed the Double DQN (DDQN) algorithm. In the DDQN algorithm, two different sets of parameters and are set. It is used to select the action corresponding to the maximum Q value and to evaluate the Q value of the optimal action. These two sets of parameters separate action selection and strategy evaluation, reducing the risk of overestimating the Q value. DDQN uses the parameters of the current value network to select the most effective action and uses the target value as the parameters of the network to evaluate the optimal action. Experimental results also show that the DDQN algorithm can estimate the Q function value more accurately, making the training algorithm and the trained strategy more stable; Liu J proposed a DQN based on priority sampling, which is an improvement on the experience replay mechanism. The method in this paper calculates the priority for each sample in the experience pool, increasing the probability of valuable training samples when sampling.

[0008] The above methods adjust the parallelism configuration or scheduling strategy of Storm, but most of them only optimize one aspect. However, some existing scheduling strategy studies use reinforcement learning for adaptive scheduling, but these methods require a lot of time resources and complex reward functions, and do not consider reducing cluster resource consumption and optimizing reward functions. Summary of the invention

[0009] In order to solve the technical problem that the existing technology does not consider reducing the consumption of cluster resources and optimizing the reward function, the embodiment of the present invention provides a method and device for optimizing the scheduling of flow system resources based on the DQN algorithm. The technical solution is as follows:

[0010] On the one hand, a method for optimizing and scheduling flow system resources based on a DQN algorithm is provided, the method being implemented by a flow system resource optimization and scheduling device, the method comprising:

[0011] S1. Based on the stream processing model, the application topology is modeled as a directed acyclic graph. Based on the starting point of the directed acyclic graph, the original data stream is received from the external environment and passed to the downstream data processing components for processing and analysis.

[0012] S2. Establish a flow application model, including the system's task completion time and resource consumption; establish a resource model, and design resource constraints based on the resource model, and then establish an environment model to form a change in the scheduling process as a model data;

[0013] S3. Optimizing resource allocation and task scheduling in stream processing systems based on the improved reinforcement learning model DQN;

[0014] S4. Update the resource allocation strategy based on the gradient strategy algorithm.

[0015] On the other hand, a stream system resource optimization scheduling device based on a DQN algorithm is provided, and the device is applied to a stream system resource optimization scheduling method based on a DQN algorithm, and the device includes:

[0016] A directed acyclic graph modeling unit is used to model the application topology as a directed acyclic graph based on a stream processing model, receive the original data stream from the external environment based on the starting point of the directed acyclic graph, and pass the data stream to a downstream data processing component for processing and analysis;

[0017] The flow model building unit is used to build a flow application model, including the system's task completion time and resource consumption; build a resource model, and design resource constraints based on the resource model, and then build an environment model to form a change in the scheduling process as a model data;

[0018] The optimization unit is used to optimize resource allocation and task scheduling in the stream processing system based on the improved reinforcement learning model DQN;

[0019] An updating unit is used to update the resource allocation strategy based on the gradient strategy algorithm.

[0020] On the other hand, a flow system resource optimization scheduling device is provided, and the flow system resource optimization scheduling device includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the flow system resource optimization scheduling methods based on the DQN algorithm is implemented.

[0021] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for optimizing and scheduling streaming system resources based on the DQN algorithm.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] Based on the relationship between task completion time and total resource consumption, a reward function of reinforcement learning is designed based on this; a flow application model is established, including the task completion time and resource consumption of the system; a resource model is established, and based on this, resource constraints are designed and an environmental model is established, forming the scheduling process into a change of model data; for different cloud server clusters, a scheduling algorithm based on the DQN algorithm and policy gradient is designed, and the reward function is set as the weighted sum of resource consumption cost and average task completion time, and the reward for each step of the action is simplified. The embodiment of the present invention proposes a method for simultaneously optimizing the scheduling strategy and optimizing the reward function under the constraints of heterogeneous clusters to achieve resource optimization, further reducing the resource consumption of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 It is a flow chart of a method for optimizing and scheduling flow system resources based on a DQN algorithm provided by an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a DQN algorithm provided by an embodiment of the present invention;

[0027] Figure 3 It is a flow chart of a policy gradient algorithm provided by an embodiment of the present invention;

[0028] Figure 4 It is a block diagram of a flow system resource optimization scheduling device based on a DQN algorithm provided by an embodiment of the present invention;

[0029] Figure 5 It is a structural diagram of a flow system resource optimization scheduling device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0031] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0032] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0033] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0034] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0035] The embodiment of the present invention provides a method for optimizing and scheduling streaming system resources based on the DQN algorithm. The method can be implemented by a streaming system resource optimization and scheduling device, which can be a terminal or a server. Figure 1 The flowchart of the method for optimizing and scheduling flow system resources based on the DQN algorithm is shown. The processing flow of the method may include the following steps:

[0036] S1. Based on the stream processing model, the application topology is modeled as a directed acyclic graph. Based on the starting point of the directed acyclic graph, the original data stream is received from the external environment and the data stream is passed to the downstream data processing components for processing and analysis.

[0037] In a feasible implementation, in a stream processing application, the application topology can usually be modeled as a directed acyclic graph (DAG). In this DAG, each node represents a data source Spout or a data processing component Bolt, and the edge represents the flow of data streams between components.

[0038] The data source node is the starting point of the DAG and is responsible for receiving raw data streams from the external environment. These raw data streams can be real-time data from various sources such as sensors, log files, user interactions, etc. The data source node introduces the data stream into the stream processing application and passes it to the downstream data processing components for further processing and analysis.

[0039] S2. Establish a flow application model, including the system's task completion time and resource consumption; establish a resource model, and design resource constraints based on the resource model, and then establish an environment model to form a change in model data during the scheduling process.

[0040] Optionally, S2 establishes a flow application model, including the task completion time and resource consumption of the system; establishes a resource model, and designs resource constraints based on the resource model, and then establishes an environment model, including:

[0041] Resource requirements and capacity limits for CPU and memory are defined as follows:

[0042]

[0043]

[0044] Among them, x ki is a binary decision variable that is set to 1 if the kth executor is located in the i-th VM and 0 otherwise;

[0045] When allocating executors to jobs, only the resources of 1 VM are used, and the scheduler will not assign mixed resources from multiple VMs to a single executor; this constraint is defined as follows:

[0046]

[0047] After the scheduling process is complete, the cost incurred by the scheduler to run the job is defined as follows:

[0048]

[0049] Additionally, define the average job completion time for all jobs as follows:

[0050]

[0051] In the problem analysis, the goal is to minimize the cost of using the cluster and the average job completion time of the jobs, so the optimization problem is to minimize the following:

[0052] β×Cos t+(1-β)×Avg T (3-6)

[0053] Where β∈[0,1], β is a system parameter that the user sets to specify the optimization priority of the scheduler.

[0054] In one possible implementation, assume that N is the total number of VMs used to deploy the Storm cluster. These VMs can be of any instance type / size (which means that the resource capacity may vary in terms of CPU and memory). M is the total number of job topologies that need to be scheduled during the entire scheduling process. When a job is submitted to the cluster, the scheduler must schedule the executors in one or more VMs and must follow the resource capacity constraints of the VMs and the resource demand constraints of the current job. The user submits the resource requirements of the executors in two dimensions, CPU and memory. Therefore, each executor of the job can be viewed as a multi-dimensional box that needs to be placed into a specific VM during the scheduling process. Therefore, the resource requirements and capacity constraints for CPU and memory can be defined as follows:

[0055]

[0056]

[0057] where x ki A binary decision variable that is set to 1 if the kth executor is located in the ith VM and 0 otherwise.

[0058] When allocating executors to jobs, only the resources of 1 VM should be used, and the scheduler should not allocate mixed resources from multiple VMs to one executor. This constraint can be defined as follows:

[0059]

[0060] After the scheduling process is over, the cost incurred by the scheduler to run the job can be defined as follows:

[0061]

[0062] Additionally, the average job completion time for all jobs can be defined as follows:

[0063]

[0064] Since we want to minimize the cost of using the cluster and the average job completion time in the problem analysis, the optimization problem is to minimize the following:

[0065] β×Cos t+(1-β)×Avg T (3-6)

[0066] Where β∈[0,1], β is a system parameter that the user can set to specify the optimization priority of the scheduler.

[0067] The above optimization problem is a mixed integer linear programming (MIP) and non-convex problem, often referred to as an NP-hard problem. In order to solve this problem in an optimal manner, the optimal scheduler needs to know the job completion time before making any scheduling decision. This makes scheduler design extremely difficult as it requires the collection of job profiles and job performance modeling, which depends on various system parameters. Moreover, it may not be feasible to solve the problem in an optimal manner if the number of jobs, number of executors, and total cluster size increase. Although heuristic-based algorithms are highly scalable in solving the problem, they cannot generalize to multiple objectives and cannot capture the inherent characteristics of the cluster and workload to improve the objectives.

[0068] Scheduling in a cluster,The goal of scheduling is to allocate resources in appropriate virtual machines to minimize the overall usage cost of the cluster or minimize the duration of the job.

[0069] S3. Optimize resource allocation and task scheduling in stream processing systems based on the improved reinforcement learning model DQN.

[0070] In a feasible implementation, in the DQN-based adaptive scheduling framework, the design of the state space and action space is crucial, which directly affects the agent's understanding of the environment and the decision-making process. This section will explore how to design the representation of the system state and how to define the actionable actions to achieve optimization and adaptation of task scheduling.

[0071] An action is the selection of a specific VM to create a topology task executor. When the agent performs an action, it is executed in the cluster by the cluster manager. Afterwards, the reward generator calculates the reward by evaluating the action according to a predefined objective function.

[0072] It is assumed that the time steps in the reinforcement learning model are discrete and event-driven. Therefore, the state space moves from one time step to the next only if the agent takes an action. The key components of the RL model are specified as follows:

[0073] Agent. The agent acts as a scheduler, responsible for scheduling tasks in the cluster. At each time step, it observes the system state and takes an action. Based on the action, it obtains a reward and the next observable state from the environment.

[0074] Episode. An episode is the time interval from when the agent sees the first job and cluster status to when all job scheduling is completed. In addition, if the agent chooses a specific operation that violates any resource / job constraints, the training of an episode can be terminated early.

[0075] The core idea of ​​the DQN algorithm is to combine deep neural networks with Q-learning to solve complex decision-making problems. It uses deep neural networks to approximate the value function, so that it can handle high-dimensional input state space and has stronger expression and generalization capabilities.

[0076] like Figure 2 As shown, the key components of DQN include the following components:

[0077] 1. Deep Neural Network: DQN uses a deep neural network to approximate the value function Q(s,a). The input of the network is the state s, and the output is the Q value corresponding to each action a.

[0078] 2. Experience replay: DQN introduces an experience replay mechanism, which stores historical samples in the experience pool and randomly extracts samples for network updates. This approach can reduce the correlation between samples and improve the efficiency and stability of training.

[0079] 3. Target network: To improve the stability of training, DQN uses two neural networks: one is the online network, which is used to select actions and update Q values; the other is the target network, which is used to calculate the target Q value. The weights of the target network are regularly copied from the online network to reduce fluctuations in training.

[0080] Q-learning can be solved as a dynamic programming (DP) problem, where the Q function can be represented as a two-dimensional matrix containing values ​​for each combination of s and a. However, the total number of state and action pairs in high-dimensional space is large, and a tabular Q-learning solution is infeasible. Therefore, neural networks are usually trained with parameters θ to approximate the Q value, that is, Q(s,a;θ)≈Q * (s,a). Among them, the loss function needs to be minimized.

[0081] L i (θ i )=E s,a,r,s′～ρ(·) [(y i -Q(s,a;θ i )) 2 ] (Formula 1)

[0082] in,

[0083] Here, ρ is the distribution over {s,a,r,s′} transitions sampled from the environment. i The target is called the temporal difference (TD), y i -Q is called TD error.

[0084] Target iis a constantly changing target. In supervised learning, there is a fixed target. Therefore, the neural network can be trained to move towards the target at each step by reducing the loss. However, in RL, as the agent gradually learns about the environment, the target y i It is always improving, and for the network, it is like a moving target, making it unstable. The target network has fixed network parameters because it is a sample from the previous iteration. Therefore, the network parameters in the target network are used to update the current network for stable training.

[0085]

[0086] Optionally, S3's improved reinforcement learning model DQN optimizes resource allocation and task scheduling in the stream processing system, including:

[0087] S301, define state space;

[0088] Optionally, the definition of the state space in S301 includes:

[0089] Use a one-dimensional vector to represent the current state of the environment, where the first part of the vector is the requirement of the virtual machine VM: The second part of the vector is the requirements of the current job: [jobID,OP cpu ,OP mem ,j E ];

[0090] in, represents the current CPU availability of all VMs in the cluster, while Indicates the current memory availability of all VMs in the cluster; jobID indicates the current job ID, OP cpu and OP mem represents the CPU and memory requirements of one executor for the current job, respectively; since all executors of a job have the same resource requirements, the only other information required is the total number of executors created for the job, represented by j E express.

[0091] In one possible implementation, in the scheduling problem under study, the state is an observation obtained by an agent after taking each action. The scheduling environment is a deployed cluster where the agent can observe the state of the cluster after taking each action. However, the agent receives only the initial state at the beginning of the scheduling process or round, without taking any actions. The cluster state contains the following parameters: CPU and memory resource availability for all VMs in the cluster, the unit price of using each VM in the cluster, and the specification of the current job that needs to be scheduled. An action is a decision or placement made by the agent to allocate resources to each executor of the job. The resource allocation for each executor is considered an action, and after each action, the environment returns the next state to the agent. The current state of the environment can be represented using a one-dimensional vector, where the first part of the vector is the requirement of the VM: And the second part of the vector is the requirements of the current job: [jobID,OP cpu ,OP mem ,j E ]

[0092] in, represents the current CPU availability of all VMs in the cluster, while Indicates the current memory availability of all VMs in the cluster. jobID indicates the current job ID, OP cpu and OP mem j represents the CPU and memory requirements of one executor for the current job, respectively. Since all executors of a job have the same resource requirements, the only additional information required is the total number of executors that must be created for the job, represented by j. E express.

[0093] Therefore, the state space only grows larger with the size of the cluster (total number of VMs) and does not depend on the total number of jobs. If all previous jobs have been scheduled, the specifications of each job will only be sent to the agent as part of the state after it arrives. After each consecutive operation, the cluster resource capacity will be updated due to the placement of executors. Therefore, the next state will reflect the updated cluster state after the most recent operation. The agent will continue to receive job specifications for the current job until all executors for the current job are placed, with the only change being j which will be reduced by 1 after each executor placement. E parameters. When it becomes 0, the job is successfully scheduled. Then, the next job demand comes as the next state along with the updated cluster parameters. The initial job demand comes from the user, while the VM specification is provided by the cluster manager. For an RL model, these two specifications can be combined to provide an observed state of the environment after each selected action.

[0094] S302, defining an action space;

[0095] Optionally, defining the action space in S302 includes:

[0096] The agent action operation is determined as selecting a VM. Assuming there are N VMs in the cluster, action 0 is defined to specify that the agent will wait and not create an executor, where actions 1 to N specify the index number of the VM selected to create an executor for the current job.

[0097] In one possible implementation, the agent action operation is to select a VM where an executor will be created for the current job. If the cluster does not have enough resources to place one or all of the executors for the current job, the agent can also do nothing and wait for previously scheduled jobs to complete. In addition, to optimize certain goals (e.g., cost, time), the agent may decide to stop scheduling work and wait for resources to arrive. Therefore, if there are N VMs in the cluster, there are N+1 possible discrete actions. Here, action 0 is defined to specify that the agent will wait and not create an executor, where actions 1 to N specify the index number of the VM selected to create an executor for the current job.

[0098] S303, defining an action reward function;

[0099] Optionally, defining the action reward function in S303 includes:

[0100] Define the agent's immediate reward and round reward;

[0101] The round reward is a fixed round reward R fixed ;

[0102] The instant reward is defined as:

[0103] R epi =R fixed ×(Cost epi +Avg Tepi )

[0104] Among them, Cost epi represents the round cost, Avg Tepi Indicates the average working duration of a round.

[0105] In one possible implementation, the agent is rewarded immediately whenever it takes an action. Each action has a positive or negative reward. Positive rewards incentivize the agent to take good actions and also optimize the overall reward for the entire episode. Negative rewards, on the other hand, usually train the agent to avoid bad behavior. In reinforcement learning, when you want to maximize the total reward at the end of each episode: the agent must take actions taking into account both the current and future discounted rewards. The overall goal is to maximize the cumulative reward over a period of time, so sometimes taking an action that immediately produces a negative reward may be a good step to get a larger positive reward in a future step. So an agent's immediate reward and episode reward are defined.

[0106] The design choice of a fixed immediate reward simplifies the RL model. In fact, many RL models for real-world problems are designed this way. For example, for a maze solver robot, a large episode reward is obtained at the end of a successful episode, while training risk-free moves during the process of traversing the maze is rewarded with a fixed smaller reward. Similarly, in the defined RL model, the environment assigns an immediate small positive reward to each position that successfully executes the current Storm job.

[0107] Additionally, if the agent chooses to wait without placing any executors, the environment also immediately assigns a small negative reward (action 0). These positive / negative immediate rewards are fixed and do not depend on the VM cost or job performance. Instead, these fixed rewards are set to help the agent learn how to satisfy the resource capacity constraints of the VMs and the resource demand constraints of the jobs. After the initial training set, the agent should be able to automatically learn the resource constraints because if the job or VM constraints cannot be met, the set is terminated with a fixed large negative reward as the context reward.

[0108] Now, let's discuss how to calculate the round reward, which will reward only the successful completion of a round. Assume that, in the worst case, all VMs are on during the entire round, and each job takes the maximum amount of time to complete due to bad scheduling decisions. Therefore, it gives the maximum cost that the scheduler can incur in a set:

[0109]

[0110] Therefore, if the round VM caused by the agent is found, the cost total (As shown in 3-4), the normalized round cost can be defined as:

[0111] Cost normalized =Cost total / Cost max (3-8)

[0112] According to the priority of the cost target, the parameter β can be used to find the round cost as:

[0113] Cost epi =β×(1-Cost normalized ) (3-9)

[0114] In an ideal situation, all job executors are placed according to the job type, and the minimum average job completion time for a round can be obtained as follows:

[0115]

[0116] Similarly, if the executors of all jobs are not placed according to job characteristics, the maximum average job completion time for a round can be obtained as:

[0117]

[0118] Therefore, if the agent's round job completion time is found (as shown in Formula 3-5), the normalized round average job completion time can be defined as:

[0119]

[0120] Depending on the priority of the average work duration target, the β parameter can be used to find the round average work duration as:

[0121] Avg Tepi =(1-β)×(1-Avg Tnormalized ) (3-13)

[0122] R fixed is a fixed round reward, which will be scaled up or down according to the agent's performance in each round to maximize the objective function. Therefore, the final scenario reward R epi It can be defined as:

[0123] R epi =R fixed ×(Cost epi +Avg Tepi ) (3-14)

[0124] Among them, β∈[0,1]. In addition, Cost epi ∈[0,1] and Avg Tepi ∈[0,1]. Therefore, Cost epi and Avg Tepi The sum of can be at most 1, which will result in a reward of exactly R fixedFor example, if β = 0 is selected, it means that the agent will only be trained to reduce the average working time. In the best case, if the agent can achieve Avg T Equal to Avg Tmin , Avg Tepi The value of Cost is 1. epi The value of is 0. Therefore, R epi will be equal to R fixed , which means that the agent has learned the optimal strategy in time. epi It is never negative and is only rewarded upon successfully completing an episode. As mentioned before, the environment automatically gives a large negative reward if the episode is terminated early, which could be due to violating any of the constraints.

[0125] Fixed round reward R fixed is scaled according to the optimization goal. In this way, the agent can continuously explore different executor placement orders to optimize the desired goal. However, this episode reward is only obtained when an episode is successfully completed. When an episode is unsuccessful (a resource constraint is violated, which can be either a resource availability constraint or a resource demand constraint), the episode ends immediately and gives the agent a negative reward.

[0126] S304, initializing an experience replay buffer, where the experience replay buffer is used to store experience replays, each experience replay being a four-tuple, a Q network, and a target Q network;

[0127] S305, reset the environment and obtain the first state quadruple, the state quadruple includes state, action, reward and next state;

[0128] S306. Generate an action using the ε-greedy strategy: randomly select an action with a probability of ε, and select the action that maximizes the Q value in other cases;

[0129] S307, execute actions and interact with the environment to obtain feedback rewards, next state, and whether to trigger termination conditions;

[0130] S308, storing the experience in an experience pool for subsequent training; the experience includes a state quadruple;

[0131] S309, randomly extract a batch of samples from the experience pool to perform network update: update the network parameters by minimizing the square error between the predicted Q value and the target Q value;

[0132] S310, repeat the above steps S306-S309 until a predetermined stop condition is reached.

[0133] S4. Update the resource allocation strategy based on the gradient strategy algorithm.

[0134] In one possible implementation, DQN optimizes state-action values, and by doing so, it indirectly optimizes the policy. However, policy gradient algorithms are a class of algorithms for solving reinforcement learning problems. Unlike value function methods, policy gradient methods do not rely on a value function, but directly optimize the policy. It does this by parameterizing the policy and using the gradient information to update these parameters to maximize the cumulative reward. The policy is usually parameterized as a function of θ, written as π θ Therefore, π(a t |s t ) is a given state s at time step t t Next select action a t The number of rewards the agent can obtain depends on the strategy, such as Figure 3 shown.

[0135] In the traditional policy gradient algorithm, a batch of samples are collected in each iteration and then updated as shown in Formula 2. The collected samples are applied to the policy:

[0136]

[0137] Among them, γ is the discount factor, and s t 、a t and r t They are used to represent the state, behavior and reward at time t. T is the length of any round. t is the discounted cumulative return, calculated as 3

[0138]

[0139] Among them, t′ starts from the current time t, which means that if the current action is taken, the agent will receive an immediate reward r t , which also affects how much reward the agent can accumulate before the end of the episode.

[0140] The expected return is expressed in Equation 3 using the maximum log-likelihood, which measures the likelihood of the observed data. In the context of RL, it means how likely the agent expects the current trajectory to be under the current policy. When the likelihood is multiplied by the reward, the likelihood of the policy increases if it produces a positive reward. On the other hand, if the policy gives less or negative rewards, the likelihood of the policy decreases. In summary, the model tries to retain policies that work well and tends to discard policies that do not work well. However, since the formula is expressed as an expectation, it cannot be used directly. Therefore, a sampling-based estimator is used, as shown in Equation 4.

[0141]

[0142] in is the policy gradient of target J, parameterized by θ. It is also assumed that N trajectories (τ1…,τ N ), where each trajectory τ i is a list of states, actions, and rewards: For a time step of t = 0 to t = T i In this work, the policy gradient algorithm is used, as shown in Algorithm 2.

[0143]

[0144] The algorithm learns by using a Monte Carlo method (by calculating the reward after executing a full episode). After the sample collection step (line 3), the algorithm updates the underlying network using the updated policy gradient with the learned parameter α (line 5). While sampling the trajectories, an ε-greedy policy is used.

[0145] Optionally, the resource allocation strategy of S4 is updated based on a gradient strategy algorithm, including:

[0146] S401, initializing strategy parameters θ; the strategy parameters are initialized by random or pre-training;

[0147] S402. For each time step, perform the following operations: sample an action from the current strategy and execute it, and observe the new state and reward;

[0148] S403, calculate the policy gradient and update the policy parameter θ; use the gradient ascent method to update the parameters of the policy network;

[0149] S404, repeat steps S402 and S403 until a preset stop condition is met.

[0150] The advantages of the policy gradient algorithm are that it can directly optimize the policy, is applicable to continuous action space problems, and can be extended to high-dimensional features and complex tasks. However, it also has some disadvantages, such as slow convergence speed and easy to fall into local optimal solutions. In addition, the policy gradient algorithm usually requires a large number of samples for effective learning.

[0151] In the embodiment of the present invention, a reward function of reinforcement learning is designed based on the fact that there is a trade-off between task completion time and total resource consumption; a flow application model is established, including the task completion time and resource consumption of the system; a resource model is established, and based on this, resource constraints are designed and an environmental model is established, forming the scheduling process into a change in model data; for different cloud server clusters, a scheduling algorithm based on the DQN algorithm and policy gradient is designed, and the reward function is set as the weighted sum of the resource consumption cost and the average task completion time, and the reward for each step of the action is simplified. The embodiment of the present invention proposes a method for simultaneously optimizing the scheduling strategy and optimizing the reward function under the constraints of heterogeneous clusters to achieve resource optimization, further reducing the resource consumption of the system.

[0152] Figure 4 1 is a block diagram of a stream system resource optimization scheduling device based on a DQN algorithm according to an exemplary embodiment. The device is used in a stream system resource optimization scheduling method based on a DQN algorithm. Figure 4 , the device comprises:

[0153] A directed acyclic graph modeling unit 410 is used to model the application topology as a directed acyclic graph based on a stream processing model, receive the original data stream from the external environment based on the starting point of the directed acyclic graph, and pass the data stream to a downstream data processing component for processing and analysis;

[0154] The flow model building unit 420 is used to build a flow application model, including the task completion time and resource consumption of the system; build a resource model, and design resource constraints based on the resource model, and then build an environment model to form a scheduling process into a model data change;

[0155] An optimization unit 430, configured to optimize resource allocation and task scheduling in a stream processing system based on an improved reinforcement learning model DQN;

[0156] The updating unit 440 is used to update the resource allocation strategy based on the gradient strategy algorithm.

[0157] In the embodiment of the present invention, a reward function of reinforcement learning is designed based on the fact that there is a trade-off between task completion time and total resource consumption; a flow application model is established, including the task completion time and resource consumption of the system; a resource model is established, and based on this, resource constraints are designed and an environmental model is established, forming the scheduling process into a change in model data; for different cloud server clusters, a scheduling algorithm based on the DQN algorithm and policy gradient is designed, and the reward function is set as the weighted sum of the resource consumption cost and the average task completion time, and the reward for each step of the action is simplified. The embodiment of the present invention proposes a method for simultaneously optimizing the scheduling strategy and optimizing the reward function under the constraints of heterogeneous clusters to achieve resource optimization, further reducing the resource consumption of the system.

[0158] Figure 5 is a structural diagram of a flow system resource optimization scheduling device provided by an embodiment of the present invention, such as Figure 5 As shown, the stream system resource optimization scheduling device may include the above Figure 4 The stream system resource optimization scheduling device based on the DQN algorithm is shown. Optionally, the stream system resource optimization scheduling device 510 may include a first processor 2001 .

[0159] Optionally, the streaming system resource optimization scheduling device 510 may further include a memory 2002 and a transceiver 2003 .

[0160] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0161] Combine the following Figure 5 The components of the flow system resource optimization and scheduling device 510 are specifically introduced:

[0162] The first processor 2001 is the control center of the stream system resource optimization scheduling device 510, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0163] Optionally, the first processor 2001 may execute various functions of the streaming system resource optimization scheduling device 510 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0164] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0165] In a specific implementation, as an embodiment, the stream system resource optimization scheduling device 510 may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0166] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0167] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0168] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0169] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0170] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0171] It should be noted that Figure 5 The structure of the flow system resource optimization scheduling device 510 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0172] In addition, the technical effects of the flow system resource optimization scheduling device 510 can refer to the technical effects of the flow system resource optimization scheduling method based on the DQN algorithm described in the above method embodiment, which will not be repeated here.

[0173] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0174] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0175] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0176] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0177] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0178] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0179] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing and scheduling flow system resources based on a DQN algorithm, characterized in that: The method comprises: S1. Based on the stream processing model, the application topology is modeled as a directed acyclic graph. Based on the starting point of the directed acyclic graph, the original data stream is received from the external environment and passed to the downstream data processing components for processing and analysis. S2. Establish a flow application model, including the system's task completion time and resource consumption; establish a resource model, and design resource constraints based on the resource model, and then establish an environment model to form a change in the scheduling process as a model data; S3. Optimizing resource allocation and task scheduling in stream processing systems based on the improved reinforcement learning model DQN; S4, updating resource allocation strategy based on gradient strategy algorithm; The S2 establishes a flow application model, including the system's task completion time and resource consumption; establishes a resource model, and designs resource constraints based on the resource model, and then establishes an environment model, including: Resource requirements and capacity limits for CPU and memory are defined as follows: Among them, x ki is a binary decision variable that is set to 1 if the kth executor is located in the i-th VM and 0 otherwise; When allocating executors to jobs, only the resources of 1 VM are used, and the scheduler will not assign mixed resources from multiple VMs to a single executor; this constraint is defined as follows: After the scheduling process is complete, the cost incurred by the scheduler to run the job is defined as follows: Additionally, define the average job completion time for all jobs as follows: In the problem analysis, the goal is to minimize the cost of using the cluster and the average job completion time of the jobs, so the optimization problem is to minimize the following: β×Cost total +(1-β)×Avg T (3-6) Where β∈[0,1], β is a system parameter, and the user specifies the optimization priority of the scheduler by setting this parameter; The S3 optimizes resource allocation and task scheduling in the stream processing system based on the improved reinforcement learning model DQN, including: S301, define state space; S302, defining an action space; S303, defining an action reward function; S304, initializing an experience replay buffer, where the experience replay buffer is used to store experience replays, each experience replay being a four-tuple, a Q network, and a target Q network; S305, reset the environment and obtain the first state quadruple, the state quadruple includes state, action, reward and next state; S306. Generate an action using the ε-greedy strategy: randomly select an action with a probability of ε, and select the action that maximizes the Q value in other cases; S307, execute actions and interact with the environment to obtain feedback rewards, next state, and whether to trigger termination conditions; S308, storing the experience in an experience pool for subsequent training; the experience includes a state quadruple; S309, randomly extract a batch of samples from the experience pool to perform network update: update the network parameters by minimizing the square error between the predicted Q value and the target Q value; S310, repeat the above steps S306-S309 until a predetermined stop condition is reached.

2. The method for optimizing and scheduling flow system resources based on the DQN algorithm according to claim 1, characterized in that: The definition state space of S301 includes: A one-dimensional vector is used to represent the current state of the environment, where the first part of the vector is the requirement of the virtual machine VM: The second part of the vector is the requirements of the current job: [jobID,OP cpu ,OP mem ,j E ]; in, represents the current CPU availability of all VMs in the cluster, while Indicates the current memory availability of all VMs in the cluster; jobID indicates the current job ID, OP cpu and OP mem represents the CPU and memory requirements of one executor for the current job, respectively; since all executors of a job have the same resource requirements, the only other information required is the total number of executors created for the job, represented by j E express.

3. The method for optimizing and scheduling flow system resources based on the DQN algorithm according to claim 1 is characterized in that: The step of defining the action space in S302 includes: The agent action operation is determined as selecting a VM. Assuming there are N VMs in the cluster, action 0 is defined to specify that the agent will wait and not create an executor, where actions 1 to N specify the index number of the VM selected to create an executor for the current job.

4. The method for optimizing and scheduling flow system resources based on the DQN algorithm according to claim 1, characterized in that: The step S303 of defining the action reward function includes: Define the agent's immediate reward and round reward; The round reward is a fixed round reward R fixed ; The instant reward is defined as: R epi =R fixed ×(Cost epi +Avg Tepi ) Among them, Cost epi represents the round cost, Avg Tepi Indicates the average working duration of a round.

5. The method for optimizing and scheduling flow system resources based on the DQN algorithm according to claim 1, characterized in that: The S4 updates the resource allocation strategy based on the gradient strategy algorithm, including: S401, initializing strategy parameters θ; the strategy parameters are initialized by random or pre-training; S402. For each time step, perform the following operations: sample an action from the current strategy and execute it, and observe the new state and reward; S403, calculate the policy gradient and update the policy parameter θ; use the gradient ascent method to update the parameters of the policy network; S404, repeat steps S402 and S403 until a preset stop condition is met.

6. A stream system resource optimization scheduling device based on a DQN algorithm, wherein the stream system resource optimization scheduling device based on a DQN algorithm is used to implement the stream system resource optimization scheduling method based on a DQN algorithm as claimed in any one of claims 1 to 5, characterized in that: The device comprises: A directed acyclic graph modeling unit is used to model the application topology as a directed acyclic graph based on a stream processing model, receive the original data stream from the external environment based on the starting point of the directed acyclic graph, and pass the data stream to a downstream data processing component for processing and analysis; The flow model building unit is used to build a flow application model, including the system's task completion time and resource consumption; build a resource model, and design resource constraints based on the resource model, and then build an environment model to form a change in the scheduling process as a model data; The optimization unit is used to optimize resource allocation and task scheduling in the stream processing system based on the improved reinforcement learning model DQN; An updating unit, used for updating a resource allocation strategy based on a gradient strategy algorithm; The flow model building unit 420 is used to: Resource requirements and capacity limits for CPU and memory are defined as follows: Among them, x ki is a binary decision variable that is set to 1 if the kth executor is located in the i-th VM and 0 otherwise; When allocating executors to jobs, only the resources of 1 VM are used, and the scheduler will not assign mixed resources from multiple VMs to a single executor; this constraint is defined as follows: After the scheduling process is complete, the cost incurred by the scheduler to run the job is defined as follows: Additionally, define the average job completion time for all jobs as follows: In the problem analysis, the goal is to minimize the cost of using the cluster and the average job completion time of the jobs, so the optimization problem is to minimize the following: β×Cost total +(1-β)×Avg T (3-6) Where β∈[0,1], β is a system parameter, and the user specifies the optimization priority of the scheduler by setting this parameter; The optimization unit 430 is used to: S301, define state space; S302, defining an action space; S303, defining an action reward function; S304, initializing an experience replay buffer, where the experience replay buffer is used to store experience replays, each experience replay being a four-tuple, a Q network, and a target Q network; S305, reset the environment and obtain the first state quadruple, the state quadruple includes state, action, reward and next state; S306. Generate an action using the ε-greedy strategy: randomly select an action with a probability of ε, and select the action that maximizes the Q value in other cases; S307, execute actions and interact with the environment to obtain feedback rewards, next state, and whether to trigger termination conditions; S308, storing the experience in an experience pool for subsequent training; the experience includes a state quadruple; S309, randomly extract a batch of samples from the experience pool to perform network update: update the network parameters by minimizing the square error between the predicted Q value and the target Q value; S310, repeat the above steps S306-S309 until a predetermined stop condition is reached.

7. A stream system resource optimization and scheduling device, characterized in that: The flow system resource optimization scheduling device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images