Storm task scheduling method based on multi-agent deep reinforcement learning model

Abstract

The invention belongs to the field of big data stream processing, and particularly relates to a Storm task scheduling method based on a multi-agent deep reinforcement learning model, which comprises the following steps of: simulating a Storm task scheduling environment, and reflecting dynamic changes of factors such as resource allocation, task scheduling and cluster load in an actual Storm cluster; according to the method, a near-end strategy optimization (PPO) algorithm based on an anti-fact strategy gradient (COMA) is adopted as a reinforcement learning model for training; a training stage centralization and execution stage decentralization method is adopted to ensure the real-time performance of Storm in actual operation. According to the method, maximization of the resource utilization rate and minimization of the resource use cost are taken as optimization objectives, and multi-agent deep reinforcement learning and task scheduling characteristics are combined, so that the Storm can obtain a more intelligent and flexible scheduling strategy when facing dynamic resources and workloads.

Description

Technical Field

[0001] The present invention belongs to the field of big data stream processing, and particularly relates to a Storm task scheduling method based on a multi-agent deep reinforcement learning model. Background Art

[0002] Cloud computing, as a technology and service model for providing computing resources through the Internet, uses a pay-per-use business model and a service-oriented paradigm that allows users to freely customize resources and services according to their own needs. Users can access resources such as computing, storage, and networking on demand, without the need to invest in and maintain physical infrastructure. Cloud computing has brought a paradigm shift to the field of Internet-based computing. It uses the shareable resources of multiple physical machines and collectively provides services for various devices on demand. This newly developed computing paradigm mainly provides three types of services, namely Software as a Service (SaaS) - this model includes using the consumer of the device as a web service for processing, without the need to participate in maintenance and product upgrades; Platform as a Service (PaaS) - all the relevant tools required to build this service provide applications, without the need to download or install software. Through this PaaS resource, applications can be designed, produced, analyzed, accessed, and presented; Infrastructure as a Service (IaaS) - the tools required to execute SaaS, as well as the services that provide PaaS programs. Cloud computing mainly consists of four deployment models, namely public cloud, private cloud, community cloud, and hybrid cloud, which provide efficient and convenient computing resource services for enterprises and individuals through different deployment methods, and have become an indispensable infrastructure in Internet and data-intensive industries, providing extensive support for various application scenarios.

[0003] With the rapid development of Internet and big data technologies, the amount of data has shown an explosive growth. This growth has brought the need for high timeliness and operability of data, which has become a hot issue in current research, and big data processing has become very important. Big data processing requires powerful computing resources, and its potential is realized through the trust of data analysis. For this reason, multiple stream processing frameworks have emerged to achieve low-latency, high-throughput, and highly scalable big data processing. Big data processing frameworks such as Hadoop, Spark, Flink, and Storm are the most common choices and are widely used in various application scenarios such as batch processing, stream processing, and hybrid processing. And by deploying to the cloud platform, the elasticity, scalability, and cost advantages of cloud computing can be fully utilized to provide more flexible and efficient support for large-scale data processing. Therefore, cloud computing, as a low-cost software storage and distribution method, has also received increasing attention. In this environment, data, software, and applications no longer exist on the client; instead, they are regarded as abstract services and exist in the cloud.

[0004] The continuous supply of big data is called a data stream, which can come from a single or multiple big data sources, and various applications continuously generate a large amount of data. Since the collected data is heterogeneous and may have any form or nature, analyzing these data streams is a huge challenge. Most of the large-scale data comes from real-time data streams, so real-time processing and analysis are required because the value of data lies in its timeliness. Generally speaking, real-time stream processing can be applied to two types of application fields: the first type is the situation where decisions need to be made by sorting out data, and the second type is the situation where a certain reaction needs to be generated in real time, especially in the case of low latency. Stream processing needs to perform multiple operations, including data cleaning, query processing, stream-stream connection, stream-disk connection, data transformation, etc. Currently, a variety of methods, tools and technologies have been developed to address the challenges of stream processing. These methods may involve various data forms and storage models and apply various operations to data streams. A stream system must process high-speed data streams under strict latency constraints. To process a large number of data streams, modern stream processing frameworks distribute the processing across multiple computing nodes in a cluster. These systems sacrifice single-node performance to achieve the scalability of a large cluster and rely on the Java virtual machine to achieve platform independence. Due to frequent cross-node communication, the Internet has an important impact on the performance of modern distributed data processing systems. The goal of parallel processing technology is to achieve fast data processing. Parallel processing can be achieved by dividing an application into small parts called tasks and sending these tasks to different computing nodes for execution. According to the dependencies between tasks, they can be executed in parallel or serially. The relationships between application tasks are usually represented by a directed acyclic graph.

[0005] However, with the increase in business complexity, there are dual pressures of performance optimization and cost control when using cloud resources to process big data. Since multiple computing nodes cooperate to process tasks, the differences in capabilities between nodes result in different performances in task execution and data transmission. Task scheduling needs to achieve the best effect between balancing the computing load and reducing latency, and may involve dynamically adjusting resource allocation to adapt to fluctuations in data traffic, and considering the impact of cross-node communication on performance, so as to ensure the efficiency and cost-effectiveness of the system. Therefore, it is of great significance to explore how to effectively schedule tasks. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a Storm task scheduling method based on a multi-agent deep reinforcement learning model, including the following steps:

[0007] S1: Based on the Storm stream processing framework, establish an environmental model for the task scheduling problem, map the Storm topology structure into a graph model, where task nodes represent computing tasks, and edges represent the data dependencies between tasks. At the same time, define the characteristics of resource nodes, including the usage of CPU and memory;

[0008] S2: Set multiple agents. Each agent independently decides the task set it schedules and learns its own reward function. For each agent, other agents are part of the environment;

[0009] S3: Construct a global reward function for multi-objective optimization based on performance and cost, adjust the balance between performance metrics and cost metrics to achieve the global optimal effect, design immediate rewards, and give direct feedback to the agents under different action results to guide the agents to place resources as completely as possible;

[0010] S4: Combine the MAPPO algorithm and conduct joint training under the guidance of a centralized Critic network. Each agent has an independent Actor network to achieve decentralized decision-making. During the training process, each agent makes actions based on the current state and updates its policy according to the environmental feedback. In actual task scheduling, the trained model is used for real-time decision-making;

[0011] S5: Embed the trained model into the custom scheduler of Storm, conduct experiments in the real Storm framework, and evaluate the performance of the method.

[0012] Furthermore, the improved Storm scheduling system based on deep reinforcement learning should include a monitoring module, a training module, and a scheduling module. The monitoring module includes cost, task, and resource monitoring, regularly collects resource utilization rate, inter-task dependencies, resource requirements, and completion time from the system, and converts the collected status information into a format suitable for input to the reinforcement learning model based on the topology structure as the input of information. The training module includes cluster environment simulation, multi-agent scheduling agent, and reward function design. The cluster environment simulation describes environmental information such as cluster resource status and attributes of jobs to be scheduled. The scheduling agent is a decision-making model constructed based on the PPO algorithm. By interacting with the simulation environment and observing the immediate resource status and immediate placement decisions of nodes, its scheduling policy is corrected through the reward function mechanism. The scheduling module generates the scheduling policy for the corresponding task according to the status information of the input task and the trained decision-making model embedded in the cluster. The execution result after scheduling will also affect the cluster status, and the monitoring module should continuously collect new status data.

[0013] Furthermore, the calculation of the resource usage cost of the cluster is as follows:

[0014]

[0015] where Cost represents the cluster cost; the binary decision variable a i represents whether the i-th node is in an active state; represents the unit price of resources of the i-th node; t represents the task execution time.

[0016] Furthermore, the average resource utilization rate of the cluster is calculated as follows:

[0017]

[0018] where and represent the CPU and memory utilization rates of the i-th node respectively; β1 and β2 represent the impact factors of the resource utilization rate of the task on the CPU and memory utilization rates respectively; U avg represents the average resource utilization rate of the cluster.

[0019] Furthermore, the proportion of the execution time of the task is calculated as follows:

[0020]

[0021] where ex_time j represents the execution time of the j-th task; wa_time j represents the waiting time of the j-th task.

[0022] Furthermore, the design of the reward function based on the PPO model includes:

[0023]

[0024] a + b + c = 1

[0025] where a, b, and c represent the degrees of emphasis of the model on cost, resource utilization rate, and proportion of execution time respectively.

[0026] Furthermore, the objective function of the PPO model based on COMA includes:

[0027]

[0028] A t = Q t (s, a t , a -t ) - Baseline t (s, a -t )

[0029]

[0030] where L t CLIP represents the objective function of the t-th agent; π θ (a t | s t ) and represent the old and new policies respectively; a t and st respectively represent the action and state at the t-th time step; ε is a hyperparameter, and by adjusting its value, the update size of the policy is controlled, combined with a clipping function to limit the policy update amplitude; A t is the counterfactual advantage function based on COMA, which can measure the quality of the current action relative to the overall policy; Q π (s, a t , a -t ) represents the joint action-state value estimated by the centralized Critic network; a -t represents the set of actions of other agents; Baseline t (s, a -t ) represents the counterfactual baseline. By fixing the actions a of other agents -t , only the expectation calculation is performed on the action a′ of agent t t ;

[0031] Furthermore, the global objective function includes:

[0032]

[0033] Furthermore, the parameter updates of the Critic and Actor networks are calculated as follows:

[0034]

[0035] where represents the state-action value function estimated by the Critic network, indicating the expected cumulative return after performing action a in state s; represents the state value function estimated by the Critic network, indicating the expected cumulative return in the next state s ′ ; r is the immediate reward; γ is the discount factor, balancing the importance of current and future rewards; α represents the learning rate.

[0036] Advantages and beneficial effects of the present invention:

[0037] 1) The present invention considers the proportion of task execution time in the total completion time as an optimization goal, enabling the scheduling policy to take both short tasks and long tasks into account. In addition, reducing resource usage costs and improving resource utilization are also considered as optimization goals, and the reward function is designed accordingly. When scheduling tasks, the model will allocate tasks in the direction of the maximum reward, achieving multi-objective optimization.

[0038] 2) The present invention adopts a deep reinforcement learning model, aiming to enable an agent to learn how to obtain the maximum long-term reward in a certain task through interaction with the environment. This model combines the advantages of deep learning and reinforcement learning, can handle complex and high-dimensional state and action spaces, and also has the ability to adapt to dynamic environments. Through continuous learning and policy adjustment, it can achieve sensitivity and response ability to changing environments.

[0039] 3) The present invention extends PPO to a multi-agent framework, making the learning tasks of each agent more controllable, effectively decomposing complex problems, and adopting a method of using centralization in the training stage and decentralization in the execution stage to specifically execute to maintain the real-time performance of Storm. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a system structure diagram for big data streaming processing;

[0041] Figure 2 It is an algorithm flowchart of the present invention based on a multi-agent deep reinforcement learning model. DETAILED DESCRIPTION OF THE INVENTION

[0042] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] A Storm task scheduling method based on a multi-agent deep reinforcement learning model includes:

[0044] S1: Based on the Storm stream processing framework, establish an environmental model for the task scheduling problem, map the Storm topology structure into a graph model, where task nodes represent computing tasks and edges represent data dependency relationships between tasks. At the same time, define the characteristics of resource nodes, including CPU and memory usage;

[0045] S2: Set multiple agents, and each agent independently decides the task set it schedules and learns its own reward function. For each agent, other agents are part of the environment;

[0046] S3: Construct a global reward function for multi-objective optimization based on performance and cost, adjust the balance between performance metrics and cost metrics to achieve a globally optimal effect, design immediate rewards, and give direct feedback to the agents under different action results to guide the agents to place resources as fully as possible;

[0047] S4: Combine with the MAPPO algorithm and conduct joint training under the guidance of the centralized Critic network. Each agent has an independent Actor network to achieve decentralized decision-making. During the training process, each agent makes actions based on the current state and updates the policy according to the environmental feedback. In actual task scheduling, the trained model is used for real-time decision-making;

[0048] S5: Embed the trained model into the custom scheduler of Storm, conduct experiments in the real Storm framework, and evaluate the performance of the method.

[0049] As Figure 1 shown, the system includes a monitoring module, a training module, and a scheduling module. The monitoring module includes cost, task, and resource monitoring, regularly collects resource utilization, inter-task dependencies, resource requirements, and completion times from the system, and converts the collected status information into a format suitable for input to the DRL model based on the topology structure as the input of information. The training module includes cluster environment simulation, multi-agent scheduling agent, and reward function design. The cluster environment simulation describes environmental information such as the cluster resource status and the attributes of the jobs to be scheduled. The scheduling agent is a decision-making model constructed based on the PPO algorithm. By interacting with the simulation environment and observing the immediate resource status and immediate placement decisions of the nodes, it corrects its scheduling strategy through the reward function mechanism. The scheduling module generates a scheduling strategy for the corresponding task according to the status information of the input task and the trained decision-making model embedded in the cluster.

[0050] The process of task scheduling based on the Storm stream processing framework of the multi-agent deep reinforcement learning model in this embodiment, as Figure 2 shown, includes the following steps:

[0051] Obtain the feature information of the tasks from the real data data stream platform to provide data support for the subsequent scheduling process of training the agents. Initialize the Storm cluster environment, including deploying tasks and node resources, setting task dependencies, preparing the running environment, and initializing the policy network and value function network.

[0052] The agent observes the current state, including the task queue, node status, resource usage, etc. The policy network outputs an action (such as allocating a certain task to a certain node) according to the input state features. After executing the action, the result is applied to the environment. The environment is affected by the action and receives the feedback from the environment (the next state and reward). After executing the action, the environment is updated and returns the updated task state and node state. The positive or negative reward obtained through the reward function calculation accumulates enough sampling data for model optimization. Store the multi-time-step interaction data in the experience replay buffer. Each time for optimization, randomly sample a part of the data from the buffer to avoid the impact of time correlation on model training.

[0053] Optimize the policy network and value function network through the reinforcement learning algorithm to improve the decision-making performance. Use the gradient descent optimization algorithm to adjust the model parameters until the specified number of training epochs is reached, and save the trained network model. Apply the trained decision-making model to the actual Storm scheduling. The scheduler monitors the running status of tasks and nodes in real time, dynamically adjusts the scheduling policy, and generates the allocation decision of tasks in real time according to the decision-making model.

[0054] The calculation of the reward function includes the following steps:

[0055] The calculation of the resource usage cost of the cluster is as follows:

[0056]

[0057] where Cost represents the cluster cost; the binary decision variable a i represents whether the i-th node is in an active state; represents the unit price of resources of the i-th node; t represents the task execution time.

[0058] The calculation of the average resource utilization rate of the cluster is as follows:

[0059]

[0060] where, and represent the CPU and memory utilization rates of the i-th node respectively; β1 and β2 represent the influence factors of the resource utilization rate of the task on the CPU and memory utilization rates respectively; U avg represents the average resource utilization rate of the cluster.

[0061] The calculation of the execution time ratio of the task is as follows:

[0062]

[0063] where ex_time j represents the execution time of the j-th task; wa_time j represents the waiting time of the j-th task.

[0064] The design of the reward function based on the PPO model includes:

[0065]

[0066] a + b + c = 1

[0067] where a, b, and c represent the degrees of importance of the model for cost, resource utilization rate, and execution time ratio respectively.

[0068] The objective function of the PPO model based on COMA includes:

[0069]

[0070] A t = Q t (s, a t , a -t ) - Baseline t (s, a -t )

[0071]

[0072] where L t CLIP represents the objective function of the t-th agent; π θ (a t | s t ) and respectively represent the old and new policies; a t and s t respectively represent the action and state at the t-th time step; ε is a hyperparameter, and by adjusting its value, the size of the policy update is controlled, combined with the clipping function to limit the policy update amplitude; A t is the counterfactual advantage function based on COMA, which can measure the quality of the current action relative to the overall policy; Q π (s, a t , a -t ) represents the joint action state value estimated by the centralized Critic network; a -t represents the set of actions of other agents; Baseline t (s, a -t ) represents the counterfactual baseline, by fixing the actions a -t of other agents, and only calculating the expectation for the action a' t of agent t;

[0073] The global objective function includes:

[0074]

[0075] The update calculations of the Critic and Actor network parameters are as follows:

[0076]

[0077] where represents the state-action value function estimated by the Critic network, indicating the expected cumulative return after executing action a in state s; represents the state value function estimated by the Critic network, indicating the next state s′ The expected cumulative return under; r is the immediate reward; γ is the discount factor, balancing the importance of current and future rewards; α represents the learning rate.

[0078] Through the reinforcement learning model, the present invention generates an efficient task allocation strategy in real time according to task characteristics, node characteristics, and the dynamic operating environment, sets multiple optimization objectives as the reward function, and extends it to a multi-agent framework. Through multiple independent but cooperative agents, the scheduling efficiency and system adaptability are further improved. This method can adjust the weights of the reward function according to actual needs to achieve the priority optimization of specific objectives, or balance the weights to achieve multi-objective optimization. Reinforcement learning can better adapt to dynamic environments and complex load patterns than heuristic algorithms, achieving more efficient resource utilization and scheduling optimization.

[0079] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Storm task scheduling method based on a multi-agent deep reinforcement learning model, characterized in that The method includes: S1: Based on the Storm stream processing framework, establish an environmental model for the task scheduling problem, map the Storm topology structure into a graph model, where task nodes represent computing tasks and edges represent data dependencies between tasks. At the same time, define the characteristics of resource nodes, including the usage of CPU and memory; S2: Set multiple agents, each agent independently decides the set of tasks it schedules and learns its own reward function. For each agent, other agents are part of the environment; S3: Construct a global reward function for multi-objective optimization based on performance and cost, adjust the balance between performance metrics and cost metrics to achieve the global optimal effect, design immediate rewards, and give direct feedback to the agent under different action results to guide the agent to place resources as completely as possible; S4: Combine the MAPPO algorithm and conduct joint training under the guidance of a centralized Critic network. Each agent has an independent Actor network to achieve decentralized decision-making. During the training process, each agent makes actions based on the current state and updates the strategy according to the environmental feedback. In actual task scheduling, use the trained model for real-time decision-making; S5: Embed the trained model into the custom scheduler of Storm, conduct experiments in the real Storm framework, and evaluate the performance of the method.

2. The Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that The improved Storm scheduling system based on deep reinforcement learning should include a monitoring module, a training module, and a scheduling module. The monitoring module includes cost, task, and resource monitoring, regularly collects resource utilization rate, inter-task dependencies, resource requirements, and completion time from the system, and converts the collected state information into a format suitable for input to the reinforcement learning model according to the topology structure as the input of information. The training module includes cluster environment simulation, multi-agent scheduling agent, and reward function design. The cluster environment simulation describes environmental information such as cluster resource status and attributes of jobs to be scheduled. The scheduling agent is a decision-making model constructed based on the PPO algorithm. By interacting with the simulation environment and observing the immediate resource status and immediate placement decisions of nodes, it corrects its scheduling strategy through the reward function mechanism. The scheduling module generates a scheduling strategy for the corresponding task according to the state information of the input task, and the execution result after scheduling will also affect the cluster state. The monitoring module should continuously collect new state data.

3. The Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that, The calculation of the resource usage cost of the cluster is as follows Among them, Cost represents the cluster cost; the binary decision variable a i indicates whether the i-th node is in an active state; represents the unit price of resources of the i-th node; t represents the task execution time.

4. A Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that The calculation of the average resource utilization rate of the cluster is as follows: Among them, and respectively represent the CPU and memory utilization rates of the i-th node; β1 and β2 respectively represent the impact factors of the resource utilization rate of the task on the CPU and memory utilization rates; U avg represents the average resource utilization rate of the cluster.

5. A Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that, The calculation of the execution time ratio of the task is as follows: Among them, ex_time j represents the execution time of the j-th task; wa_time j represents the waiting time of the j-th task.

6. The Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, wherein The reward function design based on the PPO model includes: a + b + c = 1 where a, b, and c respectively represent the degree of importance of the model for cost, resource utilization rate, and execution time ratio.

7. A Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that The objective function of the PPO model based on COMA includes: A t = Q t (s, a t , a -t ) - Baseline t (s, a -t ) where L t CLIP represents the objective function of the \(t\)-th agent; \(\pi\ θ (a t |s t ) and represent the old and new policies respectively; \(a\ t and \(s\ t represent the action and state at the \(t\)-th time step respectively; \(\varepsilon\) is a hyperparameter, and by adjusting its value, the update size of the policy is controlled. Combining with the clipping function limits the policy update amplitude; \(A\ t is the counterfactual advantage function based on COMA, which can measure the quality of the current action relative to the overall policy; \(Q\ π (s,a t ,a -t ) represents the joint action-state value estimated by the centralized Critic network; \(a\ -t represents the set of actions of other agents; Baseline t (s,a -t ) represents the counterfactual baseline. By fixing the actions \(a\ -t of other agents, the expectation calculation is only performed on the action \(a'\ t of agent \(t\); Furthermore, the global objective function includes:

8. The Storm task scheduling method based on a multi-agent deep reinforcement learning model according to claim 1, characterized in that The update of the Critic and Actor network parameters includes: Among them represents the state-action value function estimated by the Critic network, indicating the expected cumulative return after executing action a in state s; represents the state value function estimated by the Critic network, indicating the expected cumulative return in the next state s'; r is the immediate reward; γ is the discount factor, balancing the importance of current and future rewards; α represents the learning rate.

Images