Task scheduling method and system based on cross-region storm stream processing framework

By building the Storm stream processing framework in a cross-regional cloud environment, calculating the cost and latency of each node, and prioritizing the placement of tasks on high-priority nodes, the high cost problem of Storm cluster task scheduling is solved, achieving a balance between cost and latency.

CN118214747BActive Publication Date: 2026-04-14CHONGQING UNIV OF POSTS & TELECOMM
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
CHONGQING UNIV OF POSTS & TELECOMM
Filing Date
2024-02-26
Publication Date
2026-04-14

AI Technical Summary

Technical Problem

In cross-regional cloud environments, Storm cluster task scheduling suffers from high costs and fails to effectively address request latency constraints.

Method used

By building a cross-region Storm stream processing framework, the resource usage cost, energy consumption cost, and request latency of each node are calculated, and a priority list for task scheduling is constructed, prioritizing the placement of tasks on nodes with higher priority.

Benefits of technology

While ensuring request latency constraints, it significantly reduces task scheduling costs and energy consumption costs, thereby improving the cost-effectiveness of the cluster.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118214747B_ABST
    Figure CN118214747B_ABST
Patent Text Reader

Abstract

The application relates to a task scheduling method and system based on a cross-region Storm flow processing framework, and the task scheduling method comprises the following steps: constructing a cross-region Storm flow processing framework based on a geographic distribution cloud environment, and performing task scheduling between nodes in all clouds; calculating the total cost of task scheduling of each node according to the resource use cost and energy consumption cost of all nodes performing tasks on each cloud; calculating the request delay time of task scheduling to each cloud according to the average response time of all nodes performing tasks on each cloud; calculating the target value of all nodes performing task scheduling on each cloud according to the total cost and the request delay time; constructing a priority list according to the target value, and preferentially selecting nodes with higher priority in the priority list to place tasks. The application fully considers cost efficiency and request delay constraints, preferentially schedules tasks to nodes with higher priority to perform, and obviously reduces the task scheduling cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of big data streaming processing, and in particular to a task scheduling method and system based on the cross-regional Storm streaming processing framework. Background Technology

[0002] Big data processing has become critical due to the massive data analytics needs across all major business and scientific sectors, such as banking, fraud detection, healthcare, demand forecasting, and scientific exploration. In cloud computing environments, infrastructure providers (InPs, also known as cloud providers) build large data centers in geographically distributed locations to ensure reliability. Cloud services are provided by data center sites, each containing tens of thousands of servers distributed across different geographic locations. Service providers (SPs) leverage this geographical diversity of data centers to serve customers from multiple geographic regions. This geographical diversity of computing resources offers several benefits, such as high availability, effective disaster recovery, unified access for users in different regions, and the ability to select better virtual machines for placement. To process large amounts of geographically dispersed data, it is often necessary to transfer the data required by a task to the data center where the task resides for centralized processing. However, scheduling tasks deployed across clusters of geographically distributed clouds presents a technical challenge: ensuring latency while minimizing costs.

[0003] In the era of big data, stream computing is a real-time computing paradigm that can extract valuable information from input data streams in a timely manner. Various frameworks have been proposed to address the needs of stream processing, including Spark Streaming, Apache Flink, Gearpump, and Apache Storm. Among them, the Apache Storm framework, based on Java and Clojure, is free to use and supports distributed real-time computing. It can be used to reliably process unbounded data streams in a simple way. In Storm, a topology is a graph of the application (Apache Storm), similar to a job running on Hadoop (Apache Hadoop), providing a logical view of the data process. A Storm cluster has master nodes and slave nodes, with slave nodes being any number of worker nodes (Apache Storm). The master node sets up the master server, which runs a daemon called Nimbus, responsible for allocating resources, assigning tasks to worker nodes, and monitoring for failures. Most virtual machines in the Storm cluster are worker nodes. Each worker node runs a daemon called Supervisor, which listens for job assignments on its machine and starts / stops worker processes as needed. Each worker node can run any number of worker processes and participate in different topologies. All coordination between Nimbus and the Supervisor is accomplished through a ZooKeeper cluster (Apache Storm). ZooKeeper is a distributed, open-source distributed application coordination service. Furthermore, the Nimbus and Supervisor daemons are fast-failure and stateless; all their state is maintained in ZooKeeper or on local disk, and can be retrieved from ZooKeeper and restarted without data loss in the event of a failure.

[0004] Chinese patent application CN111522637A discloses a cost-effective Storm task scheduling method. This method, based on the Storm stream processing framework, constructs a cost-effective stream processing model for stream computation. While this invention can maximize the utilization of virtual machine resources and improve job performance, it can also reduce virtual machine costs. However, this method does not consider request latency constraints; therefore, in cross-regional cloud environments, request latency is a common problem, necessitating a more efficient scheduling method for task scheduling.

[0005] In summary, using Storm clusters for task scheduling in cross-regional cloud environments still faces the problem of high costs while ensuring request latency constraints. Summary of the Invention

[0006] To address the high cost of task scheduling in Storm clusters in cross-regional cloud environments, the present invention aims to: perform task scheduling in geographically distributed (i.e., cross-regional) cloud environments, ensuring request latency constraints and significantly reducing task scheduling costs.

[0007] The first aspect of this invention proposes a task scheduling method based on the cross-region Storm streaming framework, comprising:

[0008] Step A: Build a cross-region Storm stream processing framework based on a geographically distributed cloud environment, and schedule tasks among nodes in the entire cloud.

[0009] Step B: Calculate the total cost of scheduling tasks for each node based on the resource usage cost and energy consumption cost of all nodes executing tasks on each cloud.

[0010] Step C: Calculate the request latency for task scheduling to each cloud based on the average response time of all nodes executing tasks on each cloud.

[0011] Step D: Calculate the target value for all nodes executing task scheduling on each cloud based on the total cost and the request latency;

[0012] Step E: Construct a priority list for task scheduling based on the target value, and traverse the priority list to select nodes with higher priorities to place tasks.

[0013] A second aspect of this invention proposes a task scheduling system based on the cross-regional Storm streaming framework, comprising:

[0014] Master node, slave nodes, resource monitoring module, request latency monitoring module, cost-benefit information database, UI performance monitoring module, data processing module, Storm custom scheduler, specifically:

[0015] The master node is configured as a master server, which runs a daemon process called Nimbus. This daemon process is responsible for allocating resources, assigning tasks to slave nodes, and monitoring for failures.

[0016] The slave node is a worker node that executes tasks and runs a daemon called Supervisor. This daemon is responsible for listening to the work assignments on its computer and starting or stopping worker processes as needed.

[0017] The resource monitoring module sets up resource monitoring scripts to monitor the CPU and memory resource usage of slave nodes and write the usage information into the cost-benefit information database.

[0018] The request latency monitoring module sets up a request latency monitoring script to monitor the request latency time of task scheduling to each cloud and writes the request latency time into a cross-regional cost-effectiveness database.

[0019] A cross-regional cost-benefit information database is used to store runtime data of the Storm cluster and the total cost C of each node executing tasks. total The target value P and the priority list;

[0020] The UI performance monitoring module is used to monitor task execution time and write the task execution time into a cross-regional cost-benefit information database.

[0021] The data processing module is responsible for processing the data collected by the resource monitoring script, request latency monitoring script, and UI monitoring module, and calculating the total cost C of the tasks executed by each node. total The target value P and the request delay time T total ;

[0022] A custom Storm scheduler can access the resource requirements and priority list of currently scheduled tasks from a cross-regional cost-effectiveness database, directly or indirectly controlling the master and slave nodes to schedule and place tasks.

[0023] The beneficial effects of this invention are as follows: This invention considers reducing cluster resource usage costs and energy consumption costs while mitigating request latency constraints. It takes into account communication latency caused by the distance between the user's task submission location and clouds in different locations, and tries to schedule tasks to clouds (i.e., data centers) closer to the user. It also considers the electricity prices and Power Usage Effectiveness (PUE) of different locations, which result in varying energy consumption costs. During task scheduling, the target value for each node in all clouds is calculated, and a priority list for task scheduling is generated based on the target value. Tasks are placed on higher-priority nodes first, thereby reducing the costs incurred by the cluster. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the cross-regional data center system in this invention;

[0025] Figure 2 This is a flowchart illustrating the steps of the task scheduling method based on the cross-regional Storm streaming framework in Embodiment 1 of the present invention.

[0026] Figure 3 This is a system architecture diagram of the task scheduling method system based on the cross-regional Storm stream processing framework in Embodiment 2 of the present invention;

[0027] Figure 4This is a flowchart of task scheduling in a geographically distributed cloud environment according to the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The default scheduling algorithm for the Storm streaming system places executors from each job into virtual machines (VMs) in a simple round-robin fashion. First, it iterates through the topology of executors and assigns them to workers. Then, based on slot availability, it distributes the workers evenly across worker nodes. This scheduling results in a worker process with an equal number of executors, distributing them among worker nodes so that each worker node executes an equal number of workers. It evenly distributes the workload across the cluster without considering resource usage costs, and this initial placement method can increase the cluster's energy consumption. Since the number of VMs is finite and they have different functionalities, an efficient scheduling method is needed to carefully arrange the placement of application tasks into VMs.

[0030] Reference Figure 1 As shown, in a cross-regional data center system, requests from different locations in clouds (data centers) distributed across different geographical areas have varying distances to each cloud, resulting in different request latencies (i.e., request delay times). This invention proposes a task scheduling method and system based on the cross-regional Storm stream processing framework by acquiring various cluster status information and considering resource costs, energy costs, and request latency constraints. This method prioritizes tasks for execution on higher-priority nodes, improving the cost-effectiveness of the cluster and ensuring request latency is maintained.

[0031] Embodiment 1 of this invention proposes a task scheduling method based on the cross-regional Storm streaming framework. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this embodiment: a server and a terminal. In other words, the task scheduling method based on the cross-regional Storm streaming framework can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, server clusters, cloud servers, or cloud server clusters. The server can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] Reference Figure 2 As shown, the task scheduling method based on the cross-region Storm stream processing framework includes:

[0033] Step A: Build a cross-region Storm stream processing framework based on a geographically distributed cloud environment, and schedule tasks among nodes in the entire cloud.

[0034] Step B: Calculate the total cost of scheduling tasks for each node based on the resource usage cost and energy consumption cost of all nodes executing tasks on each cloud.

[0035] Step C: Calculate the request latency for task scheduling to each cloud based on the average response time of all nodes executing tasks on each cloud.

[0036] Step D: Calculate the target value for all nodes executing task scheduling on each cloud based on the total cost and the request latency;

[0037] Step E: Construct a priority list for task scheduling based on the target value, and traverse the priority list to select nodes with higher priorities to place tasks.

[0038] Furthermore, the specific formula for calculating the target value is as follows:

[0039] P = w1·C total +w2·T total (1)

[0040] Where P represents the target value, C total T represents the total cost of each node executing the task. total w1 represents the request latency time of task scheduling to each cloud, w2 represents the impact factor of the total cost of task execution on each node, and w1 represents the impact factor of the request latency time of task scheduling to each cloud.

[0041] In a geographically distributed cloud environment (i.e., a cross-regional cloud environment), the request latency constraint of a certain cloud is the total request latency T for task scheduling to that cloud. total The constraints are met.

[0042] Furthermore, the request delay time T total The specific calculation formula is as follows:

[0043]

[0044]

[0045]

[0046]

[0047] in, Let be the average response time of all nodes executing tasks on the i-th cloud. This represents the transmission time of the i-th cloud. δ represents the response time of node j in the i-th cloud. i ={N1,N2,...,N k}, k is the total number of nodes in the i-th cloud, N k Let R represent the last node in the i-th cloud, and let R represent the linear relationship between data transmission time θ and geographical distance θ. dc dc represents the Euclidean distance between clouds, and dc represents the cloud to which the cloud is scheduled.

[0048] Furthermore, the total cost C for each node to execute the task total The specific calculation formula is as follows:

[0049] C total =C resource +C energy (6)

[0050] Among them, C resource C represents the resource usage cost of each node executing the task. energy This represents the energy cost of each node executing its task.

[0051] Furthermore, based on the resource unit price and task execution time of each node in each cloud, the resource usage cost C is calculated. resource The specific calculation formula is as follows:

[0052]

[0053]

[0054]

[0055] in, Let represent the resource usage cost of the i-th cloud, where i is a positive integer, i∈DC, and DC represents the set of all clouds, DC={dc1,dc2,...,dc n}, where n is the total number of clouds. N represents the unit price of the resource at the j-th node. j Let δ represent the j-th node in the i-th cloud, t represent the task execution time, and δ represent the t-th node. i ={N1,N2,...,N k}, k is the total number of nodes in the i-th cloud, N k Let x represent the last node in the i-th cloud, and let x be the binary decision variable. j This indicates whether a node is active or inactive. "Active" means that node j is active, and "otherwise" means that node j is inactive.

[0056] The resource unit price refers to the price of different types of virtual machine instances, which is determined with reference to similar types of virtual machines on Amazon AWS (Eastern Ohio, USA).

[0057] Furthermore, based on the energy prices of each cloud and the energy consumption of each node, the energy cost C of the task is calculated. energy The specific calculation formula is as follows:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] in, Let dc1 represent the energy cost of the i-th cloud, where i is a positive integer, i∈DC, and DC represents the set of all clouds, DC={dc1,dc2,...,dc2}. n}, where n is the total number of clouds, and PUE is... i PE represents the ratio of all energy consumed by the i-th cloud to the energy consumed by the IT load.i Let x represent the energy price of the i-th cloud, a binary decision variable. j Indicates whether node j is active, δ i ={N1,N2,...,N k}, k is the total number of nodes in the i-th cloud, N k N represents the last node in the i-th cloud. j This represents the j-th node in the i-th cloud. This represents the energy consumption of the j-th node. This represents the set of threads at node j. Let m represent the energy consumption of the m-th thread, where m ∈ {1, 2, ..., g}, and g represents the number of threads. This represents the m-th thread task. m Memory power consumption, This represents the m-th thread task. m CPU power consumption; This represents the memory-based dynamic power over the time interval [t, t+Δt]. Represents the CPU-based dynamic power over the time interval [t, t+Δt]; Util MEM Indicates memory utilization, Util CPU This represents CPU utilization; a1, a2, a3, and a4 are coefficients, which can be obtained through simple linear regression analysis. a1, a2, and a4 represent Utilization. MEM The degree of impact on memory-based dynamic power, a3 and a4 represent Util CPU The extent of the impact on CPU-based dynamic power.

[0067] Embodiment 2 of the present invention provides a task scheduling system based on the cross-regional Storm stream processing framework.

[0068] Reference Figure 3 As shown, the task scheduling system based on the cross-region Storm streaming framework includes: a master node, slave nodes, a resource monitoring module, a request latency monitoring module, a cost-benefit information database, a UI performance monitoring module, a data processing module, and a Storm custom scheduler. Specifically:

[0069] The master node is configured as a master server, which runs a daemon process called Nimbus. This daemon process is responsible for allocating resources, assigning tasks to slave nodes, and monitoring for failures.

[0070] The slave node is a worker node that executes tasks and runs a daemon called Supervisor. This daemon is responsible for listening to the work assignments on its computer and starting or stopping worker processes as needed.

[0071] The resource monitoring module sets up resource monitoring scripts to monitor the CPU and memory resource usage of slave nodes and writes the usage information into a cost-benefit information database. Specifically:

[0072] Each Supervisor node contains multiple workers (processes), and each worker process contains multiple executors. The monitoring script is used to monitor the CPU and memory resource usage of each slave node and write it into the cross-regional cost-effectiveness database.

[0073] The request latency monitoring module sets up a request latency monitoring script to monitor the request latency time of task scheduling to each cloud and writes the request latency time into a cross-regional cost-effectiveness database.

[0074] A cross-regional cost-benefit information database is used to store runtime data of the Storm cluster and the total cost C of each node executing tasks. total The target value P and the priority list;

[0075] The UI performance monitoring module is used to monitor task execution time and write the task execution time into a cross-regional cost-benefit information database.

[0076] The data processing module is responsible for processing the data collected by the resource monitoring script, request latency monitoring script, and UI monitoring module, and calculating the total cost C of the tasks executed by each node. total The target value P and the request delay time T total This data processing module is a standalone project written in Java code;

[0077] The Storm custom scheduler IScheduler (i.e., the task scheduler) calls the resource requirements and priority list of currently scheduled tasks in a cross-region cost-effectiveness database, and directly or indirectly controls the master and slave nodes to complete the scheduling and placement of tasks.

[0078] Specifically, refer to Figure 1As shown, in this embodiment of the invention, the Storm cluster consists of one local cloud (i.e., a data center) and three public clouds (i.e., data centers), including one master node and eleven slave nodes. The master node is configured as a master server, running a daemon called Nimbus. This daemon is responsible for allocating resources, assigning tasks to worker nodes, and monitoring for failures. Each worker node runs a daemon called Supervisor, which listens for work assignments on its computer and starts / stops worker processes as needed. The ZooKeeper cluster coordinates the interaction between the master node (Nimbus) and the slave nodes (Supervisor). The four data centers are located in different geographical locations, such as Shanghai, Beijing, Huizhou, and Guiyang. The master node resides in the local cloud, and each slave node has multiple worker processes and resource monitoring scripts (Shell). The resource monitoring scripts and UI performance monitoring module monitor cluster runtime data and write it to a cross-regional cost-benefit information database. The data processing module retrieves the necessary data from the cross-regional cost-benefit information database, processes it, and calculates the total cost C of each node executing tasks. total The target value P and the request delay time T total Based on the target value P, a priority list for task scheduling is constructed and stored in a cross-regional cost-benefit information database. The Storm custom scheduler prioritizes tasks on higher-priority nodes based on the priority list, thereby reducing scheduling costs in a geographically distributed cloud environment while ensuring request latency constraints.

[0079] Reference Figure 4 As shown, the task scheduling process based on a geographically distributed cloud environment in this embodiment of the invention includes:

[0080] S1: Based on a geographically distributed cloud environment, build a cross-regional Storm stream processing framework to schedule tasks among nodes in the entire cloud.

[0081] S2: After submitting the topology task, if the cost-benefit information of the task exists in the database of the cross-regional cost-benefit model, then obtain the resource requirements required for the current task. If the resource requirements are overloaded, adjust the parallelism and resubmit the topology task; otherwise, obtain the node cost-benefit information and request latency of all clouds, calculate the target value of all nodes executing task scheduling on each cloud based on the node cost-benefit information and request latency, construct a priority list for task scheduling based on the target value, and traverse the priority list to select nodes with higher priority to place the topology task.

[0082] S3: If the cost-benefit information for the topology task does not exist in the database, the task is scheduled using the Storm default scheduler, the runtime data of the slave nodes is monitored using the resource monitoring script, the request latency monitoring script is monitored using the request latency time of the task scheduled to each cloud, and the runtime data and request latency time of the slave nodes are written to the database.

[0083] S4: Use the data processing module to process the cluster runtime data, calculate the target value weighted by the cost information and request latency time of each node in the cloud, and write the target value into the database;

[0084] S5: Based on the target values ​​of all nodes in the cloud in the database, update the priority list, resubmit the topology task, and return to step S2.

[0085] The cost-benefit information of the task is the total cost C of each node executing the task. total .

[0086] The geographically distributed cloud environment is also commonly referred to as a cross-regional cloud environment.

[0087] This invention proposes a task scheduling method and system based on the cross-regional Storm streaming processing framework, improving the Storm system architecture and implementing a cost-minimizing task placement strategy in a cross-regional cloud environment. The advantages of this approach are that it makes Storm-based task scheduling more cost-effective and more adaptive in a cross-regional cloud environment. This scheduling method is applicable to big data application scenarios with varying data volumes and benchmark datasets.

[0088] The beneficial effects of this invention are as follows: This invention considers reducing cluster resource usage costs and energy consumption costs while mitigating request latency constraints. It takes into account communication latency caused by the geographical location of the user's task submission and the distance to clouds in different geographical locations, and tries to schedule tasks to data centers closer to the user. It also considers the different electricity prices and power usage effectiveness (PUE) at different locations, resulting in varying energy consumption costs. During task scheduling, the target value for each node in the entire cloud is calculated, and a scheduling priority list is generated based on the target value. Tasks are placed on higher-priority nodes first, thereby reducing the costs incurred by the cluster.

[0089] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A task scheduling method based on the cross-regional Storm stream processing framework, characterized in that, include: Step A: Based on a geographically distributed cloud environment, build a cross-region Storm stream processing framework to schedule tasks among nodes across the entire cloud. Step B: Calculate the total cost of task scheduling for each node based on the resource usage cost and energy consumption cost of all nodes executing tasks on each cloud. The specific calculation formula is as follows: , , , , , , , , , , , , in, This represents the total cost of each node executing its task. This represents the resource usage cost of each node executing the task. This represents the energy consumption cost of each node executing the task. This represents the resource usage cost of the i-th cloud, where i is a positive integer. , Represents the collection of all clouds. }, where n is the total number of clouds. This represents the unit price of the resource at the j-th node. Let j represent the j-th node in the i-th cloud, and t represent the task execution time. k represents the number of all nodes in the i-th cloud. This represents the last node in the i-th cloud. This is a binary decision variable, representing whether a node is in an active state. "Otherwise" indicates that node j is in an active state, while "otherwise" indicates that node j is in an inactive state. This represents the energy consumption cost of the i-th cloud. This represents the ratio of all energy consumed by the i-th cloud to the energy consumed by the IT load. This represents the energy price of the i-th cloud. This represents the j-th node in the i-th cloud. This represents the energy consumption of the j-th node. This represents the set of threads at node j. = }, Let m represent the energy consumption of the m-th thread, where m ∈ {1, 2, ..., g}, and g represents the number of threads. Represents the m-th thread Memory power consumption, Represents the m-th thread CPU power consumption; Indicates the time period Memory-based dynamic power Indicates the time period CPU-based dynamic power; Indicates memory utilization. Indicates CPU utilization; a 1、 a 2、 a3 and a4 represent different coefficients, a 1、 a2 represents respectively The extent of the impact on memory-based dynamic power, a 3、 a4 represents respectively The extent of the impact on CPU-based dynamic power; Step C: Calculate the request latency for task scheduling to each cloud based on the average response time of all nodes executing tasks on each cloud. The specific calculation formula is as follows: , , , , in, Let represent the average response time of all nodes executing task scheduling on the i-th cloud. This represents the transmission time of the i-th cloud. Represents the i-th cloud in the i-th cloud. The response time of each node. Indicates data transmission time Geographical distance It exhibits a linear growth relationship. Represents the Euclidean distance between clouds. The cloud to which the task is scheduled; Step D: Calculate the target value for all nodes executing task scheduling on each cloud based on the total cost and the request latency; Step E: Construct a priority list for task scheduling based on the target value, and traverse the priority list to select nodes with higher priorities to place tasks.

2. The task scheduling method based on the cross-regional Storm stream processing framework according to claim 1, characterized in that, The specific formula for calculating the target value is as follows: , in, Indicates the target value. This represents the total cost of each node executing its task. This indicates the latency of task scheduling to various clouds. This represents the influencing factor of the total cost of each node executing the task. This represents the impact factor on the request latency of task scheduling to various clouds.

3. A task scheduling system based on the cross-region Storm stream processing framework, the system being used to implement the task scheduling method based on the cross-region Storm stream processing framework as described in claim 1, characterized in that, This task scheduling system includes a master node, slave nodes, a resource monitoring module, a request latency monitoring module, a cross-regional cost-effectiveness information database, a UI performance monitoring module, a data processing module, and a Storm custom scheduler. Specifically: The master node is configured as a master server, which runs a daemon process called Nimbus. This daemon process is responsible for allocating resources, assigning tasks to slave nodes, and monitoring for failures. The slave node is a worker node that executes tasks and runs a daemon called Supervisor. This daemon is responsible for listening to the work assignments on its computer and starting or stopping worker processes as needed. The resource monitoring module sets up resource monitoring scripts to monitor the CPU and memory resource usage of slave nodes and write the usage information into a cross-regional cost-benefit information database. The request delay monitoring module sets up a request delay monitoring script to monitor the request delay time of task scheduling to each cloud and writes the request delay time into a cross-regional cost-benefit information database. A cross-regional cost-benefit information database is used to store runtime data of the Storm cluster and the total cost of tasks executed by each node. The target value and the priority list; The UI performance monitoring module is used to monitor task execution time and write the task execution time into a cross-regional cost-benefit information database. The data processing module is responsible for processing the data collected by the resource monitoring script, request latency monitoring script, and UI monitoring module, and calculating the total cost of the tasks executed by each node. The target value and the request delay time ; A custom Storm scheduler can access the resource requirements and priority list of currently scheduled tasks from a cross-regional cost-benefit information database, directly or indirectly controlling the master and slave nodes to schedule and place tasks.

Citation Information

Patent Citations

  • Cloud collaborative multi-task scheduling method and device

    CN111475274A

  • Cost benefit-based storm task scheduling method

    CN111522637A