A method and system for elastic scaling of operators under fluctuating data streams

Abstract

The present invention relates to the field of distributed stream computing technology, and in particular to a method and system for elastic scaling of operators under fluctuating data streams. The method comprises: reading the configuration of a user and the data of a monitoring module; obtaining the number of operation instances in the current environment, adjusting the parallelism of the operator based on the current monitoring data and the number of operation instances, in combination with a delay constraint strategy and an elastic scaling strategy; relocating tasks between nodes through a scheduling manager according to the adjusted operator parallelism, and allocating task resources based on an adaptive scheduling strategy. The present invention implements a data monitoring module and a performance optimization module of As‑Stream, and integrates them into the distributed stream computing platform Apache Storm, and comprehensively evaluates system indicators from the perspectives of delay, throughput, resource utilization, and system load. Experimental results show that As‑Stream has a significant improvement in system performance compared with the elastic scaling method Autoscale+ at different data flow rates.

Description

Technical Field

[0001] The present invention relates to the field of distributed stream computing technology, and in particular to a method and system for elastically scaling operators under fluctuating data streams. Background Art

[0002] The elastic scaling of operators in a distributed stream computing environment is critical to the performance of the system. Existing elastic scaling strategies have adjustment lags, which can cause data overload in the system and also bring communication costs between tasks. How to reasonably adjust the parallelism of operators under latency constraints while maintaining low-latency processing remains a challenge.

[0003] High-speed continuous data streams have become ubiquitous in areas such as the Internet of Things, stock trading, real-time media streaming, and fraud detection. These areas require fast processing and analysis of real-time data streams to produce approximately correct results and return the processed results to enterprises or users. In order to process data generated in stream processing environments in real time with lower latency, a series of stream computing frameworks have emerged, including Spark Streaming, Apache Storm, Samza, Heron, and the popular open source Apache Flink. Framework, among which Apache Storm is one of the most popular open source big data stream computing systems. It processes real-time unbounded data in a distributed manner with millisecond latency. At the same time, it can provide users with scalability, fault tolerance, and availability. However, Storm's technology under fluctuating data streams is still insufficient. It cannot adjust the number of parallel tasks in the operator according to the fluctuating data stream, which will cause insufficient resources or idle resources, resulting in system performance degradation.

[0004] In order to effectively utilize resources and adapt to changes in data flows, a basic requirement is elasticity. The elastic runtime scaling strategy should be able to determine when and how to scale, and flexibly schedule resources based on the arrival rate of the current data flow. To achieve this goal, we first need to clearly understand the changes in the data flow rate of each vertex in the DAG, the consumption of communication resources between nodes, and which vertices of the graph need resource scaling, and then decide how to optimize it. At present, most of the existing research work has not considered the real-time resource requirements of application scheduling and the characteristics of real-time high-speed continuous flow, nor has it fully studied how to minimize the system response time and how to efficiently handle the trade-off between high performance and response time. Summary of the invention

[0005] In order to solve the technical problems that the existing technology does not consider the real-time resource requirements of application scheduling and the characteristics of real-time high-speed continuous flow, and does not fully study how to minimize system response time and how to efficiently handle the trade-off between high performance and response time, the embodiment of the present invention provides an operator elastic scaling method and system for fluctuating data streams. The technical solution is as follows:

[0006] On the one hand, a method for elastic scaling of operators under fluctuating data streams is provided, the method is implemented by an elastic scaling system for operators under fluctuating data streams, the system comprising: an automatic scaling module, an adaptive scheduling module and a monitoring module;

[0007] The method steps include:

[0008] S1, read the user's configuration data in the database and the monitoring data of the monitoring module;

[0009] S2. Obtain the monitoring data and the number of operators in the current environment; based on the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, adjust the parallelism of the operators through the automatic scaling module;

[0010] S3. According to the adjusted parallelism of the operator, the tasks corresponding to each operator are relocated between nodes through the adaptive scheduling module, and an adaptive scheduling strategy is defined to allocate task resources based on the adaptive scheduling strategy.

[0011] Optionally, in S1, reading the user's configuration data in the database and the monitoring data of the monitoring module includes:

[0012] Read the user's configuration data from the database;

[0013] The monitoring data of the monitoring module stored in the database is read, and the monitoring data includes: input flow rate data, system parameter configuration data and performance data.

[0014] Optionally, in S2, based on the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, the parallelism of the operators is adjusted through the automatic scaling module, including:

[0015] Based on the delay constraint strategy, the parallelism parameters of each operator are initialized according to the input rate of the data stream reaching each operator and the processing rate of the tasks corresponding to each operator;

[0016] Obtain the input rate and processing rate of the input tuple vertex, determine the bottleneck vertex on the topology through the elastic scaling strategy, and determine the bottleneck vertex as the bottleneck operator;

[0017] Identify bottleneck operators in the topology, feed monitoring data back to the predicted data load submodule, and predict the data load reaching the operator based on the delay constraint strategy;

[0018] Based on the predicted data load arriving at the operator, the elastic scaling strategy is used to calculate the number of tasks in the current operator and adjust the parallelism of the operator.

[0019] Optionally, obtain the input rate and processing rate of the input tuple vertex, and determine the bottleneck vertex on the topology through the elastic scaling strategy, including:

[0020] Preset the total residence time threshold of the input tuples and express the processing delay time of the input tuples in the stream application through mathematical expectation;

[0021] Model the processing delay of input tuples in a streaming application as a queueing network;

[0022] Based on the queuing network, it is judged whether the actual processing time of the input tuple is greater than the total residence time threshold. If it is greater, the parallelism of the bottleneck vertex is adjusted, and a task is added to the operator with the largest marginal benefit according to the queuing time of the task until the actual processing time of the input tuple is no greater than the total residence time threshold; if it is not greater, no adjustment is required.

[0023] Optionally, the data load arriving at the operator is predicted based on the delay constraint strategy; based on the predicted data load arriving at the operator, the number of tasks in the current operator is calculated through the elastic scaling strategy, and the parallelism of the operator is adjusted, including:

[0024] Based on the delay constraint strategy, the data load of the input tuple vertex in each time window is predicted through the linear regression prediction algorithm;

[0025] Traverse and identify the data load situation of each time window, calculate the number of tasks in the current operator through the elastic scaling strategy, and adjust the parallelism of the current operator based on the data load situation and the number of tasks in the current operator.

[0026] Optionally, S2 further includes:

[0027] The parallelism is preset in intervals. If the ratio of the input data volume to the processed data volume of the current vertex is not within the preset interval, the final parallelism is calculated based on the ratio of the operator input data volume to the capacity per unit time. The operator input data volume per unit time is obtained based on historical data.

[0028] Based on the amount of data input by the operator per unit time, a linear regression function is used to estimate the input amount of the current operator.

[0029] Optionally, in S3, according to the adjusted parallelism of the operator, the task corresponding to each operator is relocated between nodes through the adaptive scheduling module, and an adaptive scheduling strategy is defined, and task resources are allocated based on the adaptive scheduling strategy, including:

[0030] Obtain the adjusted parallelism of the operator and relocate the tasks corresponding to each operator between nodes through the adaptive scheduling module;

[0031] Through the traffic-aware adaptive scheduling strategy, the data transmission between tasks is monitored, the potential hot edges in the topology are identified based on the data transmission between tasks, and the tasks corresponding to the hot edges are assigned to the same Worker.

[0032] Optionally, the data transmission between tasks is monitored through a traffic-aware adaptive scheduling strategy, potential hot edges in the topology are identified based on the data transmission between tasks, and tasks corresponding to the hot edges are assigned to the same Worker, including:

[0033] Determine the number of workers required in the topology based on the resource requirements of the topology;

[0034] Get the task communication pairs in the topology and measure the number of tuples exchanged between tasks. When multiple tasks are running on the same component, sort the multiple task communication pairs in descending order according to the communication volume.

[0035] Assign task communication pairs corresponding to hot edges and place task communication pairs with high-frequency communication in the Worker with the lowest load;

[0036] The assigned Workers are assigned to the slots of different nodes in turn according to the data transmission between them.

[0037] On the other hand, a system for elastically scaling operators under fluctuating data streams is provided. The system is applied to a method for elastically scaling operators under fluctuating data streams. The system includes:

[0038] Monitoring module, used for collecting monitoring data;

[0039] The automatic scaling module is used to read the user's configuration data in the database and the monitoring data of the monitoring module; obtain the monitoring data and the number of operators in the current environment; based on the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, the automatic scaling module adjusts the parallelism of the operators;

[0040] The adaptive scheduling module is used to relocate the tasks corresponding to each operator between nodes according to the adjusted parallelism of the operator through the adaptive scheduling module, define an adaptive scheduling strategy, and allocate task resources based on the adaptive scheduling strategy.

[0041] On the other hand, a device for elastic scaling of operators under fluctuating data streams is provided, and the device for elastic scaling of operators under fluctuating data streams comprises: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for elastic scaling of operators under fluctuating data streams is implemented.

[0042] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned operator elastic scaling methods for fluctuating data streams.

[0043] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0044] The method provided by the present invention implements the data monitoring module and performance optimization module of As-Stream, and integrates them into the distributed stream computing platform Apache Storm, and comprehensively evaluates the system indicators from the perspectives of latency, throughput, resource utilization and system load. The experimental results show that As-Stream has a significant improvement in system performance compared with the elastic scaling method Autoscale+ under different data flow rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a flow chart of an operator elastic scaling method for fluctuating data streams provided by an embodiment of the present invention;

[0047] Figure 2 It is a design diagram of the As-Stream architecture provided by an embodiment of the present invention;

[0048] Figure 3 is a flow application processing example diagram provided by an embodiment of the present invention;

[0049] Figure 4 This is a block diagram of an operator elastic scaling system for fluctuating data streams provided by an embodiment of the present invention;

[0050] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0052] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0053] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0054] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be expressed in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0055] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0056] The embodiment of the present invention provides an operator elastic scaling method for fluctuating data streams. The method can be implemented by an operator elastic scaling device for fluctuating data streams. The operator elastic scaling device for fluctuating data streams can be a terminal or a server. The method is implemented by an operator elastic scaling system for fluctuating data streams. The system includes: an automatic scaling module, an adaptive scheduling module, and a monitoring module. Figure 1 The flowchart of the operator elastic scaling method for fluctuating data streams is shown in FIG. Figure 2 The As-Stream architecture design diagram is shown; the processing flow of the method may include the following steps:

[0057] S1, read the user's configuration data in the database and the monitoring data of the monitoring module;

[0058] In a feasible implementation manner, in S1, reading the user configuration and the data of the monitoring module includes:

[0059] Read the user's configuration data from the database;

[0060] The monitoring data of the monitoring module stored in the database is read, and the monitoring data includes: input flow rate data, system parameter configuration data and performance data.

[0061] S2. Obtain the monitoring data and the number of operators in the current environment; based on the user's configuration data, monitoring data, and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, adjust the parallelism of the operators through the automatic scaling module.

[0062] In a feasible implementation, the automatic scaling module decides whether to adjust the number of computing instances in the current environment based on the data read from the user's configuration and monitoring module. At the same time, after adjusting the parallelism of the operator, the task will be relocated between nodes through the scheduling manager. In Storm, you can implement a custom scheduling strategy by implementing the IsScheduler interface, and then Nimbus will generate a scheduling strategy from the topology based on the custom scheduling, and then write the generated scheduling strategy to Zookeeper, and then the Supervisor on each node will read the scheduling strategy from Zookeeper.

[0063] In a feasible implementation, in S2, based on monitoring data and the number of operators, combined with a delay constraint strategy and an elastic scaling strategy, the parallelism of the operators is adjusted through an automatic scaling module, including:

[0064] Based on the delay constraint strategy, the parallelism parameters of each operator are initialized according to the input rate of the data stream reaching each operator and the processing rate of the tasks corresponding to each operator;

[0065] Obtain the input rate and processing rate of the input tuple vertex, determine the bottleneck vertex on the topology through the elastic scaling strategy, and determine the bottleneck vertex as the bottleneck operator;

[0066] Identify bottleneck operators in the topology, feed monitoring data back to the predicted data load submodule, and predict the data load reaching the operator based on the delay constraint strategy;

[0067] Based on the predicted data load arriving at the operator, the elastic scaling strategy is used to calculate the number of tasks in the current operator and adjust the parallelism of the operator.

[0068] In a feasible implementation, the input rate and processing rate of the input tuple vertex are obtained, and the bottleneck vertex on the topology is determined through an elastic scaling strategy, including:

[0069] Preset the total residence time threshold of the input tuples and express the processing delay time of the input tuples in the stream application through mathematical expectation;

[0070] Model the processing delay of input tuples in a streaming application as a queueing network;

[0071] Based on the queuing network, it is judged whether the actual processing time of the input tuple is greater than the total residence time threshold. If it is greater, the parallelism of the bottleneck vertex is adjusted, and a task is added to the operator with the largest marginal benefit according to the queuing time of the task until the actual processing time of the input tuple is no greater than the total residence time threshold; if it is not greater, no adjustment is required.

[0072] In one feasible implementation, under the constraint of limited latency, the goal of As-Stream is to fully process every data tuple that arrives at the stream computing system in the application. For the input tuple of the application, Figure 2 The tuple processing in may go through multiple intermediate results, for example, operator A extracts the features of the data, and operator B recognizes the features.

[0073] In one possible implementation, when the input tuple is fully processed, that is, each intermediate result derived from the tuple has been processed by its corresponding operator. The total residence time is used to refer to the time period from when the tuple first arrives at the system to when the tuple is fully processed. Then, the goal of the stream computing system is to ensure that the expected total residence time of each input tuple does not exceed the user-specified duration, which is given by T max T is used to represent the total time that the data stream input stays in the stream application. In order to reduce the impact of fluctuations in tuple arrival rate and processing rate, the mathematical expectation E[T] is used to represent the processing delay time of the tuple in the stream application, that is, the expected value of T:

[0074] E[T](k)≤T max (1).

[0075] Using μ execi Represents vertex v i The average processing rate of each Executor in i Then it means that vertex v i The average arrival rate of tuples, λ in represents the average arrival rate of the external data stream input into the streaming application G. The basic idea of ​​estimating E[T] is to model it as a queuing network. The total residence time of the input tuple tuple is calculated by dividing its total processing time and total queuing delay. It is calculated by formula (2):

[0076]

[0077] From formula (2), it can be concluded that when k i ·μ execi ≤λ i, the processing rate of the cluster cannot keep up with the arriving tuples. As a result, the number of tuples in the vertex queue increases over time and the vertex becomes congested, resulting in infinite queuing delays. i ·μ execi >λ i , tuples are expected to be processed faster than they arrive. However, due to the randomness of the arrival rate and processing rate, the queue may still grow when the arrival rate temporarily exceeds the processing rate. Next, sum all E[T i ] to obtain the estimated value of E[T] for the entire topology graph G. According to the queuing network theory, E[T] is given by E[T i The expected value of ] is calculated by formula (3):

[0078]

[0079] The goal of the delay constraint algorithm is to achieve a lower delay target when the system processes data, which is described in detail in Algorithm 5-1 shown in Table 1 below.

[0080] Table 1 Delay Constraint Algorithm

[0081]

[0082] In one feasible implementation, in Algorithm 5-1, the parallelism parameters of each operator are first initialized according to the input rate of the data stream arriving at the operator and the processing rate of each task. Next, if the tuple processing time violates the user-defined threshold, the parallelism of the bottleneck vertex is adjusted. That is, a task is added to the operator with the largest marginal benefit according to the task's queuing time until the time is no greater than T max .

[0083] In a feasible implementation, the data load arriving at the operator is predicted based on the delay constraint strategy; based on the predicted data load arriving at the operator, the number of tasks in the current operator is calculated through the elastic scaling strategy, and the parallelism of the operator is adjusted, including:

[0084] Based on the delay constraint strategy, the data load of the input tuple vertex in each time window is predicted through the linear regression prediction algorithm;

[0085] Traverse and identify the data load situation of each time window, calculate the number of tasks in the current operator through the elastic scaling strategy, and adjust the parallelism of the current operator based on the data load situation and the number of tasks in the current operator.

[0086] In a feasible implementation manner, S2 further includes:

[0087] The parallelism is preset in intervals. If the ratio of the input data volume to the processed data volume of the current vertex is not within the preset interval, the final parallelism is calculated based on the ratio of the operator input data volume to the capacity per unit time. The operator input data volume per unit time is obtained based on historical data.

[0088] Based on the amount of data input by the operator per unit time, a linear regression function is used to estimate the input amount of the current operator.

[0089] In one possible implementation, As-Stream monitors the current performance of the system, (1) checks whether the performance degrades under the real-time constraint, (2) or when the system can meet the constraint with fewer resources, and (3) rescales the tasks in the operator elastically when (1) and (2) are satisfied. The main challenge lies in (2), how many processors the system needs to meet the real-time constraint requirements, and where to place them in the operator nodes. Specifically, given a certain number K max For the task, we will find the best allocation of these processors to the operators of the application to obtain the minimum expected total residence time min∑E[T](k). This minimization time constraint needs to satisfy formula (4).

[0090]

[0091] In the automatic scaling module, the first step is to obtain information about the input flow rate, system parameter configuration, and performance from the monitoring module, first identify the bottleneck operator in the topology, and then feed this information back to the prediction data load submodule to predict the data load reaching the operator, and then calculate the number of tasks in the current operator for adjustment. There are two ways to improve the performance of the bottleneck operator, namely vertical or horizontal scaling of the operator. For vertical expansion, it means an increase in the resources allocated to the Worker. In contrast, for horizontal expansion, the Worker's resources are not changed, but the parallelism of the bottleneck operator is increased, and the throughput of the operator is changed by adjusting the number of copies of the operator. Elastic scaling mainly expands in the horizontal direction, adjusting the parallelism of operators with a fixed number of physical nodes.

[0092] The elastic scaling strategy is used to identify bottleneck vertices in the topology in real time. When the arrival rate of the data stream fluctuates, the queue network is used to estimate the overall processing time of the tuples in the topology. Subsequently, the bottleneck vertices in the topology G are identified, and the linear regression prediction algorithm is applied to these bottleneck vertices to predict the load of the vertices in the next time window. At the same time, the new parallelism is calculated for each bottleneck vertex. Its core goal is to minimize the response time of the topology.

[0093] At the same time, we need to determine the bottleneck vertex on the topological graph path G. Assume that vertex vi is the bottleneck node on path G, n is v i The number of Executors. i ] represents the bottleneck vertex v i The finite set of n Executors can be expressed by formula (5).

[0094] Exec(v i )={exec i，1 ,exec i，2 , ..., exec i，n} (5)

[0095] Among them, Exec ij Indicates v i If the total processing time E[T](k) of the tuples in the stream computing subsystem exceeds the user-defined maximum threshold T max When , it indicates that the vertices in the topology are congested. First, we identify the bottleneck vertices in G, k i ·μ execi =λ i ,when p cur Indicates the current parallelism of the vertex, p min represents the lower limit of the current vertex parallelism, which can be calculated by formula (6). α is the factor for adjusting the scaling. We assume that v i It is the bottleneck vertex, and its parallelism begins to be adjusted. i >p cur , key vertex v i The processing capacity cannot meet the current data arrival rate, resulting in congestion, which may generate an infinite waiting queue. It is necessary to adjust its parallelism to improve the processing capacity of the vertex. In addition, if k i <p min , then you need to dynamically adjust the number of Executors to improve the current resource utilization.

[0096] p min =p cur α, α∈[0,1] (6)

[0097] To adjust the vertex v i The parallelism is first calculated based on the current input rate λ of the vertex. i And the average processing rate of each Executor Next, the present invention uses a linear regression prediction algorithm to predict the vertex v based on the historical data of the previous time window. iThe data load of the next time window is used to adjust the parallelism of the operator, which can be calculated by formula (7):

[0098]

[0099] Among them, adjustP is the parallelism that needs to be adjusted for the current vertex, EsInput ij is the vertex v i The predicted load of the next time window data is calculated by formula (9). ij is the capacity of the next time window of the operator, which can be calculated by formula (8). ij-1 Represents vertex v i The parallelism of the previous window. In order to adjust the parallelism of the vertex more accurately, the parallelism that needs to be set for the vertex is determined based on the predicted value and the parallelism of the previous time window.

[0100]

[0101] EsInput ij =max(In ij ,∑EsOut ij )+queue ij-1 (9)

[0102] Among them, Δ is the size of the time window, Latency ij To handle the delay of tuples, the capacity of operators in a unit time window can be estimated based on it. ij Represents the next time window v predicted using linear regression i The data load is calculated by formula (10). EsOut is accumulated ij Represents the current vertex v i The number of output tuples of all parent vertices is calculated by formula (11).

[0103]

[0104]

[0105] Where f is the input amount of tuples in the time window calculated based on the linear regression function calculated in the previous iteration, which can be calculated using formula (12). i ={m i1 , m i2 ,…,m in} represents a timestamp set, that is, within the current time window. Each m ip All belong to the next iteration within the window. ij is the vertex v i The input prediction value of the parent vertex of It is the selection factor of its output, which represents the ratio of the number of tuples processed per unit time.

[0106]

[0107] Where X represents the number and timestamp of tuples, and θ is the threshold size of X. θ is calculated based on the historical input data volume of the vertex, and a regression equation is constructed to predict the number of tuples. In the process of adjusting the parallelism, a scaling cooldown time is also set to make the adjustment effective. The elastic scaling algorithm is described in Algorithm 5-2 in Table 2 below.

[0108] Table 2 Operator scaling algorithm

[0109]

[0110] In Algorithm 5-2, the bottleneck vertex on the topology G is first determined based on the input rate and processing rate of the vertex. Next, a queuing network is used to estimate the time from when the tuple enters the topology until it is completely processed. Then the goal is to ensure that the expected residence time of each tuple does not exceed the user-defined time threshold Tmax. If the user-defined threshold is violated or a bottleneck vertex appears in the topology, start adjusting the parallelism of the bottleneck vertex. The present invention sets an interval for the parallelism. If the ratio of the input data volume and the processed data volume of the current vertex is not within the interval, start adjusting the parallelism of this vertex, that is, the final parallelism is obtained based on the ratio of the input volume and capacity of the operator per unit time. The amount of data input by the operator per unit time is based on historical data, and a linear regression function is used to estimate the input volume of the current operator. In addition, in order to avoid frequent adjustment of the parallelism of the operator, which causes a decrease in system performance, it is necessary to reasonably set the scaling time scaleTime.

[0111] S3. According to the adjusted parallelism of the operator, the tasks corresponding to each operator are relocated between nodes through the adaptive scheduling module, and an adaptive scheduling strategy is defined to allocate task resources based on the adaptive scheduling strategy.

[0112] In a feasible implementation, in S3, according to the adjusted parallelism of the operator, the task corresponding to each operator is relocated between nodes through the adaptive scheduling module, and an adaptive scheduling strategy is defined, and task resources are allocated based on the adaptive scheduling strategy, including:

[0113] Obtain the adjusted parallelism of the operator and relocate the tasks corresponding to each operator between nodes through the adaptive scheduling module;

[0114] Through the traffic-aware adaptive scheduling strategy, the data transmission between tasks is monitored, the potential hot edges in the topology are identified based on the data transmission between tasks, and the tasks corresponding to the hot edges are assigned to the same Worker.

[0115] In a feasible implementation, the data transmission between tasks is monitored through a traffic-aware adaptive scheduling strategy, potential hot edges in the topology are identified based on the data transmission between tasks, and tasks corresponding to the hot edges are assigned to the same Worker, including:

[0116] Determine the number of workers required in the topology based on the resource requirements of the topology;

[0117] Get the executor communication pairs in the topology and measure the number of tuples exchanged between executors. When one or more executors are running on the same component, sort the multiple executor communication pairs in descending order according to the communication volume.

[0118] Assign Executor communication pairs corresponding to hot edges, and place Executor pairs with high-frequency communication in the Worker with the lowest load;

[0119] The assigned Workers are assigned to the slots of different nodes in turn according to the data transmission between them.

[0120] In a feasible implementation, when a user submits a stream application or modifies the parallelism of an operator, it may lead to non-optimal scheduling of tasks, resulting in excessively high communication costs when data is transmitted between upstream and downstream tasks. Therefore, allocating tasks based on communication is the key to task scheduling. The purpose of scheduling is to consider the communication cost between Executors and schedule Executors with high-frequency communication to the same Worker to minimize the communication cost between Executors and nodes. Through experimental analysis, there are three main factors that cause communication resource consumption: (1) Workers between different nodes in the cluster; (2) between Worker processes on the same node; and (3) between different Executor threads in the same Worker. Normally, the communication resources between different threads in the same Worker process can be ignored. Therefore, this paper focuses on the communication costs between the first two types of tasks.

[0121] In order to reduce the communication cost between tasks, a traffic-aware adaptive scheduling strategy is designed. The main idea is to monitor the data transmission between Executors, identify potential hot edges in the topology based on the data transmission between Executors, and then assign the Executors corresponding to the hot edges to the same Worker (Slot). It is mainly divided into the following three steps:

[0122] (1) Determine the number of workers required in the topology based on the resource requirements of the topology. At the same time, for the Executor communication pairs in the topology, measure the number of tuples exchanged between the Executors. One or more Executors run on the same component (Spout or Bolt), and sort them in descending order according to their communication volume.

[0123] (2)(2) To allocate Executor communication pairs corresponding to hot edges, it is necessary to place the Executor pairs with high-frequency communication in the Worker with the lowest load.

[0124] (3)(3) The workers assigned in (2) are assigned to the slots of different nodes in turn according to the data transmission between the workers, thereby reducing the communication cost between the nodes.

[0125] In streaming applications, unreasonable task scheduling will result in excessively high data transmission costs between upstream and downstream tasks. Therefore, it is necessary to collect topology runtime information, data transmission rates between executors, resource consumption of each executor, and available resources on computing nodes. Then, the minimum number of workers required in the topology can be calculated based on the statistical information, which can be calculated using formula (13).

[0126]

[0127] Where Num w is the appropriate number of Workers to run the topology, R(G) is the amount of resources required by topology t during operation, which can be obtained through statistical information, Cap(Worker) is the threshold of the average capacity of each Worker, Num E is the number of all executors in topology t. Since the computing power and resources on different nodes are different, Cap(Worker) needs to be calculated based on the average capacity of Workers on different nodes, which is calculated using formula (14):

[0128]

[0129] Comput(n i ) represents the computing power of the i-th computing node, Num slot(ni) represents the number of slots on the i-th node, that is, n i The number of workers on topology t is 1, and scheduling is feasible only when the number of workers on topology t is less than the number of slots. Next, we start scheduling and allocating data based on the data communication between executors (1) to assign executors to workers, and (2) to assign workers to nodes n. i It must satisfy the restrictions that the load of the Executor pairs assigned to the Worker is less than the capacity of the Worker and the load of the Worker assigned to the node is less than the capacity of the node.

[0130] The goal of scheduling is to minimize the amount of communication between Executors. The scheduling strategy is described in detail in Algorithm 5-3.

[0131] Table 3 Task allocation algorithm

[0132]

[0133] In a feasible implementation, in Algorithm 5-3, for topology t, first sort them in descending order according to the amount of communication between Executors. Then traverse the Executor communication pairs in turn. For each Executor pair, if both Executors are not assigned, assign them to the Worker process with the least current load. Otherwise, if one of them is assigned and the other is not assigned, assign the unassigned Executor to the other assigned Worker. If the Worker load at this time is less than its capacity, assign it to it, otherwise assign them to the Worker with the smallest load. If both pairs have been assigned, put the Worker of the assigned Executor pair and the Worker with the smallest load into the set W, calculate the Cartesian product, and then check the allocation of all combinations of these Workers in turn, with the goal of finding the allocation that produces the lowest traffic between Workers.

[0134] In Algorithm 5-3, after assigning the Executor to the Worker, the next task is to assign the Worker to the slots of different nodes to minimize the consumption of communication resources between nodes. It is mainly divided into the following steps: (1) Sort the Worker communication pairs in the cluster in descending order according to their communication volume. (2) Assign the Worker communication pairs corresponding to the hot edges and place the Worker pairs with high-frequency communication on the node with the lowest load. Therefore, it is necessary to collect performance information during cluster runtime, such as the data transmission rate between Workers.

[0135] Its allocation must meet the constraints of node capacity. Its node allocation strategy is described in detail in Algorithm 5-4. The goal is to minimize the amount of communication between nodes.

[0136] Table 4 Node selection algorithm

[0137]

[0138] In Algorithm 5-4, the input includes a set of nodes, a set of workers, and the available capacity of the nodes, as well as the set of executors that have been allocated in Algorithm 5-4. The output is the allocation of workers on different nodes. First, the executor set is determined to be empty. If it is empty, it means that no executor needs to be rescheduled, and the workers do not need to be reallocated. Next, the communication volume between workers is counted and sorted in descending order. Then, the node with the smallest load is selected, and if the worker communication pairs have not been allocated, they are allocated to the node with the smallest load. If the worker communication pairs have been allocated, the set of nodes where they are located is obtained. If the worker communication pairs are not on the same node, the Cartesian product of the node sets is traversed separately to find the combination of worker allocation nodes with the lowest communication volume.

[0139] In the embodiment of the present invention, through experimental observation, under fluctuating data flow, the amount of data arriving at the operator will far exceed the processing capacity of the parallel operator tasks, resulting in insufficient number of tasks to process the excess data, causing system performance to degrade. Conversely, the amount of data arriving at the operator may be far lower than the processing capacity of the operator, resulting in idle resources.

[0140] By constructing a stream application model, the logical structure of the topology and the data flow processing process are described; by constructing a resource model, the operator scaling under limited resources is described; by constructing a real-time data flow model, the characteristics of fluctuating data flows and their calculation methods are clarified; by constructing a task communication model, the calculation of communication costs between tasks is quantified.

[0141] For data processing delay bottlenecks, we use queueing network theory to estimate the delay of tuple processing, so that we can dynamically scale operators according to delay constraints at runtime. For operator scaling problems, we use prediction algorithms to predict the data load of time windows and calculate the parallelism required by operators. For high communication costs between tasks, we identify hot edges in the topology based on data transmission between upstream and downstream tasks, and map the tasks corresponding to the hot edges to the same node.

[0142] The present invention implements As-Stream and integrates it into the distributed stream computing system Apache Storm, and comprehensively evaluates system indicators from the perspectives of system latency, throughput, resource utilization, and system load.

[0143] Figure 4 The block diagram of a system for elastically scaling operators under fluctuating data streams is shown according to an exemplary embodiment. The system is used for an elastically scaling method for operators under fluctuating data streams. Figure 4 The system includes a monitoring module 310, an automatic scaling module 320 and an adaptive scheduling module 330. Among them:

[0144] Monitoring module 310, used to collect monitoring data;

[0145] The automatic scaling module 320 is used to read the user's configuration data in the database and the monitoring data of the monitoring module; obtain the monitoring data and the number of operators in the current environment; based on the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, adjust the parallelism of the operators through the automatic scaling module;

[0146] The adaptive scheduling module 330 is used to relocate the tasks corresponding to each operator between nodes according to the adjusted parallelism of the operator through the adaptive scheduling module, define an adaptive scheduling strategy, and allocate task resources based on the adaptive scheduling strategy.

[0147] Optionally, the monitoring module 310 is used to collect input flow rate data, system parameter configuration data and performance data.

[0148] Optionally, an automatic scaling module 320 is used to initialize the parallelism parameter of each operator based on the delay constraint strategy according to the input rate of the data stream reaching each operator and the processing rate of the task corresponding to each operator;

[0149] Obtain the input rate and processing rate of the input tuple vertex, determine the bottleneck vertex on the topology through the elastic scaling strategy, and determine the bottleneck vertex as the bottleneck operator;

[0150] Identify bottleneck operators in the topology, feed monitoring data back to the predicted data load submodule, and predict the data load reaching the operator based on the delay constraint strategy;

[0151] Based on the predicted data load arriving at the operator, the elastic scaling strategy is used to calculate the number of tasks in the current operator and adjust the parallelism of the operator.

[0152] Optionally, the input rate and processing rate of the input tuple vertex are obtained, and the bottleneck vertex on the topology is determined through the elastic scaling strategy, including:

[0153] Preset the total residence time threshold of the input tuples and express the processing delay time of the input tuples in the stream application through mathematical expectation;

[0154] Model the processing delay of input tuples in a streaming application as a queueing network;

[0155] Based on the queuing network, it is judged whether the actual processing time of the input tuple is greater than the total residence time threshold. If it is greater, the parallelism of the bottleneck vertex is adjusted, and a task is added to the operator with the largest marginal benefit according to the queuing time of the task until the actual processing time of the input tuple is no greater than the total residence time threshold; if it is not greater, no adjustment is required.

[0156] Optionally, the data load arriving at the operator is predicted based on the delay constraint strategy; based on the predicted data load arriving at the operator, the number of tasks in the current operator is calculated through the elastic scaling strategy, and the parallelism of the operator is adjusted, including:

[0157] Based on the delay constraint strategy, the data load of the input tuple vertex in each time window is predicted through the linear regression prediction algorithm;

[0158] Traverse and identify the data load situation of each time window, calculate the number of tasks in the current operator through the elastic scaling strategy, and adjust the parallelism of the current operator based on the data load situation and the number of tasks in the current operator.

[0159] Optionally, the automatic scaling module 320 is further used to preset the interval of parallelism. If the ratio of the input data volume and the processed data volume of the current vertex is not within the preset interval, the final parallelism is obtained according to the ratio of the operator input data volume and the capacity per unit time; wherein the operator input data volume per unit time is obtained according to historical data;

[0160] Based on the amount of data input by the operator per unit time, a linear regression function is used to estimate the input amount of the current operator.

[0161] Optionally, an adaptive scheduling module 330 is used to obtain the adjusted parallelism of the operator, and relocate the tasks corresponding to each operator between the nodes through the adaptive scheduling module;

[0162] Through the traffic-aware adaptive scheduling strategy, the data transmission between tasks is monitored, the potential hot edges in the topology are identified based on the data transmission between tasks, and the tasks corresponding to the hot edges are assigned to the same Worker.

[0163] Optionally, the data transmission between tasks is monitored through a traffic-aware adaptive scheduling strategy, potential hot edges in the topology are identified based on the data transmission between tasks, and tasks corresponding to the hot edges are assigned to the same Worker, including:

[0164] Determine the number of workers required in the topology based on the resource requirements of the topology;

[0165] Get the task communication pairs in the topology and measure the number of tuples exchanged between tasks. When multiple tasks are running on the same component, sort the multiple task communication pairs in descending order according to the communication volume.

[0166] Assign task communication pairs corresponding to hot edges and place task communication pairs with high-frequency communication in the Worker with the lowest load;

[0167] The assigned Workers are assigned to the slots of different nodes in turn according to the data transmission between them.

[0168] In the embodiment of the present invention, through experimental observation, under fluctuating data flow, the amount of data arriving at the operator will far exceed the processing capacity of the parallel operator tasks, resulting in insufficient number of tasks to process the excess data, causing system performance to degrade. Conversely, the amount of data arriving at the operator may be far lower than the processing capacity of the operator, resulting in idle resources.

[0169] By constructing a stream application model, the logical structure of the topology and the data flow processing process are described; by constructing a resource model, the operator scaling under limited resources is described; by constructing a real-time data flow model, the characteristics of fluctuating data flows and their calculation methods are clarified; by constructing a task communication model, the calculation of communication costs between tasks is quantified.

[0170] For data processing delay bottlenecks, we use queueing network theory to estimate the delay of tuple processing, so that we can dynamically scale operators according to delay constraints at runtime. For operator scaling problems, we use prediction algorithms to predict the data load of time windows and calculate the parallelism required by operators. For high communication costs between tasks, we identify hot edges in the topology based on data transmission between upstream and downstream tasks, and map the tasks corresponding to the hot edges to the same node.

[0171] The present invention implements As-Stream and integrates it into the distributed stream computing system Apache Storm, and comprehensively evaluates system indicators from the perspectives of system latency, throughput, resource utilization, and system load.

[0172] Figure 5 is a schematic diagram of a structure of an operator elastic scaling device for fluctuating data streams provided by an embodiment of the present invention, such as Figure 5 As shown, the operator elastic scaling device for fluctuating data streams may include the above Figure 4 The operator elastic scaling system for fluctuating data streams is shown. Optionally, the operator elastic scaling device 410 for fluctuating data streams may include a first processor 2001 .

[0173] Optionally, the operator elastic scaling device 410 for fluctuating data streams may also include a memory 2002 and a transceiver 2003 .

[0174] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0175] Combine the following Figure 5 The components of the operator elastic scaling device 410 for fluctuating data streams are specifically introduced as follows:

[0176] The first processor 2001 is the control center of the operator elastic scaling device 410 for fluctuating data streams, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0177] Optionally, the first processor 2001 may perform various functions of the operator elastic scaling device 410 for fluctuating data streams by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0178] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0179] In a specific implementation, as an embodiment, the operator elastic scaling device 410 for fluctuating data streams may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0180] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0181] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0182] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0183] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0184] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0185] It should be noted that Figure 5 The structure of the operator elastic scaling device 410 for fluctuating data flows shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0186] In addition, the technical effects of the operator elastic scaling device 410 for fluctuating data streams can refer to the technical effects of the operator elastic scaling method for fluctuating data streams described in the above method embodiment, which will not be repeated here.

[0187] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0188] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0189] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0190] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0191] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0192] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0193] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the system or unit, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0198] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0199] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for elastic scaling of operators under fluctuating data streams, characterized in that: The method is implemented by an operator elastic scaling system for fluctuating data streams, the system comprising: an automatic scaling module, an adaptive scheduling module and a monitoring module; Methods include: S1, read the user's configuration data in the database and the monitoring data of the monitoring module; S2. Obtain monitoring data and the number of operators in the current environment; based on the user's configuration data, the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, adjust the parallelism of the operators through the automatic scaling module; The delay constraint strategy includes: when the tuple processing time violates the threshold, the parallelism parameter of each operator is initialized according to the input rate of the data stream arriving at the operator and the processing rate of each task; The elastic scaling strategy includes: when the data stream arrival rate fluctuates, identify the bottleneck vertex in the topology G, use the prediction algorithm to predict the load of the vertex in the next time window, and calculate the new parallelism for each bottleneck vertex to minimize the response time of the topology; Adjusting the parallelism of the operator includes: determining the bottleneck vertex on the topology G according to the vertex input rate and processing rate; using the queuing network to calculate the time from the tuple entering the topology to being fully processed, ensuring that the expected residence time of each tuple does not exceed the threshold ; S3. Relocate tasks in each operator between nodes through adaptive scheduling according to the adjusted parallelism of the operator, and allocate task resources based on the adaptive scheduling strategy; For the adjusted operator parallelism, the tasks corresponding to each operator are relocated between nodes through the adaptive scheduling module; Through the traffic-aware adaptive scheduling strategy, the data transmission between tasks is monitored, and the potential hot edges in the topology are identified based on the data transmission between tasks, and the tasks corresponding to the hot edges are assigned to the same Worker. Determine the number of workers required in the topology based on the resource requirements of the topology; Get the task communication pairs in the topology and measure the number of tuples exchanged between tasks. When multiple tasks are running on the same component, sort the multiple task communication pairs in descending order according to the communication volume. Assign task communication pairs corresponding to hot edges and place task communication pairs with high-frequency communication in the Worker with the lowest load; The assigned Workers are assigned to the slots of different nodes in turn according to the data transmission between them.

2. The operator elastic scaling method for fluctuating data streams according to claim 1, characterized in that: In S1, the configuration data of the user in the database and the monitoring data of the monitoring module are read, including: Read the user's configuration data from the database; The monitoring data of the monitoring module stored in the database is read, wherein the monitoring data includes: input flow rate data, system parameter configuration data and performance data.

3. The operator elastic scaling method for fluctuating data streams according to claim 1, characterized in that: In S2, based on the user's configuration data, the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, the parallelism of the operators is adjusted through the automatic scaling module, including: Based on the delay constraint strategy, the parallelism parameter of each operator is initialized according to the input rate of the data stream reaching each operator and the processing rate of the task corresponding to each operator; Obtaining the input rate and processing rate of the input tuple vertex, determining the bottleneck vertex on the topology through an elastic scaling strategy, and determining the bottleneck vertex as a bottleneck operator; Identify bottleneck operators in the topology, feed monitoring data back to the predicted data load submodule, and predict the data load reaching the operator based on the delay constraint strategy; Based on the predicted data load arriving at the operator, the elastic scaling strategy is used to calculate the number of tasks in the current operator and adjust the parallelism of the operator.

4. The operator elastic scaling method for fluctuating data streams according to claim 3, characterized in that: The step of obtaining the input rate and processing rate of the input tuple vertex and determining the bottleneck vertex on the topology through an elastic scaling strategy includes: Preset the total residence time threshold of the input tuples and express the processing delay time of the input tuples in the stream application through mathematical expectation; Model the processing delay of input tuples in a streaming application as a queueing network; Based on the queuing network, it is judged whether the actual processing time of the input tuple is greater than the total residence time threshold. If it is greater, the parallelism of the bottleneck vertex is adjusted, and a task is added to the operator with the largest marginal benefit according to the queuing time of the task until the actual processing time of the input tuple is no greater than the total residence time threshold; if it is not greater, no adjustment is required.

5. The operator elastic scaling method for fluctuating data streams according to claim 4, characterized in that: The method predicts the data load of the operator based on the delay constraint strategy; calculates the number of tasks in the current operator through the elastic scaling strategy based on the predicted data load of the operator, and adjusts the parallelism of the operator, including: Based on the delay constraint strategy, the data load of the input tuple vertex in each time window is predicted through the linear regression prediction algorithm; The data load situation of each time window is traversed and identified, the number of tasks in the current operator is calculated through an elastic scaling strategy, and the parallelism of the current operator is adjusted based on the data load situation and the number of tasks in the current operator.

6. The operator elastic scaling method for fluctuating data streams according to claim 5, characterized in that: Said S2 also includes: The parallelism is preset in intervals. If the ratio of the input data volume to the processed data volume of the current vertex is not within the preset interval, the final parallelism is calculated based on the ratio of the operator input data volume to the capacity per unit time. The operator input data volume per unit time is obtained based on historical data. Based on the amount of data input by the operator per unit time, a linear regression function is used to estimate the input amount of the current operator.

7. An operator elastic scaling system for fluctuating data streams, the operator elastic scaling system for fluctuating data streams is used to implement the operator elastic scaling method for fluctuating data streams as claimed in any one of claims 1 to 6, characterized in that: The system comprises: Monitoring module, used for collecting monitoring data; The automatic scaling module is used to read the user's configuration data in the database and the monitoring data of the monitoring module; obtain the monitoring data and the number of operators in the current environment; based on the user's configuration data, the monitoring data and the number of operators, combined with the delay constraint strategy and the elastic scaling strategy, adjust the parallelism of the operator through the automatic scaling module; The adaptive scheduling module is used to relocate the tasks corresponding to each operator between nodes according to the adjusted parallelism of the operator through the adaptive scheduling module, and define an adaptive scheduling strategy, and allocate task resources based on the adaptive scheduling strategy.

8. An operator elastic scaling device for fluctuating data streams, characterized in that: The operator elastic scaling device for fluctuating data streams includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images