Elastic Resource Allocation Method and Device Based on Dynamic Perception of Operator Scope

Abstract

The present invention discloses a method and apparatus for elastic resource allocation based on dynamic perception of operator scopes. The method includes: constructing a directed acyclic graph of a data stream processing application, where the nodes in the directed acyclic graph represent operators for processing data streams, and the edges represent data streams; evaluating the scope of an upstream operator according to the static screening rate index and the dynamic screening rate statistical index of the operator, and adaptively partitioning the directed acyclic graph. Predicting the load of each partition in a future period of time; for each operator in each of the partitions, generating a corresponding number of operator instances based on the load to perform real-time processing on the received data and meet the service quality requirements. The present invention plans the parallelism of operators in units of partitions, thereby ensuring end-to-end processing latency and improving resource utilization rate.

Description

Technical Field

[0001] The present invention relates to the field of data stream processing, and particularly to a method and device for elastic resource allocation based on dynamic awareness of operator scopes. Background Art

[0002] In distributed stream processing systems (hereinafter referred to as DSPSs), data stream processing applications (hereinafter referred to as DSPAs) are usually modeled as directed acyclic graphs (DAGs). The nodes in the graph represent operators for processing data streams, and the edges represent data streams. After an upstream operator finishes processing a data stream, it sends it to a downstream operator. When the load and parallelism of an upstream operator change, the load and parallelism of the downstream operator are also likely to change.

[0003] The original method only considers individual operators or static correlations between operators in the DAG graph when performing resource allocation, ignoring the dynamic correlations between operators in the DAG graph, resulting in lagging and inaccurate resource allocation and increasing data processing latency. To solve this problem, the dynamic scopes of operators are used to evaluate the dynamic correlations between operators to achieve elastic resource allocation. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method and device for elastic resource allocation based on dynamic awareness of operator scopes. The method plans the parallelism of operators in units of partitions, thereby ensuring end-to-end processing latency and improving resource utilization.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for elastic resource allocation based on dynamic awareness of operator scopes, the steps of which include:

[0007] Construct a directed acyclic graph of a data stream processing application, where the nodes in the directed acyclic graph represent operators for processing data streams, and the edges represent data streams;

[0008] Evaluate the scope of an upstream operator according to the static screening rate index and dynamic screening rate statistical index of the operator, and adaptively partition the directed acyclic graph.

[0009] Predict the load of each partition in the future for a period of time;

[0010] For each operator in each partition, generate a corresponding number of operator instances based on the load to perform real-time processing on the received data and meet the service quality requirements.

[0011] Further, the evaluating the scope of an upstream operator according to the static screening rate index and dynamic screening rate statistical index of the operator, and adaptively partitioning the DAG graph includes:

[0012] Infer the ratio of the theoretical output rate to the total input rate according to the processing logic of the operator to obtain the static screening rate;

[0013] Collect dynamic screening rate indicators;

[0014] In the startup phase, divide the operators into an initial stable operator set and an initial unstable operator set according to the static screening rate;

[0015] Truncate all input edges of the downstream operators of the operators in the initial unstable operator set, and aggregate all connected subgraphs to obtain the initial partition of the directed acyclic graph;

[0016] In the running phase, update the stable set and the unstable set according to the dynamic screening rate indicator to partition the directed acyclic graph again.

[0017] Further, the online prediction of the load of each partition for a period of time in the future includes:

[0018] Construct a training data set, which is the historical input data load of each partition;

[0019] Learn the meta-learning model based on the training data set to obtain an online load prediction model for each partition, where the meta-learning model includes: an MLP base learner and an LSTM meta-learner;

[0020] Input the historical load into the online load prediction model to obtain the load for the period of time in the future.

[0021] Further, the learning of the meta-learning model based on the training data set includes:

[0022] Input the historical input data load into the MLP base learner;

[0023] The MLP base learner is input into the LSTM meta-learner according to the loss function and the gradient of the loss function to update the cell state of the LSTM meta-learner;

[0024] The LSTM meta-learner updates the initial value of the MLP base learner.

[0025] Further, for each operator in each partition, generating a corresponding number of operator instances based on the load to process the received data to meet the service quality requirements includes:

[0026] Collect the load indicators of each partition and the parallelism indicators of the operators in each partition that meet the load processing requirements;

[0027] Perform supervised machine learning based on the load metrics and parallelism metrics to obtain an operator parallelism planning model for each partition;

[0028] Input the load into the operator parallelism planning model to obtain the number of operator instances for each operator in the partition, so as to perform real-time processing on the received data and meet the service quality requirements.

[0029] Further, the method of machine learning includes: random forest.

[0030] An elastic resource allocation device based on dynamic awareness of operator scope includes:

[0031] A graph construction module for constructing a directed acyclic graph of a data stream processing application, where the nodes in the directed acyclic graph represent operators for processing data streams, and the edges represent data streams;

[0032] An adaptive operator partitioning module that evaluates the scope of upstream operators according to the static screening rate index and dynamic screening rate statistical index of operators, and adaptively partitions the DAG graph;

[0033] An online load prediction module for predicting the load of each partition in the future for a period of time;

[0034] An operator parallelism planning module for generating corresponding numbers of operator instances for each operator in each of the partitions based on the load, so as to perform real-time processing on the received data and meet the service quality requirements.

[0035] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the data stream processing method based on dynamic resource allocation as described above.

[0036] A computer device, the computer device includes a memory and a processor, a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the data stream processing method based on dynamic resource allocation as described above.

[0037] A computer program product, when the computer program product runs on a computer device, it causes the computer device to execute the data stream processing method based on dynamic resource allocation as described above.

[0038] Compared with the prior art, the present invention has at least the following advantages:

[0039] 1. By dynamically adjusting the operator parallelism in advance, the end-to-end processing delay is guaranteed, and the resource utilization rate is improved;

[0040] 2. Resource allocation using the dynamic correlation between operators: Use static screening rate metrics and dynamic screening rate statistical metrics to evaluate the scope of upstream operators, adaptively divide the DSPA into multiple partitions, and plan the parallelism of operators in units of partitions.

[0041] 3. Online modeling of the dynamic correlation between operators within each partition: For the input load of each partition, calculate the optimal parallelism of each operator in the partition and update it dynamically.

[0042] 4. Real-time online prediction of load: We use meta-learning methods to online predict the load of each partition. Combine a strongly expressive long short-term memory (LSTM) meta-learner and an efficient multi-layer perceptron (MLP) base-learner to capture the fluctuation characteristics of the data stream in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The architecture diagram of the present invention.

[0044] Figure 2 The method flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] To make the above features and advantages of the present invention more obvious and understandable, the technical solutions of the present invention will be further described below through specific embodiments.

[0046] As Figure 1 shown, the present invention consists of three core modules: an online load prediction model, an adaptive operator partitioning model, and a partition-based operator parallelism planning model. Use the adaptive operator partitioning model to partition the DAG graph online based on the dynamic scope of the operator, use the online load prediction model to online predict the load of each partition to achieve active elastic resource allocation, and use the operator parallelism planning model to determine the optimal parallelism of the operators in each partition.

[0047] When processing the data stream, first use the metric collector to collect the dynamic screening rate statistical metrics of theoretically unstable operators, the load metrics of each partition, and the parallelism metrics of the operators in each partition that meet the load processing requirements, and store them in the metric database. Secondly, the adaptive operator partitioning model evaluates the scope of upstream operators according to the static screening rate metrics and dynamic screening rate statistical metrics of the operators, and adaptively partitions the DAG graph. Thirdly, the online load prediction model uses the load metrics to online predict the load of each partition for a period of time in the future. Next, the operator parallelism planning model uses the load and operator parallelism metrics of the partition to establish a random forest model in units of partitions, and plans the optimal parallelism of all operators in each partition to meet the load processing requirements based on the above load. Finally, the present invention refers to the scheduler in the previous work to place the instances of the operators in each partition on appropriate nodes.

[0048] Specifically, as Figure 2 shown, the working process of the present invention is as follows:

[0049] Step 1: Construct a directed acyclic graph of the data stream processing application.

[0050] Model the DSPAs into a DAG, where the nodes in the graph represent the operators for processing the data stream and the edges represent the data stream.

[0051] Step 2: Evaluate the scope of action of the upstream operator according to the static screening rate index and the dynamic screening rate statistical index of the operator, and adaptively partition the directed acyclic graph.

[0052] This step is implemented based on the adaptive operator partitioning model.

[0053] During the operation of the DSPAs, the operator sends the data stream to the downstream operator after processing it where is the set of all downstream operators of the operator The present invention uses r in (u) to represent the total input rate of the operator , and r out (u, v) to represent the data output rate sent from the upstream operator to the downstream operator . The screening rate of the operator is defined as the ratio of the output rate of the operator to the total input rate. According to the screening rate of the operator, the operator is divided into a stable operator and an unstable operator, specifically as follows:

[0054]

[0055] When the operator is deployed on the DSPSs, the present invention can infer the theoretically screening rate of the operator according to the processing logic of the operator, which is called the static screening rate; for the theoretically unstable operator, its actual screening rate can be obtained during the operation of the operator, which is called the dynamic screening rate.

[0056] The adaptive operator partitioner includes a startup phase and an operation phase. In the startup phase, the cold start problem of DAG partitioning is solved according to the static screening rate index of the operator. Specifically, first, the operator is divided into a stable operator set and an unstable operator set according to the static screening rate of the operator; second, all input edges of the downstream operator of the operator in the unstable set are truncated; finally, all connected subgraphs are aggregated into one partition. In the operation phase, the present invention analyzes the dynamic screening rate index of the unstable operator, updates the stable set and the unstable set, and performs DAG partitioning again.

[0057] Step 3: Predict the load of each partition in the future for a period of time.

[0058] This step is implemented based on an online load prediction model.

[0059] The load of each partition changes dynamically over time, showing characteristics of long-term trends and short-term fluctuations. The long-term trend refers to the changing trend and periodic change pattern of the load; the short-term fluctuation refers to the load jitter and sudden increase or decrease caused by some unexpected events. In order to capture the long-term trend and short-term fluctuation characteristics of the load and realize the online prediction of the load in the future for a period of time, the prediction model needs to be able to quickly update the model after receiving new data and calculate the latest inference results.

[0060] Compared with existing methods, the meta-learning model combines a meta-learner and a base learner, which can not only learn the long-term regular characteristics of the data, but also capture the short-term specific characteristics, and has strong non-linear generalization ability. The present invention proposes a meta-learning model based on an LSTM meta-learner and an MLP base learner. In the training stage, the MLP base learner trains the arriving small sample data to learn the short-term fluctuation characteristics, and the LSTM meta-learner summarizes the training results of the base learner and provides better initial values for the base learner. Specifically, after receiving new data, the MLP base learner inputs the loss function and the gradient of the loss function into the LSTM meta-learner to update the cell state, and the LSTM meta-learner updates the initial value of the MLP base learner.

[0061] Step 4: For each operator in each of the said partitions, generate a corresponding number of operator instances based on the load to perform real-time processing on the received data to meet the service quality requirements.

[0062] This step is implemented based on an operator parallelism planning model.

[0063] For each partition of the DAG, the present invention collects load and optimal operator parallelism data. Among them, the optimal operator parallelism refers to the minimum number of instances that can meet the processing requirements of the partition input load. Collecting data is time-consuming and the data volume is small. As the partition changes, the planner also needs to change dynamically, and as more data is collected, the planner also needs to be updated dynamically. Because the relationship between the load of each partition and the optimal operator parallelism is complex and non-linear, the present invention uses an ensemble learning method based on random forest regression for modeling.

[0064] Simulation Experiment

[0065] 1. Two GPU servers and six CPU servers. One of the GPU servers runs the job manager and the metric database, one GPU server runs and evaluates the resource allocation model, and six CPU servers run the operators of the preprocessing application.

[0066] 2. Send the data stream payload to the preprocessing application, count the end-to-end data processing latency and the number of operator start and stop times, and calculate the total overhead = the number of operator starts * start overhead + the number of operator stops * stop overhead + the number of operator instances * the running overhead of each operator instance.

[0067] 3. Evaluate the prediction performance by counting the Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) of the online load prediction model for each partition.

[0068] 4. Evaluate the planning performance by counting the Root Mean Square Error and Mean Absolute Error of the operator parallelism planning model for each partition.

[0069] Compared with the prior art, the present invention has made great progress both in terms of end-to-end processing latency and total system overhead.

[0070] In summary, first, elastic resource allocation is performed using the dynamic correlation between operators. The dynamic scope of the operator is evaluated using static screening rate indicators and dynamic screening rate statistical indicators. Based on this, DAG partitioning is adaptively performed, and the parallelism of the operator is planned in units of partitions. A correlation model between operators in each partition is established online using a random forest and dynamically updated. For the load of each partition, the optimal parallelism of each operator in the partition can be calculated. A meta-learning model is used to predict the load of each partition online, and a powerful LSTM meta-learner and an efficient MLP base-learner are fused to obtain the fluctuation characteristics of the data stream in real time.

[0071] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those of ordinary skill in the art can modify or equivalently replace the technical solutions of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for elastic resource allocation based on dynamic perception of operator scope, the steps of which include: Construct a directed acyclic graph of a data stream processing application, where the nodes in the directed acyclic graph represent operators for processing data streams, and the edges represent data streams; Evaluate the scope of upstream operators according to the static screening rate index and dynamic screening rate statistical index of operators, and adaptively partition the directed acyclic graph; among them, the step of evaluating the scope of upstream operators according to the static screening rate index and dynamic screening rate statistical index of operators and adaptively partitioning the directed acyclic graph includes: Infer the ratio of the theoretical output rate to the total input rate according to the processing logic of the operator to obtain the static screening rate; Collect dynamic screening rate indicators; In the startup phase, divide the operators into an initial stable operator set and an initial unstable operator set according to the static screening rate; Truncate all input edges of the downstream operators of the operators in the initial unstable operator set, and aggregate all connected subgraphs to obtain the initial partition of the directed acyclic graph; In the running phase, update the stable set and unstable set according to the dynamic screening rate indicators to partition the directed acyclic graph again; Predict the load of each partition in the future for a period of time; For each operator in each partition, generate a corresponding number of operator instances based on the load to perform real-time processing on the received data to meet the service quality requirements.

2. The method according to claim 1, wherein Online prediction of the load of each partition in the future for a period of time includes: Construct a training data set, where the training data set is the historical input data load of each partition; Learn the meta-learning model based on the training data set to obtain an online load prediction model for each partition, where the meta-learning model includes: an MLP basic learner and an LSTM meta-learner; Input the historical load into the online load prediction model to obtain the load in the future for a period of time.

3. The method according to claim 2, wherein The step of learning the meta-learning model based on the training data set includes: Input the historical input data load into the MLP basic learner; The MLP basic learner is input into the LSTM meta-learner according to the loss function and the gradient of the loss function to update the cell state of the LSTM meta-learner; The LSTM meta-learner updates the initial value of the MLP basic learner.

4. The method according to claim 1, wherein The step of, for each operator in each partition, generating a corresponding number of operator instances based on the load to process the received data to meet the service quality requirements includes: Collect the load indicators of each partition and the parallelism indicators of the operators in each partition that meet the load processing requirements; Perform supervised machine learning based on the load indicators and parallelism indicators to obtain an operator parallelism planning model for each partition; Input the load into the operator parallelism planning model to obtain the number of operator instances of each operator in the partition to perform real-time processing on the received data to meet the service quality requirements.

5. The method according to claim 4, characterized in that The method of machine learning includes: random forest.

6. An elastic resource allocation device based on dynamic perception of operator scope, including: A graph construction module for constructing a directed acyclic graph of a data stream processing application, where the nodes in the directed acyclic graph represent operators for processing data streams and the edges represent data streams; An adaptive operator partitioning module for evaluating the scope of upstream operators according to the static screening rate index and dynamic screening rate statistical index of operators, and adaptively partitioning the directed acyclic graph; wherein, the evaluating the scope of upstream operators according to the static screening rate index and dynamic screening rate statistical index of operators and adaptively partitioning the directed acyclic graph includes: Inferring the ratio of the theoretical output rate to the total input rate according to the processing logic of the operator to obtain the static screening rate; Collecting dynamic screening rate indicators; In the startup phase, dividing the operators into an initial stable operator set and an initial unstable operator set according to the static screening rate; Truncating all input edges of the downstream operators of the operators in the initial unstable operator set and aggregating all connected subgraphs to obtain an initial partition of the directed acyclic graph; In the running phase, updating the stable set and the unstable set according to the dynamic screening rate index to partition the directed acyclic graph again An online load prediction module for predicting the load of each partition in the next period of time; An operator parallelism planning module for generating corresponding numbers of operator instances for each operator in each of the partitions based on the load to perform real-time processing of received data to meet the service quality requirements.

7. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1-5 is implemented.

8. A computer device, the computer device includes a memory and a processor, a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1-5.

9. A computer program product, when the computer program product runs on a computer device, the computer device is caused to execute the method according to any one of claims 1-5.

Images