Distributed stream processing system parameter tuning method based on multi-objective reinforcement learning

Abstract

The application discloses a distributed stream processing system parameter tuning method based on multi-target reinforcement learning, first, a method for collecting the state of a stream processing job operator in real time is designed to provide data support for the tuning process; then, an offline multi-target reinforcement learning method is designed to quickly learn the optimal configuration value of the full parameter space under different weights of system performance and resource use efficiency; in addition, the application also designs a method for dynamically adjusting parameters to achieve smooth adaptation under dynamic workloads, ensure job graph compatibility, and minimize the cost of switching configurations in a production environment. Therefore, the application can quickly adaptively tune under dynamic workloads, effectively balance the performance and resource overhead of the distributed stream processing system, and reduce the operating cost of stream processing service users.

Description

Technical Field

[0001] This invention belongs to the field of distributed system optimization technology, specifically relating to a parameter tuning method for distributed stream processing systems based on multi-objective reinforcement learning. Background Technology

[0002] With the rapid development of big data technology and network applications, the demand for real-time computation of massive amounts of rapidly generated data covers multiple fields such as e-commerce, the Internet of Things, and machine learning, giving rise to distributed stream processing systems such as Apache Flink. These systems abstract continuously generated data into data streams and data computation operations into directed acyclic graphs (DAGs) connected by operators. Each operator has multiple parallel replicas running on different physical machines. Data flows into the system from upstream data sources, flows along the edges of the DAG between operators, and finally merges into a data pool. In the cloud computing era, distributed stream processing systems are typically provided as cloud-hosted services on a pay-as-you-go basis. Therefore, users need to minimize the use of resources such as processors and memory while maintaining system processing capacity. Through adaptive parameter tuning, users can dynamically adjust processing capacity and resource consumption according to the current system workload, thereby saving costs.

[0003] Parameter tuning in distributed stream processing systems faces numerous challenges: First, distributed stream processing systems have a vast number of parameters; for example, Apache Flink has over 200 parameters, making it difficult for traditional machine learning models to handle such a large parameter space. Second, parameter tuning in distributed stream processing systems is a multi-objective optimization problem, requiring a balance between system performance and resource consumption. The time-varying nature of the workload necessitates real-time adjustment of the weights of these two objectives, but existing parameter tuning methods typically only consider system performance as an optimization objective, failing to adapt to tuning requirements under different workloads. Furthermore, stream processing jobs are usually long-running; switching system configurations pauses the job and incurs significant overhead. If the new configuration alters the job graph structure, it may lead to unrecoverable operator states, causing job restart failures. Existing methods typically involve online learning of tuning strategies, which involves multiple configuration switches during the learning process and does not consider job graph compatibility issues.

[0004] Chinese patent application CN115934342A discloses a method, apparatus, device, and storage medium for job resource optimization. It obtains user-configured scaling policies through a resource monitor, collects real-time resource information and metrics of the Flink cluster, determines whether job resource optimization is necessary, and if so, sends the scaling policy to the scaling monitor for resource optimization. This method helps increase resource utilization efficiency and reduce user costs by monitoring Flink cluster metrics in real time and dynamically optimizing resources. However, the scaling policy requires manual configuration by the user, which cannot guarantee optimization performance on new jobs; furthermore, the operation only involves resource scaling and does not involve other parameters that may affect the behavior of the stream processing system.

[0005] Chinese patent application CN115687304A discloses a method and apparatus for optimizing Flink state storage. It uses a performance acquisition device to obtain system information and sends it to a tuning controller. The tuning controller determines whether parameter tuning is needed; if so, it sends the parameter model and tuning rules to a machine learning tuner. The machine learning tuner uses a Gaussian process regression model combined with a Bayesian optimization algorithm to derive parameter adjustment results and sends them to the tuning controller. The tuning controller dynamically adjusts the configuration parameters of the Flink state storage. This method uses a Bayesian optimization algorithm to learn optimal parameters and has a certain degree of adaptability. However, the tuning results obtained by Bayesian optimization are probabilistic and may generate invalid parameters, leading to multiple system restart attempts and additional overhead. Furthermore, this method only optimizes system performance and does not involve resource consumption, which can easily lead to resource excess and is not conducive to saving users' job operation costs.

[0006] Therefore, it is evident that existing distributed stream processing system parameter tuning methods only consider a portion of the configurable parameters and lack the ability to adjust all parameters. Secondly, existing distributed stream processing system parameter tuning methods have a single optimization objective and cannot dynamically adjust the balance between processing capacity and resource overhead according to workload. Furthermore, existing distributed stream processing system parameter tuning methods cannot control the number of restarts and job graph compatibility during online operation, resulting in high overhead for online tuning. Summary of the Invention

[0007] In view of the above, the present invention provides a parameter tuning method for distributed stream processing systems based on multi-objective reinforcement learning. This method is based on a multi-objective reinforcement learning framework, which enables efficient tuning of the complete parameter space of the distributed stream processing system, provides rapid adaptability under dynamic workloads, and effectively balances system performance and resource overhead.

[0008] A parameter tuning method for a distributed stream processing system based on multi-objective reinforcement learning includes the following steps:

[0009] (1) Use an index collector to continuously monitor the performance indicators of each operator in the distributed stream processing system;

[0010] (2) When training the model offline, maintain a set of reinforcement learning models. Each reinforcement learning model in the set uses different weights to perform multi-objective optimization on the throughput and resource utilization efficiency of the streaming processing job, forming the optimal set of models.

[0011] (3) Run each model in the optimal model set, collect its actions, rewards and corresponding job graph structures, and group all actions and rewards according to job graph structures;

[0012] (4) For each group, calculate the Pareto frontier based on the reward, and select the Pareto frontier with the highest quality for online optimization;

[0013] (5) During the online optimization process, the data source rate of the distributed stream processing system is obtained from the index collector, and the action with the highest quality Pareto front with a throughput greater than the data source rate and the minimum resource consumption is dynamically selected, and the parameter value corresponding to the action is applied to the distributed stream processing system.

[0014] Further, the specific implementation of step (1) is as follows: for any operator in the distributed stream processing system, its performance indicators are collected every 10 seconds, including idle time, busy time, backpressure time, number of input records per second and number of output records per second. The maximum, minimum and average values ​​of the above indicators are collected on all parallel instances. Finally, these performance indicator values ​​of all operators are concatenated to form the state vector of the distributed stream processing system.

[0015] Further, the specific implementation of step (2) is as follows: the parameter tuning process is modeled as a Markov decision process, including states, actions, and rewards, where the states are the state vectors of the distributed stream processing system, the actions are the values ​​of all configurable parameters of the distributed stream processing system, and the rewards are... Among them, throughput t The number of records received per second by the stream processing task, and resource utilization efficiency. The number of CPU cores used by a stream processing task is inversely proportional to the amount of memory; initialization includes or A collection of reinforcement learning models , where the model m i The weighted sum of the rewards is Each model uses different weights. α i , i =1,2,…, orEach reinforcement learning model is trained independently, and a set of optimal models under each reward weight is obtained by solving the Markov decision process.

[0016] Furthermore, the specific implementation of step (3) is as follows: for any model in the optimal model set, execute its strategy for several steps, and perform the action of each step in that round. a ,award r and the structure of the work diagram G Store to history In the above, the operation diagram structure G This refers to the directed acyclic graph formed by all operators in the stream processing job after applying the configuration options; after all models have been executed, all historical records will be... Classify according to the structure of the work diagram to form multiple subsets. .

[0017] Furthermore, the specific implementation of step (4) is as follows: First, calculate the reward for each subset. The Pareto front is calculated, then the hypervolume of each Pareto front is calculated, and finally the Pareto front with the largest hypervolume is selected for online tuning.

[0018] Further, the specific implementation of step (5) is as follows: continuously monitor the data source rate of the distributed stream processing system, and trigger the optimization operation when the data source rate fluctuation exceeds 10% of the data source rate at the time of the last optimization; after the optimization operation is triggered, select the action with the highest throughput and resource utilization efficiency from the selected Pareto front, and apply the parameter value corresponding to the action to the distributed stream processing system.

[0019] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described parameter tuning method for a distributed stream processing system based on multi-objective reinforcement learning.

[0020] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described parameter tuning method for a distributed stream processing system based on multi-objective reinforcement learning.

[0021] To address the parameter tuning problem of distributed stream processing systems in the field of distributed system optimization technology, this invention designs a method for real-time acquisition of the operator status of stream processing jobs to provide data support for the tuning process. Furthermore, this invention designs an offline multi-objective reinforcement learning method to quickly learn the optimal configuration values ​​of the entire parameter space under different weights of system performance and resource utilization efficiency. In addition, this invention also designs an online dynamic parameter adjustment method to achieve smooth adaptation under dynamic workloads, ensuring job graph compatibility and minimizing the cost of switching configurations in a production environment. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the parameter tuning method for a distributed stream processing system based on multi-objective reinforcement learning, as described in this invention. Detailed Implementation

[0023] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] This embodiment uses a large-scale real-time traffic flow prediction task as an example to illustrate the specific application of the technical solution of the present invention. In this task, sensors (such as cameras, geomagnetic coils, etc.) deployed throughout the city continuously generate traffic data (such as traffic flow, average vehicle speed, etc.), forming a high-speed data stream. These data streams are uploaded to a data collection gateway, and then sent to a distributed stream processing system for real-time computation via a message queue. In this system, the entire traffic flow prediction task is modeled as a directed acyclic graph, or operation graph, composed of multiple operators, including data input operators, data cleaning operators, data aggregation operators, and model-based prediction operators. To distribute the computational pressure of massive amounts of data, each operator can be deployed as multiple parallel instances distributed across different physical machines. Traffic data flows along the edges of the operation graph between operators, undergoes progressive computation, and finally generates prediction results which are output to downstream application systems.

[0025] It is worth noting that distributed stream processing systems have a large number of configurable operating parameters (such as operator parallelism, memory allocation, network buffer size, job graph optimization strategies, etc.), which directly affect the system's processing capacity and resource consumption. In actual operation, it is necessary to ensure that the system's throughput is always higher than the data input rate to avoid backpressure caused by data accumulation, while at the same time controlling the resource overhead such as processors and memory as much as possible to achieve cost-effectiveness optimization. To address this requirement, this implementation provides a parameter tuning method for distributed stream processing systems based on multi-objective reinforcement learning. This method can dynamically adjust the operating parameters of the distributed stream processing system according to the real-time traffic data load, thereby ensuring an effective balance between system performance and resource utilization in scenarios where the speed of traffic data generation changes dynamically, achieving the goal of energy saving and cost reduction. The specific implementation process of this method is as follows: Figure 1 As shown:

[0026] Step S11: Use the metrics collector to continuously monitor the performance metrics of each operator in the distributed stream processing system.

[0027] 1-1: Every 10 seconds, a request is sent to the metrics query interface of the distributed stream processing system to query the idle time, busy time, backpressure time, average number of input records per second, and average number of output records per second of all parallel instances of each operator in the past 10 seconds as performance metrics (5 in total).

[0028] 1-2: For each operator, calculate the maximum, minimum, and average values ​​of the above metrics across all parallel instances, and concatenate the metric values ​​of all operators to form the state vector of the entire stream processing system. For example, if the job contains... N If there are 15 operators, then the state vector dimension is 15. N .

[0029] Step S12: When training the model offline, maintain a set of reinforcement learning models. Each model uses different weights to perform multi-objective optimization on throughput and resource utilization efficiency in the reward, and iterates multiple times to form the optimal set of models.

[0030] 2-1: The parameter tuning process is modeled as a Markov decision process, including states, actions, and rewards; the state refers to the state vector of the distributed stream processing system collected by the index collector; the action refers to the values ​​of all configurable parameters of the distributed stream processing system; the reward is... Among them, throughput t The number of records received per second by the stream processing task, and resource utilization efficiency. The number of CPU cores used by a stream processing task is inversely proportional to the amount of memory.

[0031] Specifically, the state vectors collected in step S11 are modeled as the states of a Markov decision process. s Modeling all configurable parameter values ​​of a distributed stream processing system as actions of a Markov decision process. a ; Transform the binary vector Rewards modeled as Markov decision processes r Among them, throughput t Equals the sum of the number of input records per second for all operators directly connected to the upstream data source, representing resource utilization efficiency. ,in c The sum of the number of CPU cores used by all operators. m The sum of memory used by all operators, in GB. β This is a weighted hyperparameter used to balance the importance of CPU and memory.

[0032] 2-2: Initialization includes or A collection of reinforcement learning models , where the model m i The weighted sum of the rewards is Each model uses different weights. α i .

[0033] Specifically, the set M Each model in the algorithm employs the Soft Actor-Critic reinforcement learning algorithm, defining the optimization objective for each round as... ,in t For time steps, For the first t The time step is in the state s t Action as a t Weighted sum of rewards at the time , π As a strategy, For strategy π In state s t The entropy below, oh To control the temperature parameters for the exploration and utilization tradeoffs. For each model. m i Assign different reward weights α i The calculation formula is: This ensures that the weights are evenly distributed within the interval [0,1].

[0034] 2-3: Train each reinforcement learning model independently to solve the Markov decision process, and obtain a set of optimal models under each reward weight.

[0035] Specifically, each reinforcement learning model is trained independently using the Soft Actor-Critic algorithm. After training, the final policy network parameters of all models are saved, with each model corresponding to a weight. α i The following optimization strategy maximizes the reward.

[0036] Step S13: Run each model in the optimal model set, collect its actions, rewards and corresponding task graph structures, and group all actions and rewards according to task graph structures.

[0037] 3-1: For each optimal model, execute its policy. N Step, the action of each step in this round. a ,award r and the structure of the work diagram GStore to history In this context, the job graph refers to the directed acyclic graph composed of all operators in a stream processing job after the application of configuration options.

[0038] Specifically, create an empty collection of history records. This is used to store the execution results of all models; for each model in the optimal model set... m i ,implement N Step-by-step strategy, generating one action per step. a Receive corresponding rewards r And record the current job diagram structure. G The work diagram G Indicates application configuration action a A directed acyclic graph consisting of all operators and their connections in the post-stream processing job; the triples of each step... Add to history collection middle.

[0039] 3-2: After all the optimal models have been executed, classify all historical records according to the job graph structure to form multiple subsets. .

[0040] Specifically, extracting historical records All the work diagram structures in the set are used to form a work diagram collection. For each work diagram Create the corresponding subset .

[0041] Step S14: For each group, calculate the Pareto front based on the reward and select the Pareto front with the highest quality.

[0042] 4-1: Extract each subset Calculate the Pareto front for all rewards. Defined as A subset of, for all Does not exist Make and .

[0043] 4-2: Calculate the hypervolume for each Pareto front, where the Pareto front... super volume ,in n Pareto frontier Size, The subscripts of the elements are arranged in ascending order of throughput.

[0044] 4-3: Select the set of motions corresponding to the Pareto front with the largest supervolume. ,in It is used for online optimization.

[0045] Step S15: During the online tuning process, obtain the data source rate of the stream processing system from the index collector, dynamically select the action with a throughput greater than the data source rate and the minimum resource consumption from the highest quality Pareto front, and apply the parameter value corresponding to the action to the distributed stream processing system.

[0046] 5-1: Continuously monitor the data source rate of the stream processing job. When the data source rate fluctuates by more than 10% of the data source rate at the time of the last optimization, trigger the optimization operation.

[0047] Specifically, every 10 seconds, the metric collector collects the average number of input records per second from all data source nodes, and the sum is recorded as follows: l If the current data source rate l t lie in Outside the range, parameter tuning is triggered.

[0048] 5-2: When a tuning operation is triggered, select the action with the highest throughput and resource utilization efficiency from the selected action set, and apply the parameter value corresponding to that action.

[0049] Specifically, from the selected optimal set of actions A Select the subset of actions that satisfy the throughput constraints. ;from Select resource utilization efficiency Biggest movement , will the action The corresponding parameter configurations are applied to the distributed stream processing system to update system operating parameters; the current data source rate is recorded. l Waiting for the next workload change.

[0050] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A method for parameter tuning of a distributed stream processing system based on multi-objective reinforcement learning, the method comprising: The method comprises the following steps: (1) continuously monitoring performance indicators of each operator of the distributed stream processing system using an indicator collector; (2) when training the model offline, maintaining a set of reinforcement learning models, each reinforcement learning model in the set using different weights to perform multi-objective optimization on the throughput and resource use efficiency of the stream processing job to form an optimal model set; (3) running each model in the optimal model set, collecting its actions, rewards and corresponding job graph structure, and grouping all actions and rewards according to the job graph structure; (4) for each group, calculating a Pareto frontier according to the rewards, and selecting the highest-quality Pareto frontier for online tuning; (5) in the online tuning process, obtaining the data source rate of the distributed stream processing system from the indicator collector, dynamically selecting an action with a throughput greater than the data source rate and a minimum resource consumption from the highest-quality Pareto frontier, and applying the parameter values corresponding to the action to the distributed stream processing system.

2. The method of claim 1, wherein, The specific implementation of step (1) is that for any operator in the distributed stream processing system, the performance indicators are collected every 10 seconds, including idle time, busy time, back pressure time, input record number per second and output record number per second, and the maximum, minimum and average values of the above indicators on all parallel instances are collected; finally, these performance indicator values of all operators are spliced to form a state vector of the distributed stream processing system.

3. The method of claim 1, wherein, The specific implementation of step (2) is as follows: the parameter tuning process is modeled as a Markov decision process, including state, action and reward. The state is the state vector of the distributed stream processing system, the action is the value of all configurable parameters of the distributed stream processing system, and the reward is r = f(τ, ∈), where the throughput τ is the number of records received per second by the stream processing task, and the resource utilization efficiency ∈ is inversely proportional to the number of CPU cores and memory size used by the stream processing task; initialize a set M = {m1, m2, ..., m} containing η reinforcement learning models. η }, where model m i The weighted sum of the rewards is r = (1-α) i )τ+α i ∈, each model uses different weights α i , i = 1, 2, ..., η; train each reinforcement learning model independently, and obtain a set of optimal models under each reward weight by solving the Markov decision process.

4. The method of claim 1, wherein, The specific implementation of the step (3) is: for any model in the optimal model set, execute its strategy for several steps, and store the action a, the reward r and the job graph structure G of each step in the round to the history record In the specific implementation, the job graph structure G is a directed acyclic graph composed of all operators in the flow processing job after the application configuration item is applied; after all the models are executed, all the history records are classified according to the job graph structure to form a plurality of sub-sets 5. The method of claim 1, wherein, The specific implementation of the step (4) is: first, calculating the Pareto front of each sub-set according to the reward, then calculating the hyper-volume of each Pareto front, and finally selecting the Pareto front with the largest hyper-volume for online tuning. The specific implementation of the step (4) is: first, calculating the Pareto front of each sub-set according to the reward, then calculating the hyper-volume of each Pareto front, and finally selecting the Pareto front with the largest hyper-volume for online tuning.

6. The method of claim 1, wherein, The specific implementation of step (5) is that the data source rate of the distributed stream processing system is continuously monitored, and when the data source rate fluctuation exceeds 10% of the last tuning data source rate, a tuning operation is triggered; when the tuning operation is triggered, an action with a throughput greater than the current data source rate and a maximum resource use efficiency is selected from the selected Pareto frontier, and the parameter values corresponding to the action are applied to the distributed stream processing system.

7. A computer device comprising a memory and a processor, said memory having stored therein a computer program, characterized in that: The processor is configured to execute the computer program to implement the multi-objective reinforcement learning-based parameter tuning method for a distributed stream processing system according to any one of claims 1-6.

8. A computer readable storage medium storing a computer program, characterized in that: The computer program is configured to be executed by the processor to implement the multi-objective reinforcement learning-based parameter tuning method for a distributed stream processing system according to any one of claims 1-6.

Images