Method for balancing load in real time in distributed stream computing engine Apache Flink

Abstract

The invention discloses a method for balancing load in real time in a distributed stream computing engine Apache Flink. The method comprises the following steps of: adding a restart function capable of redistributing subtasks and an interface of the restart function in an Flink kernel; a subtask distribution strategy for evaluating the subtask calculation amount according to the number of nodes and comprehensively considering the calculation capacity of the calculation nodes is added to the Flink; when the computing node is newly added, the workload of the flow task in execution is automatically balanced to the newly added computing node; and the workload of the node cluster is automatically balanced and calculated when the burst traffic is encountered. By means of an efficient restart function, a more balanced sub-task allocation strategy and a more real-time and automatic load balancing means, the load balancing of the distributed Flink system can be realized more quickly and better on the premise of ensuring that a user does not perceive, and the overall task execution efficiency and the resource utilization rate of a flow calculation engine are improved.

Description

Technical Field

[0001] The present invention relates to the field of stream computing technology, and in particular to a method for real-time load balancing in a distributed stream computing engine Apache Flink. Background Art

[0002] Stream computing technology can solve problems that traditional batch data processing methods cannot handle. Traditional batch data processing methods usually store data and then process it in batches. This method is incapable of handling applications with high real-time requirements. Stream computing can process data at the moment it is generated by real-time collection, calculation and feedback results, greatly reducing the loss of data timeliness and providing important support in the fields of finance, Internet of Things, and real-time traffic analysis. For stream computing technology, the mainstream solution is to rely on distributed systems to deploy stream computing engines. By splitting the entire stream computing task into subtasks that can be executed separately, this architecture can take advantage of the parallel computing subtasks of multi-machine computing node clusters to improve the performance of stream computing tasks, but it also introduces the problem of load imbalance between different machines in the computing node cluster.

[0003] The most popular distributed stream computing engine is Apache Flink. In Flink, a stream task may contain multiple operations, called vertices. Connecting vertices is the execution pipeline of the task, and each operation can customize the parallelism of the operation when executing. In this way, when executing tasks, a pipeline will be divided into multiple sub-pipelines for parallel execution. Different sub-pipelines will include different numbers of vertices due to inconsistent parallelism. The sub-pipelines here are subtasks. At the same time, Flink defines execution resources through task slots. Each computing node TaskManager has one or more task slots. Each task slot can run a subtask. Different task slots have independent memory space, but share the CPU of the computing node. The purpose of the subtask allocation strategy, also known as the resource allocation strategy, is to allocate subtasks to the appropriate task slot for execution. Flink detects the remaining resource quantity and CPU utilization on different machines in the computing node cluster TaskManagers when deploying tasks, and tries to evenly distribute subtasks to all computing nodes when allocating subtasks. However, there are differences in the amount of computation between different subtasks, which means that simply distributing subtasks to computing nodes evenly may result in multiple subtasks with large computational loads being assigned to the same computing node, which will become the performance bottleneck of the entire stream task. In addition, the solution that evenly distributes tasks by CPU utilization during subtask allocation may result in multiple subtasks with large computational loads being assigned to the same computing node, which will become the performance bottleneck of the entire stream task, because all subtasks have not started to execute at the time of allocation and the load status during execution is unknown. At the same time, these existing load balancing solutions are all performed when subtasks are allocated. During the running of the stream task, it is impossible to optimize the load of the executing stream task according to the real-time status of the computing node cluster.

[0004] Many technical methods have also emerged to solve the above-mentioned Flink problems in resource allocation and load balancing. For example, in terms of subtask allocation strategy, "Dynamic Resource Allocation Strategy for Flink Iterative Jobs" (Journal of Software, 2021) evaluates and allocates the required amount of resources in each step of the iterative job to reduce the problem of excessive resource allocation and waste or insufficient resource allocation and low task efficiency that may occur in static resource allocation. Similarly, the Chinese patent document with publication number CN113986523A discloses a Flink system resource allocation optimization method, system, device and medium, which uses the user-defined resource size applied for each time a task is submitted to avoid resource waste caused by the system pre-defining too large resources in advance and running tasks with small resource requirements. In order to solve the problem of increasing real-time load fluctuations in Flink computing node clusters, "Elastic Resource Scheduling Strategy for Flink Platform Based on Stream Network" (Journal on Communications 2019) proposes to establish a stream network model to detect whether there is accumulated data when the load fluctuates, so as to determine whether a performance bottleneck has been reached, and use the stream network to locate the performance bottleneck, and then expand the computing node capacity to achieve real-time load balancing.

[0005] However, previous methods have some limitations. (1) Adjusting the resource slot size only involves adjusting the memory allocation according to the subtask requirements. However, since different slots on the same Task Manager share CPU resources, these methods may still allocate subtasks with high CPU computing requirements to the same computing node, resulting in load imbalance during the subtask allocation stage. (2) When performance bottlenecks occur, load balancing is performed by expanding computing nodes, which brings challenges to operation and maintenance and increases system costs. (3) When faced with the situation of actively adding computing nodes, there is no effective way to automatically distribute the load of the tasks being executed to the newly added nodes, resulting in a waste of resources. Summary of the invention

[0006] The two subtask allocation strategies used by the most mainstream distributed stream computing engine Apache Flink, namely, simply allocating subtasks evenly to computing nodes and allocating subtasks according to CPU utilization, cannot measure the subtask computing amount in the subtask allocation stage to allocate subtasks more evenly, resulting in performance bottlenecks in stream tasks due to unbalanced loads on computing nodes. At the same time, the subtask allocation strategy that only acts on the subtask allocation stage cannot perform real-time operating status monitoring and automatic load optimization for stream tasks and computing node clusters that have been successfully deployed and are being executed. The present invention provides a method for real-time load balancing in the distributed stream computing engine Apache Flink.

[0007] A method for real-time load balancing in a distributed stream computing engine Apache Flink includes the following steps:

[0008] (1) Add a new task restart function to Flink and provide an external restart function interface for calling;

[0009] (2) Add a subtask allocation strategy for Flink that evaluates the computational effort of subtasks based on the number of nodes;

[0010] (3) Monitor the CPU utilization of each computing node in the computing node cluster. When the CPU utilization of one or some nodes increases before reaching the maximum, and the CPU utilization of the computing node reaches the maximum and does not decrease within ten minutes, if there are still idle task slots (the smallest unit of computing resources in Flink) in the computing node cluster at this time, and the CPU utilization of the computing node with idle task slots has not reached the maximum, then check the traffic of the subtasks on the computing node with the sudden increase in CPU utilization;

[0011] Find the flow task with the subtask that has the largest increase in traffic during the period of sudden increase in CPU utilization. For other flow tasks on the computing node, select the last successfully deployed flow task and use the restart function in step (1) to redeploy it. At the same time, use the strategy in step (2) to allocate subtasks, ignoring the computing node with sudden increase in CPU utilization at this time.

[0012] In step (1), a new function for restarting tasks is added to Flink, specifically:

[0013] Use the task execution plan graph generated during the stream task deployment process, ignore the existing subtask allocation plan, redeploy the specified stream task according to the subtask allocation strategy selected in the configuration, and then rely on the save point and checkpoint mechanism provided by Flink to restore the original execution state and continue execution.

[0014] In step (1), a calling interface is provided by adding a REST API, and the input parameter of the restart function interface is the ID of the flow task, which is used to specify the restart task.

[0015] By adding a restart button to the task execution interface in the Flink webpage source code and binding the restart function interface, you can click the button on the webpage to restart the current task.

[0016] In step (2), the subtask allocation strategy is specifically as follows:

[0017] When assigning subtasks, the subtasks that include the same nodes are grouped together, and the vertexNumber attribute is added to the group, which indicates the number of nodes contained in a single subtask in the group;

[0018] At the same time, for each computing node, first sort them in descending order according to the number of idle task slots, and then sort the computing nodes with the same number of idle task slots in ascending order according to the current CPU utilization; use the vertextNumber to sort the subtask groups in descending order, traverse the subtask groups and their subtasks in turn, and give priority to assigning task slots on the computing nodes with higher sorting to execute subtasks.

[0019] Step (3) also includes:

[0020] Monitor the number of computing nodes in the computing node cluster. When a new computing node appears, redeploy all the stream tasks running in Flink in the order of their successful deployment using the restart function in step (1). At the same time, use the method in step (2) to allocate subtasks. That is, when a new computing node is added, the workload of the running stream tasks is automatically distributed to the new node in a timely manner.

[0021] After step (3) is completed, if the computing node whose CPU utilization reached the maximum due to the sudden increase in traffic of the executed subtask is still in the state of maximum CPU utilization, and at the same time, there are idle task slots in the computing node cluster other than the computing node, and the CPU utilization of the computing node with idle task slots has not reached the maximum, and there is still surplus computing power, then the last successfully deployed flow task in the flow task where the subtask with non-surge in traffic is located is selected again for redeployment and subtask allocation, and the computing node with the sudden increase in CPU utilization is ignored during allocation; repeat the above operation until the following four situations occur, then stop the operation;

[0022] a. The CPU utilization of the computing node that had the highest CPU utilization due to burst traffic returns to normal levels;

[0023] b. There are no subtasks of other stream tasks on the computing node where the exception occurs;

[0024] c. There are no idle task slots in the computing node cluster except for the abnormal node;

[0025] d. The CPU utilization of other computing nodes with idle task slots is also about to reach the maximum.

[0026] At this point, dynamic load balancing can be achieved when facing burst traffic.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention adds a restart function to Flink, which minimizes the amount of operations required to redeploy stream tasks and continue to execute this process. The fast restart provides good performance for subsequent real-time load balancing, making it less noticeable to users. A new subtask allocation strategy for Flink that evaluates the computational amount of subtasks based on the number of nodes achieves load balancing in the subtask allocation stage while taking into account both the computational amount of subtasks and the capabilities of computing nodes, reducing the possibility of performance bottlenecks due to uneven distribution during deployment. The number of computing nodes in the computing node cluster and the CPU status of each computing node are monitored in real time, and dynamic adjustments are made when new computing nodes are added or the CPU of existing computing nodes reaches a bottleneck due to sudden traffic, so as to achieve real-time load balancing, which can reduce operation and maintenance time and manpower consumption while improving Flink's overall task execution efficiency and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the task state transition diagram after restart in Flink's new running state;

[0030] Figure 2 It is a flowchart of the newly added subtask allocation strategy that evaluates the computational amount of subtasks based on the number of nodes;

[0031] Figure 3 It is a flowchart of automatically balancing the load of computing nodes after adding computing nodes;

[0032] Figure 4 It is a flowchart of automatically balancing the load of computing nodes when facing burst traffic; DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0034] A method for real-time load balancing in a distributed stream computing engine Apache Flink includes the following steps:

[0035] (1) Add a new task restart function to Flink: Modify the kernel code of Flink and add the restart function. After the selected stream task receives the restart command, check whether the task status is Running. If it is Running, perform subsequent operations. Figure 1 As shown, first record the current task's stream task execution plan graph ExecutionGraph, which already contains all subtask information. Then stop the current task and add the attribute ignoreAllocation to the current task's ExecutionGraph. This attribute indicates that the current task will ignore the existing allocation plan when allocating subtasks, and add corresponding processing in subsequent related allocation operations. Otherwise, the default behavior will allocate subtasks according to the last allocation plan. Finally, use ExecutionGraph to redeploy the specified stream task according to the selected subtask allocation strategy. After redeployment, rely on the savepoints and checkpoints mechanism provided by Flink to restore the execution state before the task is restarted and continue to execute. At the same time, since Flink supports providing an external interface for functions by adding RESTAPI, the interface parameter of the REST API is set to the ID of the stream task. In this way, an external manual call method is provided to facilitate operation and maintenance operations.

[0036] (2) Add a subtask allocation strategy to Flink that evaluates the computational load of subtasks based on the number of nodes: A stream task may contain multiple operations, called vertices. Connecting the vertices together is the execution pipeline of the task, and each operation can customize the parallelism of the operation during execution. Each vertex will contain subvertices equal to the number of parallelism. The subvertices are connected from top to bottom according to the connection rules to form a subpipeline of the task. A single subvertice will only be included in one subpipeline. In this way, when actually executing the task, since the parallelism of each vertex is not completely consistent, different subpipelines will contain different numbers of subvertices. The subpipelines here are subtasks. At the same time, Flink defines execution resources through task slots. Each computing node TaskManager has one or more task slots. Each task slot can run a subtask. Different task slots have independent memory space, but share the CPU of the computing node. The purpose of the subtask allocation strategy is to allocate subtasks to appropriate task slots for execution.

[0037] Modify the Flink kernel code and add a new subtask allocation strategy, such as Figure 2 As shown. When allocating subtasks, subtasks including the same nodes are grouped together, and the attribute vertexNumber is added to the group, which indicates the number of nodes contained in a single subtask in the group. At the same time, for each computing node, first sort them in descending order according to the number of idle task slots, and then sort the computing nodes with the same number of idle task slots in ascending order according to the current CPU utilization. In this way, the computing nodes with the higher order have more available resources, indicating that the task slots on the computing nodes have stronger potential to execute subtasks and higher efficiency. Use vertexNumber to sort the subtask groups in descending order, traverse the subtask groups and the subtasks therein in turn, and give priority to assigning the task slots on the computing nodes with the higher order to execute subtasks. At the same time, after each assignment, the computing nodes to which the subtasks are assigned are reordered because the number of idle task slots of the computing nodes has changed. In this way, the load balancing in the subtask assignment stage is achieved while taking into account both the computing amount of the subtasks and the computing node capabilities.

[0038] (3) Monitor the number of computing nodes in the computing node cluster. When a new computing node appears, obtain the task IDs of all stream tasks running in Flink in the order of their successful deployment, and redeploy them using the restart function in step (1). At the same time, use the method in step (2) to allocate subtasks, such as Figure 3 As shown in the figure, during the restart process, subtasks will be redistributed and the last allocation plan will be ignored. At this time, the idle task slots of the newly added computing nodes will be added to the queue of subtasks to be allocated. In this way, the workload of all running stream tasks can be distributed to the newly added computing nodes, improving the overall task execution efficiency and resource utilization.

[0039] (4) Monitor the CPU utilization of each computing node in the computing node cluster. Under normal circumstances, when no new flow tasks are deployed, the CPU utilization of each computing node should be basically stable or decrease due to the completion of subtasks. When the CPU utilization of one or some nodes suddenly increases in a short period of time, causing the CPU utilization of the computing node to reach the maximum and not decrease for a period of time, this is because the receiving traffic or output traffic of a subtask being executed on the computing node suddenly increases, resulting in a sudden increase in the amount of computing, such as Figure 4As shown. If there are still free task slots in the computing node cluster at this time, and the CPU utilization of the computing nodes with free task slots has not reached the maximum, and there is surplus computing power, then check the received bytes and output bytes of the subtasks on the computing node with a sudden increase in CPU utilization, find the flow task where the subtask with the largest increase in receiving and sending traffic during the period of sudden increase in CPU utilization is located, and select the last successfully deployed flow task for other flow tasks on the computing node, and use the restart function in step (1) to redeploy it. At the same time, use the method in step (2) to allocate subtasks, and ignore the computing node with a sudden increase in CPU utilization during allocation.

[0040] (5) If, after the redeployment of the flow task in step (4) is completed, the computing node whose CPU utilization reached the maximum due to a surge in the amount of traffic sent and received by the executed subtask is still in the state of maximum CPU utilization, and there are idle task slots in the computing node cluster other than this computing node, and the CPU utilization of the computing nodes with idle task slots has not reached the maximum, and there is still surplus computing power, the last successfully deployed flow task in the flow task where the subtask with non-surge in traffic is located is selected again for redeployment and subtask allocation, and the computing node with a surge in CPU utilization is ignored during allocation.

[0041] (6) Repeat step (5) until the following four situations occur, then stop the operation.

[0042] 1. The CPU utilization of the computing node whose CPU utilization reaches the maximum due to burst traffic drops below the maximum value and returns to normal level.

[0043] 2. There are no subtasks of other stream tasks on the computing node where the exception occurs.

[0044] 3. There are no idle task slots in the computing node cluster except the abnormal node.

[0045] 4. The CPU utilization of other computing nodes with idle task slots is also about to reach maximum.

[0046] In this way, the workload of the computing node cluster can be balanced as much as possible when facing sudden increases in computing traffic without expanding the number of computing nodes, thereby improving the overall task execution efficiency.

[0047] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time load balancing in the distributed stream computing engine Apache Flink, characterized in that: The steps include: (1) Add a new task restart function to Flink and provide an external restart function interface for calling; (2) Add a subtask allocation strategy for Flink that evaluates the computational effort of subtasks based on the number of nodes; (3) Monitor the CPU utilization of each computing node in the computing node cluster. When the CPU utilization of one or some nodes increases before reaching the maximum, and causes the CPU utilization of the computing node to reach the maximum and does not decrease within ten minutes, if there are still idle task slots in the computing node cluster at this time, and the CPU utilization of the computing node with idle task slots has not reached the maximum, then check the traffic of the subtasks on the computing node with the sudden increase in CPU utilization; Find the flow task with the subtask that has the largest increase in traffic during the period of sudden increase in CPU utilization. For other flow tasks on the computing node, select the last successfully deployed flow task and use the restart function in step (1) to redeploy it. At the same time, use the strategy in step (2) to allocate subtasks, ignoring the computing node with sudden increase in CPU utilization at this time.

2. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 1 is characterized in that: In step (1), a new function for restarting tasks is added to Flink, specifically: Use the task execution plan graph generated during the stream task deployment process, ignore the existing subtask allocation plan, redeploy the specified stream task according to the subtask allocation strategy selected in the configuration, and then rely on the save point and checkpoint mechanism provided by Flink to restore the original execution state and continue execution.

3. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 1, characterized in that: In step (1), a calling interface is provided by adding a REST API, and the input parameter of the restart function interface is the ID of the flow task, which is used to specify the restart task.

4. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 3 is characterized in that: By adding a restart button to the task execution interface in the Flink webpage source code and binding the restart function interface, you can click the button on the webpage to restart the current task.

5. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 1, characterized in that: In step (2), the subtask allocation strategy is specifically as follows: When assigning subtasks, the subtasks that include the same nodes are grouped together, and the vertexNumber attribute is added to the group, which indicates the number of nodes contained in a single subtask in the group; At the same time, for each computing node, first sort them in descending order according to the number of idle task slots, and then sort the computing nodes with the same number of idle task slots in ascending order according to the current CPU utilization; use the vertextNumber to sort the subtask groups in descending order, traverse the subtask groups and their subtasks in turn, and give priority to allocating task slots on the computing nodes with the highest sorting to execute subtasks.

6. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 1, characterized in that: Step (3) also includes: Monitor the number of computing nodes in the computing node cluster. When a new computing node appears, redeploy all the stream tasks running in Flink in the order of their successful deployment using the restart function in step (1). At the same time, use the method in step (2) to allocate subtasks. That is, when a new computing node is added, the workload of the running stream tasks is automatically distributed to the new node in a timely manner.

7. The method for real-time load balancing in the distributed stream computing engine Apache Flink according to claim 1, characterized in that: After step (3) is completed, if the computing node whose CPU utilization reached the maximum due to the sudden increase in traffic of the executed subtask is still in the state of maximum CPU utilization, and at the same time, there are idle task slots in the computing node cluster other than the computing node, and the CPU utilization of the computing node with idle task slots has not reached the maximum, and there is still surplus computing power, then the last successfully deployed flow task in the flow task where the subtask with non-surge in traffic is located is selected again for redeployment and subtask allocation, and the computing node with the sudden increase in CPU utilization is ignored during allocation; repeat the above operation until the following four situations occur, then stop the operation; a. The CPU utilization of the computing node that had the highest CPU utilization due to burst traffic returns to normal levels; b. There are no subtasks of other stream tasks on the computing node where the exception occurs; c. There are no idle task slots in the computing node cluster except for the abnormal node; d. The CPU utilization of other computing nodes with idle task slots is also about to reach the maximum.

Images