A method for balancing load in real time in a distributed stream computing engine Apache Flink

Abstract

The application discloses a method for balancing load in real time in a distributed stream computing engine Apache Flink, comprising adding a restart function and an interface thereof which can perform subtask redistribution in a Flink kernel; adding a subtask distribution strategy to the Flink, which can evaluate subtask computing amount according to the number of nodes and comprehensively consider the computing capacity of the computing nodes; automatically balancing the work load of the stream task being executed to the newly added computing node when the computing node is added; and automatically balancing the work load of the computing node cluster when encountering burst traffic. The application can faster and better realize load balancing of the distributed Flink system, improve the overall task execution efficiency and resource utilization of the stream computing engine under the premise of guaranteeing no user perception, by virtue of the efficient restart function, the more balanced subtask distribution strategy and the more real-time and automatic load balancing means.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of stream computing, and particularly relates to a method for balancing load in real time in a distributed stream computing engine Apache Flink. BACKGROUND

[0002] The stream computing technology can solve problems that cannot be processed by the traditional batch data processing mode. The traditional batch data processing mode usually stores data and then processes the data in batches. This mode is not competent when dealing with applications with high real-time requirements. The stream computing can process data in the moment of data generation through real-time collection, calculation and feedback of results, greatly reduces the loss of timeliness of data, and provides important support in the fields of finance, Internet of Things, traffic real-time analysis and the like. For the stream computing technology, the mainstream scheme is to deploy a stream computing engine by relying on a distributed system, and to split the entire stream computing task into subtasks that can be executed separately. This architecture can utilize the advantage of parallel computing of subtasks by the multi-machine computing node cluster to improve the performance of the stream computing task, but at the same time, the problem of load imbalance between different machines in the computing node cluster is introduced.

[0003] The most popular distributed stream computing engine is Apache Flink. In Flink, a stream task may contain multiple operations, called vertices. The vertices are connected to form a pipeline of the task. Each operation can customize the parallelism of the operation. When a task is executed, a pipeline is divided into multiple sub-pipelines to be executed in parallel. Different sub-pipelines may include different numbers of vertices due to different parallelism. The sub-pipeline is called a sub-task. Flink defines the execution resources by task slots. Each computing node TaskManager has one to multiple task slots. Each task slot can run a sub-task. Different task slots have independent memory spaces but share the CPU of the computing node. The purpose of the sub-task allocation strategy, also called the resource allocation strategy, is to allocate sub-tasks to appropriate task slots for execution. Flink detects the remaining resource quantity and CPU utilization of different machines in the computing node cluster TaskManagers when deploying a task. When allocating sub-tasks, Flink tries to evenly distribute the sub-tasks to all computing nodes. However, there is a difference in the amount of calculation between different sub-tasks, which may cause a simple average allocation of sub-tasks to the computing nodes to allocate multiple sub-tasks with large amounts of calculation to the same computing node. This computing node becomes the performance bottleneck of the entire stream task. In addition, the task allocation scheme based on CPU utilization during sub-task allocation may also allocate multiple sub-tasks with large amounts of calculation to the same computing node. This computing node becomes the performance bottleneck of the entire stream task. At the same time, these existing load balancing schemes are performed when the sub-tasks are allocated. The running stream task cannot be optimized according to the real-time computing node cluster during the execution of the stream task.

[0004] To solve the above problems of Flink in resource allocation and load balancing, many technical methods have emerged. For example, in terms of subtask allocation strategy, Dynamic Resource Allocation Strategy for Flink Iterative Jobs (Journal of Software, 2021) evaluates and allocates the required amount of resources in each step of the iterative job to reduce the problem of excessive resource allocation and low task efficiency caused by static resource allocation. Similarly, the Chinese patent document CN113986523A discloses a Flink system resource allocation optimization method, system, device and medium, which adopts the method of applying for resource size according to user-defined each time the task is submitted, to avoid resource waste caused by system pre-definition of too large resources and running of tasks with small resource demand. In solving the problem of real-time load fluctuation of Flink computing node cluster, Flink Platform Elastic Resource Scheduling Strategy Based on Flow Network (Journal on Communications, 2019) proposes to establish a flow network model to detect whether there is accumulated data when the load fluctuates, to determine whether the performance bottleneck is reached, and then to locate the performance bottleneck using the flow network, and then to expand the computing nodes to achieve real-time load balancing.

[0005] However, the previous methods have some limitations. (1) Adjusting the resource slot size only adjusts the memory allocation according to the subtask demand, but since different slots on the same Task Manager share CPU resources, these methods may still allocate subtasks with high CPU computing capacity to the same computing node, resulting in load imbalance in the subtask allocation stage. (2) When performance bottlenecks occur, load balancing is achieved by expanding computing nodes, which challenges the operation and maintenance work and increases the system cost. (3) When facing the case of actively increasing computing nodes, there is no effective method to automatically distribute the task load being executed to the newly added nodes, causing resource waste. SUMMARY

[0006] The two subtask allocation strategies of simply allocating subtasks to computing nodes and allocating subtasks according to CPU utilization used by the most mainstream distributed stream computing engine Apache Flink cannot measure the subtask computing amount in the subtask allocation stage to more evenly allocate subtasks, resulting in performance bottlenecks of stream tasks due to unbalanced load of computing nodes, and the subtask allocation strategy only acting on the subtask allocation stage cannot monitor the running status of the stream tasks and the computing node cluster in real time and automatically optimize the load, the application provides a method for balancing load in real time in the distributed stream computing engine Apache Flink.

[0007] A method for balancing load in real time in the distributed stream computing engine Apache Flink, comprising the following steps:

[0008] (1) adding a new function of restarting tasks to Flink, and providing an external restart function interface for calling;

[0009] (2) adding a subtask allocation strategy for evaluating subtask computing amount according to the number of nodes to Flink;

[0010] (3) monitoring the CPU utilization of each computing node in the computing node cluster, when the CPU utilization of one or more nodes increases under the premise that the CPU utilization does not reach the maximum, and the CPU utilization of the computing node reaches the maximum and does not decrease within ten minutes, if there are idle task slots (the smallest unit of computing resources in Flink) in the computing node cluster at this time, and the CPU utilization of the computing node with idle task slots does not reach the maximum, then the traffic of the subtasks on the computing node with the CPU utilization explosion is checked;

[0011] finding the stream task where the subtask with the most increased traffic during the CPU utilization explosion, for other stream tasks on the computing node, selecting the last successfully deployed stream task to use the restart function in step (1) to redeploy, and using the strategy in step (2) to allocate subtasks, ignoring the computing node with the CPU utilization explosion at this time.

[0012] In step (1), the new function of restarting tasks is added to Flink, specifically:

[0013] using the task execution plan graph generated in the stream task deployment process, ignoring the existing subtask allocation plan, redeploying the specified stream task according to the selected subtask allocation strategy in the configuration, and then relying on the save point and checkpoint mechanism provided by Flink to restore the original execution state and continue execution.

[0014] In step (1), a call interface is provided by adding a REST API. The input parameter of the restart function interface is the ID of the stream task, which is used to specify the restart task.

[0015] By adding a restart button to the task execution interface in the Flink web page source code and binding the restart function interface, the current task can be restarted by clicking the button on the web page.

[0016] In step (2), the subtask allocation strategy is as follows:

[0017] When assigning subtasks, subtasks that include the same nodes are grouped together, and the group is given the attribute vertexNumber, which indicates the number of nodes contained in a single subtask within the group.

[0018] Meanwhile, for each computing node, it is first sorted in descending order according to the number of idle task slots. Computing nodes with the same number of idle task slots are then sorted in ascending order according to the current CPU utilization. The subtask groups are sorted in descending order using vertextNumber, and the subtask groups and their subtasks are traversed in turn. The task slots of the computing nodes with higher sorting are assigned to execute the subtasks first.

[0019] Step (3) also includes:

[0020] Monitor the number of computing nodes in the computing node cluster. When a new computing node is added, redeploy all running stream tasks in Flink according to the order of deployment success time using the restart function in step (1). At the same time, use the method in step (2) to allocate subtasks. That is, when a new computing node is added, the workload of the stream tasks being executed is automatically distributed to the new node in a timely manner.

[0021] After step (3) is completed, if the computing node whose CPU utilization reached its maximum due to the surge in traffic of the executed subtask is still in the state of maximum CPU utilization, and there are idle task slots in the computing node cluster other than the computing node, and the CPU utilization of the computing node with idle task slots has not reached its maximum and there is still spare computing power, then select the last successfully deployed stream task in the stream task where the non-traffic surge subtask is located, redeploy and reassign the subtask, ignoring the computing node with the surge in CPU utilization at this time during the reassignment; repeat the above operation until the following four situations occur, then stop the operation;

[0022] a. The CPU utilization of computing nodes that reached their maximum due to sudden traffic surges returns to normal levels;

[0023] b. The abnormal computing node has no other sub-tasks of the stream task;

[0024] c. There is no idle task slot in the computing node cluster except the abnormal node;

[0025] d. The CPU utilization of other computing nodes with idle task slots is also about to reach the maximum.

[0026] Up to now, dynamic load balancing in the face of burst traffic can be realized.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The present application is a newly added restart function of Flink, which minimizes the amount of operation of redeploying stream tasks and continuing to execute the process, and the fast restart provides good performance for subsequent real-time load balancing, making the user more unaware. The sub-task allocation strategy newly added by the present application according to the node number to evaluate the sub-task calculation amount realizes the balanced load in the sub-task allocation stage while considering the sub-task calculation amount and the computing node capacity, and reduces the possibility of performance bottleneck due to uneven allocation during deployment. The number of computing nodes of the computing node cluster and the CPU state of each computing node are monitored in real time, and dynamic adjustment is made when new computing nodes appear, the CPU of the existing computing nodes reaches the bottleneck due to burst traffic, etc., to realize real-time load balancing, which can reduce the operation and maintenance time and manpower consumption while improving the overall task execution efficiency and resource utilization of Flink BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a task state transition diagram after restart in the newly added running state of Flink;

[0030] Figure 2 is a flow diagram of the newly added sub-task allocation strategy according to the node number to evaluate the sub-task calculation amount;

[0031] Figure 3 is a flow diagram of automatic balancing of computing node load after adding a computing node;

[0032] Figure 4 is a flow diagram of automatic balancing of computing node load in the face of burst traffic; DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in combination with the drawings and embodiments, and it should be pointed out that the following embodiments are intended to facilitate the understanding of the present application and do not limit the present application in any way.

[0034] A method for real-time load balancing in the distributed stream computing engine Apache Flink includes the following steps:

[0035] (1) Add a new feature to Flink: Modify the Flink kernel code to add a restart function. After the selected streaming task receives a restart command, check if the task status is Running. If it is Running, execute subsequent operations. Figure 1 As shown, the process first records the current task's execution plan graph (ExecutionGraph), which already contains all subtask information. Then, the current task is stopped, and the `ignoreAllocation` attribute is added to the ExecutionGraph. This attribute indicates that the current task will ignore existing allocation plans when assigning subtasks, and corresponding processing will be added in subsequent allocation operations. Otherwise, the default behavior is to allocate subtasks according to the previous allocation plan. Finally, the ExecutionGraph is used to redeploy the specified stream task according to the selected subtask allocation strategy. After redeployment, the execution state before the task restart is restored using Flink's savepoints and checkpoints mechanisms, and execution continues. Simultaneously, since Flink supports providing external interfaces by adding REST APIs, the interface parameter of this REST API is set to the stream task's ID. This provides a way for external manual calls, facilitating operations and maintenance.

[0036] (2) Add a subtask allocation strategy to Flink that evaluates subtask computation based on the number of nodes: A streaming task may contain multiple operations, called vertices. Connecting the vertices forms the task's execution pipeline, and each operation can define its own parallelism during execution. Each vertex contains a number of subvertices equal to the parallelism. Subvertices are connected from top to bottom according to connection rules to form a subpipeline of the task. A single subvertice is only included in one subpipeline. Thus, in actual task execution, since the parallelism of each vertex is not completely consistent, different subpipelines will contain different numbers of subvertices. These subpipelines are the subtasks. Meanwhile, Flink defines execution resources through task slots. Each compute node's TaskManager has one or more task slots, and each task slot can run one subtask. Different task slots have independent memory spaces but share the compute node's CPU. The purpose of the subtask allocation strategy is to allocate subtasks to appropriate task slots for execution.

[0037] Modify the Flink kernel code to add a new subtask allocation strategy, as shown in Figure 2 . When allocating subtasks, subtasks of the same node are grouped together, and a property vertexNumber is added to the group, which indicates the number of nodes contained in a single subtask in the group. At the same time, for each computing node, first sort in descending order according to the number of idle task slots, and then sort in ascending order according to the current CPU utilization for computing nodes with the same number of idle task slots. The earlier the computing node is sorted, the more resources it has available, indicating that the potential ability of the task slot on the computing node to execute subtasks is stronger and more efficient. Sort the subtask groups in descending order using the vertexNumber, and traverse the subtask groups and subtasks in them in order, preferentially allocating task slots on computing nodes sorted earlier to execute subtasks. At the same time, after each allocation, the number of idle task slots on the computing node changes, and the computing node to which the subtask is allocated is re-sorted. In this way, the load balancing of the subtask allocation phase is achieved while considering both the subtask computation and the computing node capability.

[0038] (3) Monitor the number of computing nodes in the computing node cluster. When a new computing node is added, for all running stream tasks in Flink, get the task ID in order according to the deployment time, and use the restart function in step (1) to redeploy, and use the method in step (2) to allocate subtasks, as shown in Figure 3 . Since the restart process will re-allocate subtasks, the last allocation plan is ignored, and the idle task slots of the newly added computing node are added to the queue of subtasks to be allocated. In this way, the workloads of all running stream tasks can be distributed to the newly added computing node, improving the overall task execution efficiency and resource utilization.

[0039] (4) Monitor the CPU utilization of each computing node in the computing node cluster. Under normal conditions, when no new stream tasks are deployed, the CPU utilization of each computing node should be basically stable or decrease due to the completion of subtasks. When the CPU utilization of one or more nodes suddenly increases in a short period of time, causing the CPU utilization of the computing node to reach the maximum and not decrease for a period of time, it is due to the sudden increase in the receiving or output traffic of a subtask being executed on the computing node, causing the computation to suddenly increase, as shown in Figure 4As shown. If there are still idle task slots in the computing node cluster at this time, and the CPU utilization of the computing nodes with idle task slots has not reached the maximum, and there is spare computing power, then check the received bytes and sent bytes of the subtasks on the computing nodes with the surge in CPU utilization, find the stream task where the subtask with the largest increase in sent and received bytes during the surge in CPU utilization is located, and select the last successfully deployed stream task on the other stream tasks on the computing node to redeploy using the restart function in step (1), and at the same time use the method in step (2) to allocate subtasks, ignoring the computing nodes with the surge in CPU utilization at this time during allocation.

[0040] (5) If the computing node whose CPU utilization reached its maximum due to the surge in the sending and receiving traffic of the subtasks executed in step (4) is still in the state of maximum CPU utilization after the redeployment of the streaming task in step (4), and there are idle task slots in the computing node cluster other than the computing node, and the CPU utilization of the computing node with idle task slots has not reached its maximum, and there is still spare computing power, then select the last successfully deployed streaming task in the streaming task where the subtasks that did not experience a surge in sending and receiving traffic are located, and redeploy and allocate the subtasks. When allocating, ignore the computing node whose CPU utilization has surged at this time.

[0041] (6) Repeat step (5) until one of the following four situations occurs, then stop the operation.

[0042] 1. The CPU utilization of computing nodes that reached their maximum due to sudden traffic spikes will decrease to below the maximum value and return to normal levels.

[0043] 2. There are no other subtasks of the streaming task on the computing node where the anomaly occurred.

[0044] 3. There are no free task slots in the compute node cluster except for the abnormal node.

[0045] 4. The CPU utilization of other compute nodes with available task slots is also about to reach its maximum.

[0046] This allows for a more balanced workload across the computing node cluster when faced with sudden spikes in computing traffic, without increasing the number of computing nodes, thereby improving overall task execution efficiency.

[0047] The above-described embodiments have described the technical solutions and beneficial effects of the present application in detail, and it should be understood that the above-described is only a specific embodiment of the present application and is not used to limit the present application, and any modification, supplement and equivalent replacement made within the principle range of the present application should be included in the protection range of the present application.

Claims

1. A method for balancing load in real time in a distributed stream computing engine Apache Flink, characterized in that, The method comprises the following steps: (1) adding a new function of restarting task for Flink, and providing an external restart function interface for calling; The new function of restarting task for Flink is specifically as follows: using the task execution plan graph generated in the process of deploying a stream task, ignoring the existing subtask allocation plan, re-deploying the specified stream task according to the selected subtask allocation strategy in the configuration, and then relying on the save point and checkpoint mechanism provided by Flink to restore the original execution state and continue execution; (2) adding a subtask allocation strategy for Flink to evaluate the subtask calculation amount according to the number of nodes; (3) monitoring the CPU utilization of each computing node in the computing node cluster, when the CPU utilization of one or more nodes increases under the premise that the original CPU utilization does not reach the maximum, and the CPU utilization of the computing node reaches the maximum and does not decrease within ten minutes, if there are idle task slots in the computing node cluster at this time, and the CPU utilization of the computing node with idle task slots does not reach the maximum, then the traffic of the subtask on the computing node with the CPU utilization explosion is checked; The stream task where the subtask with the largest traffic increase during the CPU utilization explosion is found, and the last successfully deployed stream task on the computing node is selected to use the restart function in step (1) to re-deploy, and the strategy in step (2) is used for subtask allocation, and the subtask allocation ignores the computing node with the CPU utilization explosion at this time.

2. The method for balancing load in real time in a distributed stream computing engine Apache Flink according to claim 1, characterized in that, In step (1), the calling interface is provided by adding a REST API, and the input parameter of the restart function interface is the ID of the stream task, which is used to specify the task to be restarted.

3. The method for balancing load in real time in a distributed stream computing engine Apache Flink according to claim 2, characterized in that, A restart button is added to the task execution interface in the Flink web source code, and the restart function interface is bound to realize the restart operation of the current task by clicking the button on the web page. 4.The method for balancing load in real time in a distributed stream computing engine Apache Flink according to claim 1, characterized in that, In step (2), the subtask allocation strategy is specifically as follows: When allocating subtasks, subtasks including the same node are grouped into a group, and the group is added with an attribute vertexNumber, which indicates the number of nodes contained in a single subtask in the group; Meanwhile, for each computing node, first, sort the computing nodes in descending order according to the number of idle task slots, and then sort the computing nodes with the same number of idle task slots in ascending order according to the current CPU utilization; sort the subtask groups in descending order using vertexNumber, and traverse the subtask groups and subtasks in the subtask groups in turn, and preferentially allocate the task slots on the computing nodes with high sorting to execute the subtasks.

5. The method for balancing load in real time in a distributed stream computing engine Apache Flink according to claim 1, characterized in that, Step (3) further comprises: Monitoring the number of computing nodes in the computing node cluster, when a new computing node is added, all stream tasks running in Flink are re-deployed using the restart function in step (1) according to the deployment success time in chronological order, and the method in step (2) is used for subtask allocation, that is, the workload of the stream task being executed is automatically allocated to the new node in time when the new computing node is added. 6.The method for balancing load in real time in a distributed stream computing engine Apache Flink according to claim 1, characterized in that, After step (3) is completed, if the computing node whose CPU utilization reaches the maximum due to the burst of the subtask flow before execution is still in the state of the maximum CPU utilization, at the same time, there are idle task slots in the computing node cluster except the computing node, and the CPU utilization of the computing node with idle task slots does not reach the maximum, and there is still surplus computing capacity, then the last successfully deployed stream task in the stream task of the subtask with no flow burst is selected again for redeployment and subtask allocation, and the computing node with CPU utilization burst at this time is ignored during allocation; the above operation is repeated until the following four situations occur, and the operation is stopped; a. The CPU utilization of the computing node whose CPU utilization reaches the maximum due to the burst of the subtask flow returns to the normal level; b. There is no subtask of other stream tasks on the abnormal computing node; c. There is no idle task slot in the computing node cluster except the abnormal node; d. The CPU utilization of other computing nodes with idle task slots also reaches the maximum.

Images