A dynamic resource scheduling method for a space-time big data stream processing engine

Abstract

A dynamic resource scheduling method for a space-time big data stream processing engine is proposed in the present application. Current stream processing engines usually use clusters or high-performance hardware to improve the efficiency of processing data, only support processing real-time stream data, cannot process unbounded real-time data and bounded storage data at the same time, and the random allocation of processing tasks cannot improve the data processing rate according to the upper demand. The present application faces space-time big data, improves the data division algorithm, divides the stream data into two types of bounded and unbounded data, and designs a dynamic resource scheduling strategy for data processing tasks from the global and local resource range respectively, so that the algorithm can ensure the real-time processing of unbounded tasks while expanding the total amount of processing bounded tasks, thus enabling the stream processing engine to process unbounded and bounded data simultaneously, and realizing efficient allocation of resources in a cluster environment to improve the throughput of the stream processing engine and reduce the processing delay.

Description

Technical Field

[0001] The present invention relates to the field of data processing engines, and in particular to the design of a dynamic resource scheduling algorithm in a stream data processing engine.

[0002] Dynamic resource scheduling algorithm is an important method and key technology to improve the performance of stream data processing engines. Background Art

[0003] In recent years, stream processing engines (SPEs) have garnered significant attention for their ability to process spatiotemporal big data streams with low latency. Resource scheduling algorithms significantly impact the latency and throughput of stream processing engines. To meet the demands of real-time stream processing, various SPEs have been proposed, such as S4, Spark, Kafka, and Flink. Flink is a typical stream processing engine and a general-purpose data stream processing framework. It addresses the processing of all types of data from a broad data perspective. However, Flink does not construct distinct processing strategies for different data types simultaneously. However, the analysis and mining of spatiotemporal big data requires stream processing engines capable of processing both types of data simultaneously, posing new challenges for current data stream processing engines.

[0004] For example, when performing distributed mining and analysis on various data types in a traffic monitoring system, not only does it require high-speed processing of real-time traffic flow monitoring data, but it also requires processing of stored data such as the number of traffic accidents and congestion. This can affect the analysis of traffic flow, as the stream processing engine will simultaneously receive unbounded real-time data and bounded stored data. To enable stream processing engines to meet complex knowledge management requirements, they must not only be able to divide data into bounded and unbounded categories, but also handle more bounded data tasks while ensuring the real-time performance of unbounded data processing tasks. Summary of the Invention

[0005] The present invention proposes an effective resource scheduling algorithm, which optimizes the stream data processing framework by leveraging the characteristics of data, and provides a faster and more stable data stream processing engine for data mining tasks.

[0006] Resource scheduling for stream processing engines addresses the crucial task of efficient data processing, and is applicable to various big data sectors, including smart healthcare, transportation, and finance. Current approaches to stream processing engine resource scheduling fail to consider the division of data tasks into unbounded and bounded data types. Furthermore, these approaches fail to integrate the needs of higher-level applications from the perspective of these two data types, enabling stream processing engines to meet the real-time requirements of higher-level applications for unbounded data processing while also meeting their capacity requirements for bounded data processing.

[0007] In order to achieve the above object, the technical solution provided by the present invention is:

[0008] In summary, in response to the above-mentioned performance issues of the stream processing engine, the present invention aims to improve the throughput of the stream processing engine and reduce the processing delay of unbounded data. First, the water level algorithm is improved to divide the stream data into unbounded and bounded data types, and then corresponding task construction strategies are designed for different data. Finally, the optimal TaskManager partition is obtained for the data processing tasks from the global and local resource ranges, thereby realizing dynamic resource scheduling, effectively improving the throughput of the stream processing engine, and reducing the delay in processing unbounded data.

[0009] A dynamic resource scheduling method for a spatiotemporal big data stream processing engine, characterized by:

[0010] Step 1: The data acquisition system acquires data; at the same time, the water level algorithm is improved;

[0011] Step 2: The stream processing engine receives data and divides the received data using the new watermark algorithm obtained in step 1.

[0012] Step 3: Construct corresponding processing tasks for the data types divided in step 2 to obtain the total amount of bounded and unbounded task processing;

[0013] Step 4: Combine the bounded and unbounded task processing amounts obtained in step 3 and perform dynamic resource scheduling from the global and local resource scopes respectively.

[0014] Beneficial effects

[0015] The present invention aims to solve the problem that existing stream processing engines cannot process both unbounded and bounded data processing tasks at the same time, and designs a dynamic resource scheduling method for stream processing engines. The method first improves the water level algorithm by introducing a water level delay threshold, and divides the stream data into two types of data: unbounded and bounded, in combination with the characteristics of spatiotemporal data. Afterwards, different ETL task construction strategies are designed for the two types of data, and on this basis, dynamic resource scheduling is performed on the data processing tasks from both global and local resource scopes. This method enables the stream processing engine to not only have the ability to process both unbounded and bounded data processing tasks at the same time, but also effectively reduces the delay in processing unbounded data, while also greatly improving the task volume of the stream processing engine for bounded data processing. This is of great significance for meeting the upper-level business needs of fast access, processing, storage, and mining of big data in the context of smart interconnection and high information volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are part of the specification and are included to provide further understanding of the application and, together with the following detailed description, serve to explain the present disclosure but do not limit the application. In the drawings:

[0017] Figure 1 Flowchart for the data processing of the spatiotemporal data stream engine;

[0018] Figure 2 Example diagram for improving the water level algorithm in step two;

[0019] Figure 3 Example diagram for constructing the unbounded data ETL task in step three;

[0020] Figure 4 Example diagram for constructing the bounded data ETL task in step three;

[0021] Figure 5 Schematic diagram of global resource scheduling in step four;

[0022] Figure 6 Schematic diagram of local resource scheduling in step four. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the specific embodiments of the present application will be described clearly and completely below with reference to the drawings. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0024] The specific implementation process of the present application is shown in Figure 1 .

[0025] A dynamic resource scheduling method for a spatiotemporal big data stream processing engine, characterized in that it comprises

[0026] Step 1: The data acquisition system acquires data; at the same time, the water level algorithm is improved;

[0027] Step 2: The stream processing engine accesses the data, and divides the accessed data using the new water level algorithm obtained in step 1;

[0028] Step 3: For the data types divided in step 2, corresponding processing tasks are constructed, and the total amount of bounded and unbounded task processing is obtained;

[0029] Step 4: In combination with the total amount of bounded and unbounded task processing obtained in step 3, resource dynamic scheduling is performed from the global and local resource ranges respectively.

[0030] Each step is described in detail as follows.

[0031] Step 1: The data acquisition system acquires data; at the same time, the water level algorithm is improved,

[0032] 1.1 Setting up the window mechanism

[0033] The event time is defined as the time when the data actually occurs, that is, the time when the data acquisition system where the event occurs is recorded. The event processing time is the time it takes to process or calculate the data in the actual stream processing engine.

[0034] Assume that each window has a start time startTime and an end time endTime, and the event time of a certain arriving data is eventTime, then the data acquisition system will check eventTime and find the corresponding window (startTime <eventTime<endTime),并从该窗口所有的已到达数据中找到maxEventTime。

[0035] 1.2 Introducing the water level delay threshold to improve the water level algorithm

[0036] Set the water level delay T and water level delay threshold T' two parameters (such as Figure 2 As shown), and based on the window mechanism in step 1.1, the water level algorithm is set. The judgment criteria of the water level algorithm are as follows:

[0037]

[0038] Step 2: The stream processing engine receives data and divides the received data using the new watermark algorithm obtained in step 1.

[0039] Combined with the improved watermark algorithm obtained in step 1.2, the data received by the stream processing engine is divided into bounded and unbounded data types. Specifically, when the event time is less than the difference between the maximum event time and the watermark delay, and the watermark delay is less than the watermark delay threshold, the data corresponding to the event time in this window is classified as unbounded data. When the event time is less than the difference between the maximum event time and the watermark delay, and the watermark delay is not less than the watermark delay threshold, the data corresponding to the event time in this window is classified as bounded data. Conversely, when the event time is not less than the difference between the maximum event time and the watermark delay, and the watermark delay is less than the watermark delay threshold, the data corresponding to the event time in this window is classified as unbounded data. When the event time is not less than the difference between the maximum event time and the watermark delay, and the watermark delay is not less than the watermark delay threshold, the data corresponding to the event time in this window is classified as bounded data. The data classification results are input into steps 3.2 and 3.3.

[0040] like Figure 2 As shown, the data corresponding to the water level delay T1 is bounded data, while the data corresponding to the water level delay T2 is unbounded data.

[0041] Step 3: For the data types divided in step 2, build corresponding processing tasks to obtain the total amount of bounded and unbounded task processing;

[0042] 3.1 Design corresponding ETL task construction strategies for bounded and unbounded data

[0043] Data extraction, transformation and loading (ETL) is a key step for a data system to obtain high-quality data, and is a process of extracting, transforming, cleaning and loading flow data, so that the data becomes useful data required by upper-layer applications or analysis systems.

[0044] Unbounded data usually takes a short window as a trigger for data processing tasks, and the total amount of data contained in the same window is usually small, so that most of the extraction tasks of unbounded data create new data items, while reducing the maintenance cost of incremental extraction time and timestamp field, thereby improving the real-time performance of unbounded data tasks as much as possible. In addition, unbounded data processing tasks can often filter data conversion tasks according to the needs of upper-layer applications. Therefore, for unbounded data processing tasks, the present application designs the following ETL task construction strategies:

[0045] (1) According to the task triggering mode of window and water line;

[0046] (2) Full-amount extraction data extraction mode;

[0047] (3) Data conversion mode for filtering and converting data according to demand conditions;

[0048] (4) Horizontally divided task division mode.

[0049] Bounded data is often more extensive and complete. In order to reduce the operation cost of the data system, it is necessary to minimize the total amount of data processing, while the bounded data processing task needs to be processed completely, including strict data format conversion and mathematical calculation of various data analysis items. Therefore, the present application maintains bounded data by incremental updating, and designs the following ETL task construction strategies for bounded data processing tasks:

[0050] (1) According to the event-driven task triggering mode;

[0051] (2) Incremental extraction data extraction mode;

[0052] (3) Data conversion mode for complete conversion of all bounded data;

[0053] (4) Vertical division division mode.

[0054] 3.2 Implementing the Construction of Unbounded Data ETL

[0055] like Figure 3 As shown, for the unbounded data obtained in step 2, combined with the construction strategy in step 3.1, an unbounded data processing task is constructed:

[0056] 3.2.1 Horizontally slice the data using fixed-length windows. Figure 3 The medium window length is 5;

[0057] 3.2.2 Within a single sharded window ( Figure 3 , which contains three windows, extracts all data, filters and transforms the data according to upper-level requirements, aggregates and calculates the data in the window, and loads it into the data warehouse;

[0058] 3.2.3 The running results within different single computing windows are independent of each other. The calculation of the window with delayed data will not affect the calculation of the normal window. Therefore, multiple operators (task slots) are used to calculate multiple windows in parallel to improve the efficiency and real-time performance of unbounded data task processing.

[0059] 3.3 Implementing Bounded Data ETL

[0060] like Figure 4 As shown, for the bounded data obtained in step 2, combined with the construction strategy in step 3.1, a bounded data processing task is constructed:

[0061] 3.3.1 The task of processing bounded data is to process the entire bounded data. Therefore, the data stream is first divided vertically into slices of unequal lengths according to the fixed-length data items. The reason for the unequal-length slices here is that the data items contained in the data stream are random. Some slices will be divided into more, and some slices will be divided into fewer. Figure 4 If the data item length is set to 30, the data stream of the bounded data is divided into 3 pieces, containing 8, 5, and 2 data items respectively;

[0062] 3.3.2 Using different operators, simultaneously complete the data incremental extraction task for all shards obtained in step 3.3.1, that is, extract new data based on the extracted data;

[0063] 3.3.3 Use different operators to complete the data conversion task for the data extracted in step 3.3.2;

[0064] 3.3.4 For the data transformed in step 3.3.3, perform segmented calculations and then aggregate the data. The aggregated data is then loaded into the data warehouse. This vertical segmentation approach separates tasks such as data extraction, data transformation, and data loading, thereby reducing blocking between subtasks and expanding the total amount of data processing within bounded data processing tasks.

[0065] Step 4: Combine the bounded and unbounded task processing amounts obtained in step 3 and dynamically schedule resources from the global and local resource scopes respectively;

[0066] 4.1 Complete resource scheduling from the global resource scope, such as Figure 5 shown

[0067] The goal of the global resource adjustment task is to obtain an efficient TaskManager (Worker) that can handle more unbounded and bounded data tasks while ensuring processing latency and load.

[0068] 4.1.1 Based on the total amount of unbounded and bounded data tasks loaded into the data warehouse in steps 3.2 and 3.3, define the overall load of a TaskManager, that is, the total amount of data tasks that the current TaskManager can execute at most. The specific calculation of the TaskManager's overall load is:

[0069] Worker load =Cost Unbounded ×k1+Cost Bounded ×k2

[0070] Where k1≤Window num , k2≤Task num , Cost Unbounded is the processing cost of the unbounded data processing task within a single window, Window num The total number of windows is, Cost Bounded is the processing cost of a bounded data processing task, Task num Is the total number of tasks. When the timer of the data system exceeds the preset period T, the TaskManager overall load Worker load The calculation needs to be retriggered.

[0071] 4.1.2 Based on Worker load , through the heuristic algorithm, we gradually converge to a partition with the smallest load, and thus get an optimal partition, that is, the Worker load The values ​​corresponding to k1 and k2.

[0072] First, when the data stream engine starts for the first time, the algorithm will set the initial values of k1 and k2 to obtain an initial partition, i.e., k1 = Window num , k2 = 0. Second, the initial partition is adjusted by means of the Share Slot mechanism of the Flink engine to quickly narrow the values of k1 and k2 to a reasonable interval. Then, k1 and k2 are adjusted according to the real-time requirements of the current unbounded data task, and finally, the best partition that guarantees the real-time performance of the task and has relatively small load is obtained, and the stable initial values of k1 and k2 are also obtained.

[0073] 4.1.3 According to step 4.1.2, the best partition is recursively divided, and in the subsequent execution process of the data stream engine, when the overall load of the TaskManager needs to be recalculated after a period T, the stable initial values of k1 and k2 can be fine-tuned.

[0074] 4.2 Resource scheduling is completed within the local resource range, as shown in Figure 6

[0075] The goal of local resource adjustment is to use the limited Task Slot operators in the TaskManager to ensure the real-time performance of the unbounded data processing task as much as possible.

[0076] 4.2.1 According to the total amount of unbounded and bounded data tasks loaded into the data warehouse in steps 3.2 and 3.3, a trigger condition for resource scheduling is defined. The processing delay t is the most intuitive reflection of the real-time performance of the unbounded data processing task, and the algorithm will monitor the processing delay after each unbounded data processing task window is completed. When the processing delay t exceeds the delay threshold in multiple windows, it is considered that resource scheduling is needed. The delay threshold can be adjusted according to the requirements of the upper layer application;

[0077] 4.2.2 When the trigger condition in step 4.2.1 is met, first, resource adjustment is performed between TaskManagers. Although the best partition is divided in step 4.1.2, the algorithm tries to give each TaskManager the same best partition in the recursive task division process to ensure the maximum system efficiency. However, in actual data task total amount, there is a heterogeneous computing capability between different TaskManagers, so there will be TaskManagers with larger load and TaskManagers with smaller load. Under the above conditions, the algorithm assigns more tasks to TaskManagers with smaller load based on the Akka communication and "heartbeat" RPC interaction mechanism of the Flink native engine, rather than strictly following the best partition mechanism;

[0078] ​4.2.3 If all TaskManagers in step 4.2.2 are fully loaded, the algorithm can only delay the processing of bounded data tasks, that is, reduce the value of k2. Although this will cause a large number of bounded data processing tasks to accumulate in the message queue, it will allocate more Task Slots to process unbounded data processing tasks, which at least ensures the real-time performance of unbounded data processing tasks.

[0079] Innovation

[0080] This paper proposes a dynamic resource scheduling method for stream processing engines for spatiotemporal big data. Unlike existing methods for improving stream processing engine performance, this method divides stream data into two categories: unbounded and bounded. It then designs different task construction strategies for these two data distributions, enabling the stream processing engine to process both unbounded and bounded data simultaneously. Furthermore, this method performs resource scheduling from both a global and local resource perspective, enabling the stream processing engine to improve the throughput of bounded data processing tasks while ensuring the real-time performance of unbounded data processing tasks.

[0081] The resource scheduling method proposed in the present invention has good performance in improving the performance of the stream processing engine, reducing processing delay and improving throughput.

Claims

1. A dynamic resource scheduling method for a spatiotemporal big data stream processing engine, characterized by: Step 1: The data acquisition system acquires data; at the same time, the water level algorithm is improved; Step 2: The stream processing engine receives data and divides the received data using the new watermark algorithm obtained in step 1. Step 3: Construct corresponding processing tasks for the data types divided in step 2 to obtain the total amount of bounded and unbounded task processing; Step 4: Combine the bounded and unbounded task processing amounts obtained in step 3 and dynamically schedule resources from the global and local resource scopes respectively; The step 1 is specifically as follows: 1.1 Setting up the window mechanism The event occurrence time is defined as the actual time when the data occurs, that is, the time recorded by the data acquisition system where the event occurs; The event processing time is the time it takes to process or calculate the data in the actual stream processing engine; Assume that each window has a start time and end time , and the event time of a certain arriving data is , at this time the data acquisition system will check And find the corresponding window ( ), and find out from all the arrived data in this window ; 1.2 Introducing the water level delay threshold to improve the water level algorithm Set water level delay , and water level delay threshold Two parameters are used, and the water level algorithm is set based on the window mechanism in step 1.

1. The judgment criteria of the water level algorithm are as follows: The step 2 is specifically as follows: Combined with the improved waterlevel algorithm obtained in step 1.2, the data accessed by the stream processing engine is divided into bounded and unbounded data types. That is, when the event time is less than the difference between the maximum event time and the waterlevel delay, and the waterlevel delay is lower than the waterlevel delay threshold, the data corresponding to the event time in this window is divided into unbounded data; when the event time is less than the difference between the maximum event time and the waterlevel delay, and the waterlevel delay is not lower than the waterlevel delay threshold, the data corresponding to the event time in this window is divided into bounded data; conversely, when the event time is not less than the difference between the maximum event time and the waterlevel delay, and the waterlevel delay is lower than the waterlevel delay threshold, the data corresponding to the event time in this window is divided into unbounded data; when the event time is not less than the difference between the maximum event time and the waterlevel delay, and the waterlevel delay is not lower than the waterlevel delay threshold, the data corresponding to the event time in this window is divided into bounded data; the data division results are input into steps 3.2 and 3.3; Water level delay The corresponding data is bounded data, and the water level delay The corresponding data is unbounded data.

2. The method according to claim 1, wherein: The step 3 is specifically as follows: 3.1 Design corresponding ETL task construction strategies for bounded and unbounded data Data extraction, conversion, and loading are the processes of extracting, converting, cleaning, and loading streaming data; 3.2 Implementing the Construction of Unbounded Data ETL For the unbounded data obtained in step 2, combined with the construction strategy in step 3.1, construct an unbounded data processing task: 3.3 Implementing Bounded Data ETL For the bounded data obtained in step 2, combined with the construction strategy in step 3.1, a bounded data processing task is constructed.

3. The method according to claim 2, wherein: 3.1 Design corresponding ETL task construction strategies for bounded and unbounded data Data extraction, conversion, and loading are the processes of extracting, converting, cleaning, and loading streaming data; For unbounded data processing tasks, the following ETL task construction strategy is designed, including: (1) Task triggering method based on window and waterline; (2) Data extraction method of full extraction; (3) Data conversion method for converting filtered data according to required conditions; (4) Horizontal division of tasks; The following ETL task construction strategy is designed for bounded data processing tasks, including: (1) Task triggering method driven by events; (2) Incremental data extraction method; (3) Data conversion method that completely converts all bounded data; (4) Vertical division method; 4. The method according to claim 2, wherein: 3.2 Implementing the Construction of Unbounded Data ETL For the unbounded data obtained in step 2, combined with the construction strategy in step 3.1, construct an unbounded data processing task: 3.2.1 Horizontally slice the data using fixed-length windows, with a window length of 5; 3.2.2 Within a single sharded window, extract all data, filter and transform the data based on upper-layer requirements, aggregate and calculate the data within the window, and load it into the data warehouse. 3.2.3 The running results within different single computing windows are independent of each other. The calculation of windows with delayed data will not affect the calculation of normal windows. Therefore, multiple operator task slots are used to calculate multiple windows in parallel to improve the efficiency and real-time performance of unbounded data task processing.

5. The method according to claim 2, wherein: 3.3 Implementing Bounded Data ETL For the bounded data obtained in step 2, combined with the construction strategy in step 3.1, a bounded data processing task is constructed: 3.3.1 The bounded data processing task is to process the entire bounded data. Therefore, the data stream is first divided vertically into unequal-length slices based on fixed-length data items. The reason for the unequal-length slices is that the data items contained in the data stream are random, and some slices will have more and some will have fewer. If the data item length is set to 30, the bounded data stream is divided into three slices, containing 8, 5, and 2 data items respectively. 3.3.2 Using different operators, simultaneously complete the data incremental extraction task for all shards obtained in step 3.3.1, that is, extract new data based on the extracted data; 3.3.3 Use different operators to complete the data conversion task for the data extracted in step 3.3.2; 3.3.4 For the data converted in step 3.3.3, perform segmentation calculations separately and then aggregate the data. Then, load the aggregated data into the data warehouse. This vertical segmentation method can separate the data extraction, data conversion, and data loading tasks, thereby reducing blocking between subtasks and expanding the total data processing capacity of bounded data processing tasks.

6. The method according to claim 1, wherein: The step 4 is specifically as follows: 4.1 Complete resource scheduling from the global resource scope; The goal of the global resource adjustment task is to obtain an efficient TaskManager (i.e., Worker) that can handle more unbounded and bounded data tasks while ensuring processing latency and load. 4.2 Complete resource scheduling within the local resource range; The goal of the local resource adjustment task is to use the limited Task Slot operators in the TaskManager to ensure the real-time performance of the unbounded data processing task as much as possible.

7. The method according to claim 6, wherein: 4.1 Completing resource scheduling from the global resource scope is as follows: 4.1.1 Based on the total number of unbounded and bounded data tasks loaded into the data warehouse in steps 3.2 and 3.3, define the overall load of a TaskManager, that is, the maximum total number of data tasks that the current TaskManager can execute; the specific calculation of the TaskManager's overall load is: in, , , is the processing cost of the unbounded data processing task within a single window, The total number of windows is, is the processing cost of the bounded data processing task, is the total number of tasks; when the timer of the data system exceeds the preset period T, the overall load of TaskManager The calculation needs to be retriggered; 4.1.2 Based on , through the heuristic algorithm, we gradually converge to get a partition with the smallest load, and thus get an optimal partition, that is, middle and The corresponding value; First, when the data flow engine is started for the first time, the algorithm will be set by default and The initial value gets an initial partition, namely: , Secondly, the initial partition is adjusted with the help of the native Share Slot mechanism of the Flink engine to quickly and The value converges to a reasonable range; then, adjust according to the real-time requirements of the current unbounded data task and , and finally get an optimal partition that guarantees the real-time performance of the task and a relatively small load, and also get and The stable initial value of 4.1.3 Recursively divide the optimal partitions according to step 4.1.

2. In the subsequent execution of the data flow engine, when the overall load of TaskManager needs to be recalculated after a period T, the optimal partitions are calculated based on this stable initial value. and Make fine adjustments.

8. The method according to claim 6, wherein: 4.2 Completing resource scheduling within the local resource range is as follows: 4.2.1 Define a trigger condition for resource scheduling based on the total amount of unbounded and bounded data tasks loaded into the data warehouse in steps 3.2 and 3.

3. The processing delay t is the most intuitive indicator of the real-time nature of unbounded data processing tasks. The algorithm monitors the processing delay after each unbounded data processing task window is completed. When the processing delay t exceeds the delay threshold in multiple windows, resource scheduling is considered necessary. The delay threshold here can be adjusted according to the needs of the upper-level application. 4.2.2 When the trigger conditions in step 4.2.1 are met, resources are first adjusted among TaskManagers. Although the optimal partitions are assigned in step 4.1.2, the algorithm tries to assign the same optimal partitions to each TaskManager as evenly as possible during the recursive task partitioning process to ensure maximum system efficiency. However, in reality, the total number of data tasks is not evenly distributed, and different TaskManagers have heterogeneous computing capabilities. Therefore, there will inevitably be TaskManagers with heavy loads and TaskManagers with light loads. Under these conditions, the algorithm uses the "heartbeat" RPC interaction mechanism between Akka communication and the Flink native engine to assign more tasks to TaskManagers with light loads, rather than strictly following the optimal partitioning mechanism. 4.2.3 If all TaskManagers in step 4.2.2 are fully loaded, the algorithm can only delay the processing of bounded data tasks, that is, reduce Although this will cause a large number of bounded data processing tasks to accumulate in the message queue, it will allocate more Task Slots to handle unbounded data processing tasks, which at least ensures the real-time performance of unbounded data processing tasks.

Images