Dynamic data grouping method and system based on time delay perception under Flink framework

Abstract

The invention relates to a dynamic data grouping method and system based on time delay perception under an Flink framework, and the method comprises the steps: carrying out the statistics of key value frequencies in a data stream, and dividing high and low frequency key values according to the key value frequencies; calculating the number of candidate instances allocated by high and low frequency key values, and calculating the completion time delay of each candidate instance to obtain a priority list; according to the time delay priority list, traversing a set of current to-be-distributed data, and selecting data of instances needing to be distributed; traversing the time delay priority list, and preferentially distributing the data to the instance with the minimum completion time delay for execution; updating the time delay information of the selected task instance by using a time delay sensing load model according to the runtime data of the selected task instance in the task execution process; and repeatedly executing the steps S2-S6 to execute the data packet. According to the method, the load balancing degree of the cluster when the data packet of the Flink cluster is executed in the heterogeneous cloud environment is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of big data stream processing, and particularly to a Flink dynamic data grouping method and system based on latency awareness under the Flink framework. Background Art

[0002] As a key strategic asset, big data is becoming a powerful driving force for economic growth and social progress. In recent years, with the commercial application of 5G technology, the rapid development of fields such as artificial intelligence, cloud computing, and the Internet of Things, and the transformation of many traditional industries towards digitization and informatization, the growth rate of data is accelerating exponentially. The development of these technologies has brought continuous data streams, leading to a sharp expansion of the global total data volume. According to the "Data Age 2025 Report" jointly released by Seagate, the world's largest hard disk manufacturer, and the Internet Data Center, the global total data volume is expected to reach 1,630 zettabytes by 2025. In the past three decades, data has played a crucial role in all aspects of human life. It has not only reshaped the way we obtain education and entertainment but also greatly expanded our experience of business and the world. Data has become a core element of our increasingly digital life, which we can define as the "Global Datasphere", that is, the sum of all data created, collected, and replicated globally in a specific year. The development speed of this field is remarkable. In the past decade, our society has achieved a complete transformation from analog to digital. Looking ahead to the next decade, the potential of data is expected to bring unprecedented changes.

[0003] As the amount of data continues to grow, the role played by real-time data analysis will become increasingly important. Globally, driven by the Internet of Things, the proportion of real-time data in the data sphere will increase from 12% in 2015 to 29% in 2025. Applying the above percentages to China, the growth of real-time data will exceed 34 times by 2025, accounting for 28% of the global real-time data. The growth of real-time data will enhance the automation level of edge computers, and at the same time, IT organizations will introduce more interrupt-driven traffic. In interrupt-driven traffic, signal or information input must be processed in real time. With the increasing demand for instant data analysis in various industries, the application scenarios of real-time data processing are extensive, including but not limited to the financial industry, e-commerce, the Internet of Things, social media, etc. In the financial industry, real-time data processing can be used to monitor trading activities, detect fraud, and manage risks. On e-commerce platforms, it can provide personalized recommendations for users by analyzing real-time user behavior data, thereby increasing the conversion rate. A large amount of real-time data generated by Internet of Things devices can help enterprises monitor device status and predict maintenance needs through real-time data processing technology. Social media platforms use real-time data processing to analyze user interactions, content dissemination, and trend monitoring. All these application scenarios rely extremely on real-time data analysis to obtain valuable insights.

[0004] The growth in the demand for real-time data processing has promoted the rapid development of distributed stream computing technologies, and at the same time, some new challenges have emerged. With the increasing growth of massive data, the demand for data processing speed has become increasingly urgent, which has prompted big data processing systems to be able to respond and process data quickly. In recent years, a large number of excellent stream processing platforms have emerged, including Apache Storm, Apache Spark, Twitter Heron, Apache Flink, etc. Apache Flink is an open-source distributed processing engine, renowned for its excellent real-time big data analysis capabilities. It not only supports the concept of unified stream and batch data processing but also can seamlessly interface with the Hadoop ecosystem, quickly rising to become one of the standards in the field of real-time computing. The core advantages of Flink lie in its high-throughput, low-latency real-time computing capabilities, as well as its sub-second response capabilities for massive data, supporting event-driven applications, being able to extract data from multiple event streams, and triggering calculations or state updates based on events. Its precise state management capabilities support event time and exactly-once state consistency guarantees. At the same time, Flink can run on multiple resource management frameworks, such as YARN, Mesos, Kubernetes, and supports independent deployment on bare-metal clusters, with high-availability options, capable of scaling to thousands of cores and state reaching the TB level. In summary, Flink is a real-time stream computing framework with high throughput, low latency, and high performance.

[0005] Since the default key-value grouping in Flink uses a set of specific fields in a tuple as the key-value and assigns tuples with the same key-value to the same instance. Key-value grouping is usually used for stateful operators. However, in the case of severe data skew, the key-value-based assignment will cause the downstream operator instances to bear an excessive load because they may receive a large number of data tuples with high-frequency key-values, thus having a greater impact on system performance. In addition, a Flink cluster is generally composed of multiple different heterogeneous nodes, and different nodes have different resource capacities. The different processing capabilities will result in different data processing rates among different instances, that is, the number of tuples processed per unit time is different, thus affecting the grouping strategy. Therefore, the default key-value grouping method in Flink does not consider the heterogeneity of nodes in the cluster and the data processing capabilities of different nodes, which will also lead to uneven load in the big data cluster and affect the performance of the cluster and other problems. Summary of the Invention

[0006] To solve the problems existing in the background technology, the present invention provides a Flink dynamic data grouping method and system based on delay awareness under the Flink framework, which effectively improves the load balancing degree of the cluster and the performance of the system.

[0007] To achieve the above technical objectives, one aspect of the present invention provides a dynamic data grouping method based on delay awareness under the Flink framework, including:

[0008] S1: Initialize the delay information of task instances in the Flink cluster;

[0009] S2: Generate a set of data to be assigned according to the Flink Job submitted by the client;

[0010] S3: Count the key-value frequencies in the data stream, and divide the high-frequency and low-frequency key-values according to the key-value frequencies;

[0011] S4: Calculate the number of candidate instances for high-frequency and low-frequency key-value assignment, calculate the completion delay of each candidate instance, and obtain a priority list;

[0012] S5: According to the delay priority list, traverse the current set of data to be assigned, and select the data that needs to be assigned to an instance; when traversing the delay priority list, preferentially assign the data to the instance with the least completion delay for execution;

[0013] S6: Update the delay information of the selected task instance using the delay-aware load model according to the runtime data during the task execution of the selected task instance; repeat steps S2 - S6 for data grouping execution;

[0014] Preferably, initializing the latency information of the task instances in the Flink cluster includes: allocating data to the task instances in the Flink cluster and executing the data using a key-value grouping strategy, and initializing the latency information of the task instances after the allocation is completed.

[0015] Preferably, the input rate of the downstream task instance includes:

[0016]

[0017] Preferably, the load of the task instance includes:

[0018]

[0019] in, Indicates the number of tuples that the mth task instance can accept, Indicates the processing rate of the mth task instance.

[0020] Preferably, the average load value of the task instance includes:

[0021]

[0022] Preferably, the load balancing deviation of the task instance includes:

[0023]

[0024] in, Indicates all The average load of the instance.

[0025] Preferably, the task instance To the task instance Delays include:

[0026]

[0027] in, Represents a task instance To the task instance The network delay between LA(v jm ) process Represents a task instance processing delay, including processing time and queue time.

[0028] Preferably, the task instance To task instance v j The average latency includes:

[0029]

[0030] Another aspect of the present invention provides a dynamic data grouping system based on delay awareness under the Flink framework. The system applies the method for dynamic data grouping based on delay awareness under the Flink framework, and includes: a Flink architecture, a resource monitoring module, a data processing module, and a MySQL database;

[0031] The Flink architecture includes: a JobManager node and multiple TaskManager nodes;

[0032] The JobManager node is used to generate a set of data to be allocated according to the Flink Job submitted by the client, and allocate data to task instances according to the delay priority list of all current task instances;

[0033] The TaskManager node is used to execute the received tasks;

[0034] The resource monitoring module is used to obtain the runtime data of task instances during data allocation and execution;

[0035] The data processing module is used to calculate the delay information of task instances using a delay-aware load model according to the runtime data of task instances during data allocation and execution, and store the delay information of task instances in the MySQL database to update the delay priority list of task instances.

[0036] Preferably, allocating data to task instances according to the delay priority list of current task instances includes: counting the key-value frequencies in the data stream, dividing high-frequency and low-frequency key values according to the key-value frequencies; calculating the number of candidate instances for high-frequency and low-frequency key value allocation, calculating the completion delay of each candidate instance, and obtaining a priority list; according to the delay priority list, traversing the set of data to be allocated currently, and selecting the data that needs to be allocated to an instance; traversing the delay priority list, and preferentially allocating the data to the instance with the least completion delay for execution;

[0037] The present invention has at least the following beneficial effects

[0038] By means of a load model based on delay awareness, the present invention proposes a method for calculating the delay of task instances on the TaskManager node. When the Flink job is executed, the most suitable task instance for data allocation is selected according to the delay priority list, and the data is preferentially allocated to the instance with less completion delay, so as to achieve good load balancing among instances as much as possible, maximize the throughput of the Flink cluster and minimize the delay, thereby improving the performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0040] Figure 2 Data allocation diagram for vertex instances of the present invention;

[0041] Figure 3 Schematic diagram of traditional data allocation;

[0042] Figure 4 Schematic diagram of data allocation for the method of the present invention;

[0043] Figure 5 Schematic diagram of the framework of the task scheduling system of the present invention. Detailed implementation manners

[0044] The technical solutions of the present invention will be clearly and comprehensively described below with reference to the accompanying drawings in the embodiments of the present invention. It should be noted that the described embodiments only represent a part of the embodiments of the present invention, not all. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0045] Please refer to Figure 1 , for the problem of load imbalance generated during data allocation for a Flink cluster deployed in a heterogeneous cloud environment, a latency-aware load model is constructed. This model takes into account the different processing capabilities of task instances on different nodes in a heterogeneous cluster, and evaluates the load capacity of task instances through the completion latency of tasks. Based on the above-mentioned load model, a latency-aware dynamic data grouping algorithm is proposed, which effectively improves the load balance degree of the cluster. In order to improve the load balance degree of the cluster, this embodiment proposes a latency-aware Flink dynamic data grouping method under the Flink framework, including:

[0046] S1: Initialize the latency information of task instances in the Flink cluster;

[0047] S2: Generate a set of data to be allocated according to the Flink Job submitted by the client;

[0048] S3: Count the key-value frequencies in the data stream, and divide the high-frequency and low-frequency key values according to the key-value frequencies;

[0049] S4: Calculate the number of candidate instances for high-frequency and low-frequency key-value allocation, calculate the completion latency of each candidate instance, and obtain a priority list;

[0050] S5: According to the latency priority list, traverse the current set of data to be allocated, and select the data that needs to be allocated to an instance; traverse the latency priority list, and preferentially allocate the data to the instance with the least completion latency for execution;

[0051] S6: Update the latency information of the selected task instance by using the latency-aware load model based on the runtime data during the execution of the selected task instance; repeat steps S2 - S6 for the data packet execution;

[0052] Please refer to Figure 2 , Figure 2 a data allocation graph represented as vertex instances, and the generated set of allocated data is represented as: {v i1 -v i4 , v j1 -v j3}. The traditional key-value grouping method takes the set of specific fields in the tuple as the key-value and assigns the tuples with the same key-value to the same instance. Key-value grouping is usually used for stateful operators. However, in the case of severe data skew, the key-value based allocation will cause the downstream operator instances to bear an excessive load because they may receive a large number of data tuples with high-frequency key-values, thus having a greater impact on the system performance. As Figure 3 shown, the figure contains 3 downstream instances. v j1 received 5 tuples, v j2 received 2 tuples, and v j3 received 1 tuple; while after being processed by the method of the present invention, the data will be preferentially allocated to the instance with a smaller completion latency. As Figure 4 shown, the data is evenly allocated to the 3 downstream instances, and each instance is allocated 3 tuples. This allocation method of the method of the present invention can effectively reduce the load imbalance generated during data allocation on the Flink platform.

[0053] Preferably, the input rate of the downstream task instance includes:

[0054]

[0055] Preferably, the load of the task instance includes:

[0056]

[0057] wherein, represents the number of tuples that the m-th task instance can accept, and represents the processing rate of the m-th task instance.

[0058] Preferably, the average load value of the task instance includes:

[0059]

[0060] Preferably, the load balance deviation of the task instance includes:

[0061]

[0062] Among them, represents the average load of all instances.

[0063] Preferably, the task instance to task instance delay includes:

[0064]

[0065] Among them, represents the network delay between task instance to task instance LA(v jm ) process represents the processing delay of task instance , including processing time and queuing time.

[0066] Preferably, the average delay of the task instance to task instance v j includes:

[0067]

[0068] Please refer to Figure 5 , another aspect of the present invention provides a dynamic data grouping system based on delay awareness under the Flink framework. The system is applied to the method for dynamic data grouping based on delay awareness under the Flink framework, including: Flink architecture, resource monitoring module, data processing module, and MySQL database;

[0069] The Flink architecture includes: a JobManager node and multiple TaskManager nodes;

[0070] The JobManager node is used to generate a data set to be allocated according to the Flink Job submitted by the client, and allocate data to the task instances according to the delay priority list of all current task instances;

[0071] The TaskManager node is used to execute the received tasks;

[0072] The resource monitoring module is used to obtain the runtime data of the task instances during data allocation and execution;

[0073] The data processing module is used to calculate the latency information of the task instance by using the latency-aware load model based on the runtime data during the data allocation and execution of the task instance, and store the latency information of the task instance in the MySQL database to update the latency priority list of the task instance.

[0074] In this embodiment, the resource monitoring module mainly collects the runtime data when the task instance executes the task, such as the completion time of the downstream instance, through the Shell script running on the TaskManager node, and monitors the performance metrics of the Flink cluster when executing the task, such as throughput and load, through Prometheus + Grafana.

[0075] Preferably, allocating data to the task instance according to the latency priority list of the current task instance includes: counting the key-value frequencies in the data stream, dividing the high and low frequency key-values according to the key-value frequencies; calculating the number of candidate instances allocated to the high and low frequency key-values, calculating the completion latency of each candidate instance to obtain the priority list; according to the latency priority list, traversing the set of data to be allocated currently, selecting the data that needs to be allocated an instance; traversing the latency priority list, and preferentially allocating the data to the instance with the least completion latency for execution;

[0076] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that without departing from the purpose and scope of the present technical solution, modifications, equivalent replacements, or improvements can be made to it. These modifications or replacements should still fall within the scope of the claims of the present invention.

Claims

1. A dynamic data grouping method based on delay perception under the Flink framework, characterized in that Including: S1: Initialize the latency information of task instances in the Flink cluster; S2: Generate a set of data to be allocated according to the Flink Job submitted by the client; S3: Statistically analyze the key-value frequencies in the data stream, and divide the high-frequency and low-frequency key-values according to the key-value frequencies; S4: Calculate the number of candidate instances allocated for high-frequency and low-frequency key-values, calculate the completion latency of each candidate instance, and obtain a priority list; S5: According to the latency priority list, traverse the set of currently unallocated data, and select the data that needs to be allocated to an instance; when traversing the latency priority list, preferentially allocate the data to the instance with the least completion latency for execution; S6: Update the latency information of the selected task instance using the latency-aware load model based on the runtime data of the selected task instance during task execution; Repeat steps S2 - S6 for data packet execution.

2. The Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 1, wherein The initialization of the latency information of task instances in the Flink cluster includes: using a key-value grouping strategy to allocate data to task instances in the Flink cluster and execute, and initializing the latency information of task instances after the allocation is completed.

3. A Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 2, characterized in that, The input rate of the downstream instance includes:

4. The Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 3, characterized in that The load of the task instance includes: Among them, represents the number of acceptable tuples of the m-th task instance, represents the processing rate of the m-th task instance.

5. A Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 4, characterized in that, The average load value of the task instance includes:

6. The Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 5, characterized in that The load balancing deviation of the task instance includes: Among them, represents the average load of all instances.

7. A Flink dynamic data grouping method based on delay awareness under the Flink framework according to claim 6, characterized in that The task instance to the task instance The latency includes: Among them, represents the network latency between task instance and task instance , LA(v jm ) process represents the processing latency of task instance , including processing time and queuing time.

8. A Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 7, characterized in that The task instance to task instance v j The average latency includes:

9. A Flink dynamic data grouping system based on delay awareness under the Flink framework according to claim 8, wherein the system is applied to a Flink dynamic data grouping method based on delay awareness under the Flink framework according to any one of claims 1-8, and includes: Flink architecture, resource monitoring module, data processing module, and MySQL database; The Flink architecture includes: a JobManager node and multiple TaskManager nodes; The JobManager node is used to generate a set of data to be allocated according to the Flink Job submitted by the client, and allocate data to task instances according to the latency priority list of all current task instances; The TaskManager node is used to execute the received tasks; The resource monitoring module is used to obtain the runtime data of task instances during data allocation and execution; The data processing module is used to calculate the latency information of task instances using the latency-aware load model based on the runtime data of task instances during data allocation and execution, and store the latency information of task instances in the MySQL database to update the latency priority list of task instances.

10. A Flink dynamic data grouping method based on delay perception under the Flink framework according to claim 10, characterized in that, The allocation of data to task instances according to the latency priority list of current task instances includes: statistically analyzing the key-value frequencies in the data stream, and dividing the high-frequency and low-frequency key-values according to the key-value frequencies; calculating the number of candidate instances allocated for high-frequency and low-frequency key-values, calculating the completion latency of each candidate instance, and obtaining a priority list; according to the latency priority list, traverse the set of currently unallocated data, and select the data that needs to be allocated to an instance; when traversing the latency priority list, preferentially allocate the data to the instance with the least completion latency for execution.

Images