An out-of-order data stream aggregation rate limiting method based on Apache Flink

Abstract

The application discloses a method for out-of-order data stream aggregation rate limiting based on Apache Flink. The stream processing system additionally comprises a stream aggregation rate limiter and a water line. The application is applied to the aggregation of a hierarchical structure (a layered DAG graph), and is used for the stateful distributed stream aggregation problem, so that the aggregation performance is optimized in the mode of transmitting the node state upwards. Apache Kafka is used for communication between the producer and the consumer. When the aggregation is updated, whether the intermediate aggregation result is uploaded or the blocking state is uploaded is judged according to the aggregation update rate, the maintenance cost of the meaningless intermediate aggregation result in the update is reduced, then the water line is flexibly marked to form a deep water area according to the blocking state, the aggregation update is limited in the deep water area, so that the aggregation update frequency is reduced, and the overall overhead of the aggregation algorithm is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data stream processing, and relates to a method for limiting aggregation overhead by passing node state in stream processing of out-of-order stream. BACKGROUND

[0002] Existing out-of-order stream processing usually works in a water line manner, that is, a timestamp is used to divide the area for different processing, only before the timestamp. For example, a paper entitled "CPiX: Real-Time Analytics Over Out-of-Order Data Streams by Incremental Sliding-Window Aggregation" published in TKDE in 2021 discloses an out-of-order stream processing method for algebraic aggregation problems. The paper believes that the reason for the low performance of the mainstream stream aggregation algorithm in the current big data field is that the stream processing system wastes too much time to maintain the intermediate aggregation result in the case of serious out-of-order. CPiX shortens the aggregation upload path, speeds up the aggregation and reduces the cost of maintaining the intermediate result in the case of serious out-of-order of data stream. The way to shorten the aggregation upload path is to place the aggregation nodes in the same pool, use the commutative property of some aggregation types, abandon the maintenance of all recent intermediate aggregation nodes, and instead maintain only the oldest node. Because the delay is limited, records exceeding the maximum time to live (TTL) should be discarded, and all records can be considered as the earlier the more likely to have no unarrived records, so that the out-of-order situation will not occur. This way of aggregating out-of-order data streams has defects. Since it shortens the aggregation path, its processing capacity becomes single, and it has no ability to deal with commutative order aggregation, so there is a problem of sacrificing processing range for processing performance improvement. In processing some complex overall aggregation, especially in processing aggregation problems with commutative property but without reversibility, because the aggregation path is not preserved, all the aggregation must be completely reprocessed.

[0003] Data flow is a concept in computer science that describes the path of data movement in a system. It shows the transmission process of data from the generation point to the consumption point, as well as the storage, conversion, query and other operations that data may undergo in this process. Data flow is unbounded and has infinite nature. Stream processing system refers to a system that can process data flow in real time and continuously. They are usually used in application scenarios that require fast response, such as real-time analysis, monitoring and decision support systems.

[0004] In stream processing algorithms, partitioning is often designed into the algorithm. Partitioning in stream processing is a key step that affects the performance, scalability, and fault tolerance of the system. The number of partitions should match the parallelism of the processing resources to fully utilize the computing resources. Related data should be kept in the same partition to reduce cross-partition communication. The partitioning strategy should support fault tolerance, such as through data backup or multi-copy mechanisms. The partitioning strategy should support dynamic expansion to adapt to changes in data volume. The partitioning algorithm used in CPiX is Cutty, which is a method of partitioning based on sliding division. In stream processing systems, user programs that generate data are usually called producers, and user programs that obtain data provided by the system are usually called consumers.

[0005] WaterMark mechanism is an important concept in stream processing frameworks for handling event time, especially in scenarios where out-of-order events are processed or calculations based on time windows are needed. It is a logical timestamp that marks the point in time before which all data has arrived at the stream processing system. It can be considered that data before this point in time will not appear again. In other words, WaterMark is a delay boundary that allows the system to perform calculations after observing all related events. WaterMark can be divided into periodic Watermark and punctuated Watermark according to the type of operation. Periodic Watermark is periodically sent, for example, after receiving a certain number of records or after a certain time interval; while punctuated Watermark is sent when a specific event occurs, for example, after processing events with specific attributes.

[0006] Apache Flink is an open-source stream processing framework for stateful computation with high throughput and low latency. It can handle bounded and unbounded data streams and is easy to develop and deploy large-scale data processing applications. Apache Flink provides multiple APIs for writing data processing applications, including: DataSet API (for batch processing), DataStream API (for stream processing), Table API (for structured data processing, similar to SQL), and SQL (directly using SQL for querying).

[0007] Apache Kafka is a platform for building real-time data pipelines and stream processing applications. Kafka can be considered as a super message queue with high performance, high scalability, high availability, and support for persistence.

[0008] Stateful distributed streaming processing is a common challenge faced by streaming platforms such as Apache Flink. "Stateful" means that the streaming process maintains a cumulative state, that is, all events received in the past history; simple tasks like Sum counting and complex tasks like machine learning involve accumulating state. "Distributed" means that the servers that generate and process information are geographically dispersed and independent computers; these hosts are physically separate but are treated as a single host.

[0009] The commutativity of aggregation refers to the fact that if the order of the elements in an aggregation is changed, the result of the aggregation cannot be guaranteed to be the same as the original result. For aggregation problems that do not possess commutativity, it is usually necessary to maintain many intermediate results. Therefore, the nodes in such stream aggregations can be divided into three types: bottom-level exchange nodes, intermediate nodes, and result output nodes. Bottom-level exchange nodes are the lowest-level nodes, which can directly interact with the input data stream and directly obtain updates to the aggregation information from the input data stream. Intermediate nodes are nodes that summarize some of the information within the bottom-level exchange nodes. Intermediate nodes can be invisible to the outside world and have no meaning to the outside world. Each intermediate node is a summary of information from several bottom-level exchange nodes. Intermediate nodes often have more than one layer (bottom-level nodes rarely have only one layer). The result node is the output of the entire aggregation. Consumers directly obtain the result from the result output node. Usually, there is only one such result node. This invention will use a hierarchical structure to refer to the data organization structure in this stream processing algorithm that divides the nodes into three categories. It is essentially a directed hierarchical graph (DAG), but this paper emphasizes its layered characteristics, so it uses hierarchical structure to refer to it. This hierarchical structure is necessary when aggregation operations are not commutative, and the research of this invention is based on this. Summary of the Invention

[0010] To overcome the excessive overhead of existing stream aggregation techniques when handling problems lacking commutativity, this invention provides a rate limiting method for out-of-order data stream aggregation based on Apache Flink. The proposed stream processing system additionally includes a stream aggregation rate limiter and a depth line to limit the excessive overhead of operations that transmit node states. This invention is applied to aggregations with a hierarchical structure (a layered DAG graph), addressing stateful distributed stream aggregation problems. This invention uses Apache Kafka for communication between producers and consumers.

[0011] The stream aggregation rate limiter requires the stream processing system to store recent records of stream aggregation updates in the corresponding nodes. During aggregation updates, the system determines whether to upload intermediate aggregation results or upload in a blocked state (the blocked state is reset periodically) based on the aggregation update rate. This reduces the maintenance cost of updating meaningless intermediate aggregation results. Then, based on the blocked state, it flexibly marks the "deep water zone" to limit aggregation updates, thereby reducing the aggregation update frequency and the overall overhead of the aggregation algorithm. The data structure is hierarchical, with different levels of data covering different amounts of information; in a parent-child relationship, the parent node always covers a larger amount of data than its child nodes.

[0012] In this invention, the stream processing system operates in a distributed environment, providing output and input ports to clients providing data sources. The aggregation format at the sender's output port is consistent with the receiver's format, while the underlying nodes at the receiver's input port are set as time-oriented aggregation child nodes. Aggregation updates begin from the underlying nodes, which are child nodes, and propagate upwards to more abstract levels by summarizing and refining the hierarchical relationships of information through the aggregation structure. The underlying nodes on the receiver that directly interact with the input data stream, the intermediate nodes that aggregate lower-level information, the node storage pool, and the user program that receives the data together constitute a complete structure for aggregating and receiving data.

[0013] The depth line is an extension of the traditional waterline. While traditional waterlines have various types, there is only one, and its influence extends only to the timestamp after the waterline's location. The depth line defines a deep-water zone, which, like a congestion state, restricts the upload of aggregated updates. This deep-water zone is also reset when the congestion state is reset. During data transmission, the complexity of the network environment makes data arrival delays completely unpredictable. CPiX considers earlier records to be safer, as they will require less handling of out-of-order issues in the future. Logically, this is not problematic. However, the issue is that more in the future does not necessarily mean more now. Therefore, using a depth line allows for flexible limitation of the scope of aggregated update uploads.

[0014] The technical solution adopted by this invention to solve its technical problem is:

[0015] A method for limiting the aggregation rate of out-of-order data streams based on Apache Flink, which uses the state of the passing node to limit invalid aggregation overhead, includes the following steps:

[0016] Step (1): The user sets the data structure of stateful computation in the current stream processing system according to the requirements of query aggregation results (only the stateful computation of Apache Flink is changed).

[0017] Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides clients with standardized output and input ports. Data provided by the producer is handed over to Kafka and sent to the stream processing system on the consumer side. The underlying nodes on the input ports are set as time-oriented aggregation child nodes.

[0018] Step (3): Read the input tuple from Kafka using the stream processing system, find the node to which the timestamp of the input tuple belongs, and update that node. Note that when the processing of each tuple is finished, the processing of the next tuple will start again from this step.

[0019] Step (4): Calculate the aggregation update frequency of the corresponding node, and determine whether the aggregation update frequency exceeds the user-given rate limit. If the current node exceeds the rate limit, perform the operation of propagating the blocking state upwards, which will update the water level; otherwise, proceed to the next step. Note that the blocking state will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval between the user's access to the result node (if the user periodically accesses the result node); otherwise, the user determines the length of the reset cycle. In addition, the blocking state has the characteristic of propagating upwards, which means that all ancestor nodes will be blocked together. That is, when any child node of a node is blocked, the node itself will also be blocked.

[0020] Step (5): When the aggregation update frequency of the current node does not exceed the rate limit, determine whether the node to which the input tuple belongs is in a blocked state or in a deep water zone. If the current node is in a deep water zone, end the processing of the current tuple. If the current node is not in a blocked state and is not in a deep water zone, pass the update upward. If the upload encounters a node in a blocked state, end the processing of the current tuple in advance. Otherwise, keep uploading the update to the result node and then end the processing of the current tuple. If the current node is in a blocked state, proceed to the next step.

[0021] Step (6): When the current node is in a blocked state, determine the source of the blockage. If there are no other sources of blockage, restore the current underlying node to a free state and perform the operation of propagating the free state upwards. It should be noted that because the blocked state is determined from bottom to top, the propagation of the state can only be attempted. When encountering a blockage caused by other child nodes, the upload stops.

[0022] Furthermore, the specific operation of step (1) is as follows:

[0023] Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure (tree or network, such as binary tree, B-tree, neural network, etc.) according to the type of aggregation operation.

[0024] The data structure includes bottom-level nodes, intermediate nodes, and result nodes. Information within each node includes: an update count field and the stream aggregation data structure; the number of records that need to be updated in the bottom-level node and the time period for those updates. Bottom-level nodes are segmented using a Cutty method, and to ensure availability to upper-level nodes, they are decomposed into the largest shared child nodes based on window sliding. Bottom-level nodes have two states: propagation state and blocking state. Intermediate and result nodes are also affected by these two node states; their node state is a summary of the bottom-level nodes they encompass.

[0025] Furthermore, the specific steps for the upward propagation of the blocking state in step (4) are as follows:

[0026] Step 4-1: Determine the current node's state. If the current node is in a blocked state, the processing of the current tuple ends; if the current node is not in a blocked state, then the current node is blocked.

[0027] Step 4-2: After blocking the current node, check if the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the depth line operation and end the processing of the current tuple.

[0028] Step 4-3: Determine the state of the currently pointed-to node (i.e., the parent node of the original node). If the currently pointed-to node is not blocked, then block the current node and execute Step 4-2; if the currently pointed-to node is blocked, then execute the update depth line operation and then end the processing of the current tuple.

[0029] Furthermore, the specific steps for step (6) of transmitting the unobstructed status upwards are as follows:

[0030] Step 6-1: Determine if the current node has a parent node. If not, redefine the water depth line and end the processing of the current tuple. If a parent node exists, proceed to step 6-2.

[0031] Step 6-2: Point the pointer to the parent node of the current node, and then check if the currently pointed-to node (which is already the parent node of the original node) is set to a blocked state. If the node is in a blocked state, visit the other child nodes of the node to see if there are other child nodes that also cause the node to be blocked. If there are other child nodes that cause the parent node to be blocked, then redefine the depth line and the processing of the tuple ends. If there are no other child nodes that cause the parent node to be blocked, then execute step 6-3. If the node is not in a blocked state, directly update the result of the currently pointed-to node and pass the aggregation update upwards, unless a blockage is encountered (in which case the blockage has other sources).

[0032] Step 6-3: Set the node state of the currently pointed-to node to smooth and update the aggregation result of the currently pointed-to node, then execute step 6-1.

[0033] Furthermore, the update of the water depth line occurs when there is a blockage during upload and the parent node is also blocked. The update steps are as follows:

[0034] Step 1: During the upload process in a blocked state, when a parent node is already in a blocked state, the position of that parent node is the intersection of the blocked states. If the intersection of the blocked states is a result node, no operation is performed; otherwise, proceed to Step 2.

[0035] Step 2: Based on the blocking level of the intersecting nodes in the blocking state, Blockage = aggregation update frequency of the intersecting nodes in the blocking state / tree height of the intersecting nodes in the blocking state. If the Blockage is lower than the user-preset Blockage threshold, no operation is performed; otherwise, proceed to Step 3.

[0036] Step 3: Starting from the intersection node, proceed downwards according to the parent-child relationship to find the leftmost bottom-level node in a blocked state (leftmost means the node represents the earliest system time) and the rightmost bottom-level node in a blocked state (rightmost means the node represents the latest system time). Mark a depth line on each of these two nodes. Define the time stamp between the two depth lines as the deep water zone. Nodes within this time range are in the deep water zone. Note: The existence of the deep water zone is determined by the intersection nodes in the blocked state that define the deep water zone. If the blocking no longer intersects, then the resulting depth lines and deep water zones will be discarded.

[0037] The beneficial effects of this invention are as follows:

[0038] This invention limits the system's overhead by restricting updates to some intermediate aggregation results through the transmission of node states. This effectively eliminates the problems of high processing overhead and low update utilization in out-of-order stream aggregation methods when dealing with highly disordered data. By setting the initial constraint conditions at the bottom-level nodes, the node blocking state can be easily set according to the frequency of recent aggregation updates at the node, and the upward transmission of states can replace the transmission of aggregation updates when the invalid overhead is too large. Attached Figure Description

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] Figure 1 This is a flowchart of the stream processing system according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of the present invention on a binary tree. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, a method for limiting the aggregation rate of out-of-order data streams based on Apache Flink is described below:

[0044] Step (1): The user sets the data structure for stateful computation in the current stream processing system according to the requirements of query aggregation results (only the stateful computation of Apache Flink is modified). The specific operation is as follows:

[0045] Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure (tree or network, such as binary tree, B-tree, neural network, etc.) according to the type of aggregation operation. Figure 2 This is a schematic diagram of the structure of the present invention on a binary tree.

[0046] The data structure includes bottom-level nodes, intermediate nodes, and result nodes. Information within each node includes: an update count field and the stream aggregation data structure; the number of records that need to be updated in the bottom-level node and the time period for those updates. Bottom-level nodes are segmented using a Cutty method, and to ensure availability to upper-level nodes, they are decomposed into the largest shared child nodes based on window sliding. Bottom-level nodes have two states: propagation state and blocking state. Intermediate and result nodes are also affected by these two node states; their node state is a summary of the bottom-level nodes they encompass.

[0047] Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides clients with standardized output and input ports. Data provided by the producer is handed over to Kafka and sent to the stream processing system on the consumer side. The underlying nodes on the input ports are set as time-oriented aggregation child nodes.

[0048] The proposed stream processing system further includes a stream aggregation rate limiter and a depth line. The stream aggregation rate limiter requires the stream processing system to store recent records of stream aggregation updates in the corresponding nodes. During aggregation updates, the system determines whether to upload intermediate aggregation results or upload a blocked state (the blocked state is periodically reset) based on the aggregation update rate, reducing the maintenance cost of updating meaningless intermediate aggregation results. Then, based on the blocked state, a depth line is flexibly marked to form a deep-water zone. The deep-water zone restricts aggregation updates, thereby reducing the aggregation update frequency and the overall overhead of the aggregation algorithm.

[0049] Step (3): Read the input tuple from Kafka using the stream processing system, find the node to which the timestamp of the input tuple belongs, and update that node. Note that when the processing of each tuple is finished, the processing of the next tuple will start again from this step.

[0050] Step (4): Calculate the aggregation update frequency of the corresponding node, and determine whether the aggregation update frequency exceeds the user-given rate limit. If the current node exceeds the rate limit, perform the operation of propagating the blocking state upwards, which will update the water level; otherwise, proceed to the next step. Note that the blocking state will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval between the user's access to the result node (if the user periodically accesses the result node); otherwise, the user determines the length of the reset cycle. In addition, the blocking state has the characteristic of propagating upwards, meaning that when any child node of a node is blocked, the node itself will also be blocked. The specific steps for propagating the blocking state upwards are as follows:

[0051] Step 4-1: Determine the current node's state. If the current node is in a blocked state, the processing of the current tuple ends; if the current node is not in a blocked state, then the current node is blocked.

[0052] Step 4-2: After blocking the current node, check if the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the depth line operation and end the processing of the current tuple.

[0053] Step 4-3: Determine the state of the currently pointed-to node (i.e., the parent node of the original node). If the currently pointed-to node is not blocked, then block the current node and execute Step 4-2; if the currently pointed-to node is blocked, then execute the update depth line operation and then end the processing of the current tuple.

[0054] Step (5): When the aggregation update frequency of the current node does not exceed the rate limit, determine whether the node to which the input tuple belongs is in a blocked state or in a deep water zone. If the current node is in a deep water zone, end the processing of the current tuple. If the current node is not in a blocked state and is not in a deep water zone, pass the update upward. If the upload encounters a node in a blocked state, end the processing of the current tuple in advance. Otherwise, keep uploading the update to the result node and then end the processing of the current tuple. If the current node is in a blocked state, proceed to the next step.

[0055] Step (6): When the current node is in a blocked state, determine the source of the blockage. If there are no other sources of blockage, restore the current underlying node to a free state and perform the upward propagation of the free state operation. It is important to note that because the blocked state is determined from bottom to top, the propagation of the state can only be attempted. The propagation stops when encountering a blockage caused by other child nodes. The specific steps for the upward propagation of the free state operation are as follows:

[0056] Step 6-1: Determine if the current node has a parent node. If not, redefine the water depth line and end the processing of the current tuple. If a parent node exists, proceed to step 6-2.

[0057] Step 6-2: Point the pointer to the parent node of the current node, and then check if the currently pointed-to node (which is already the parent node of the original node) is set to a blocked state. If the node is in a blocked state, visit the other child nodes of the node to see if there are other child nodes that also cause the node to be blocked. If there are other child nodes that cause the parent node to be blocked, then redefine the depth line and the processing of the tuple ends. If there are no other child nodes that cause the parent node to be blocked, then execute step 6-3. If the node is not in a blocked state, directly update the result of the currently pointed-to node and pass the aggregation update upwards, unless a blockage is encountered (in which case the blockage has other sources).

[0058] Step 6-3: Set the node state of the currently pointed-to node to smooth and update the aggregation result of the currently pointed-to node, then execute step 6-1.

[0059] The update of the water depth line occurs when there is a blockage in the upload and the parent node is blocked. The update steps are as follows:

[0060] Step 1: During the upload process in a blocked state, when a parent node is already in a blocked state, the position of that parent node is the intersection of the blocked states. If the intersection of the blocked states is a result node, no operation is performed; otherwise, proceed to Step 2.

[0061] Step 2: Based on the blocking level of the intersecting nodes in the blocking state, Blockage = aggregation update frequency of the intersecting nodes in the blocking state / tree height of the intersecting nodes in the blocking state. If the Blockage is lower than the user-preset Blockage threshold, no operation is performed; otherwise, proceed to Step 3.

[0062] Step 3: Starting from the intersection node, proceed downwards according to the parent-child relationship to find the leftmost bottom-level node in a blocked state (leftmost means the node represents the earliest system time) and the rightmost bottom-level node in a blocked state (rightmost means the node represents the latest system time). Mark a depth line on each of these two nodes. This invention defines the time stamp between the two depth lines as the deep water zone. Nodes within this time range are located in the deep water zone. Note: The existence of the deep water zone is determined by the intersection nodes in the blocked state that define the deep water zone. If the blocking no longer intersects, then the resulting depth lines and deep water zones will be discarded.

[0063]

Example 1

[0064] The stock trading data provided by securities firms is aggregated and then made available to the exchange as various real-time trading data. The steps are as follows:

[0065] 1) Based on the time frame for which the brokerage firm needs to provide information, divide the data into a sufficient number of bottom-level nodes according to the Cutty structure, and then organize them into a tree structure. If multiple results are required, organize them into a hierarchical structure to ensure data aggregation.

[0066] 2) The brokerage firm and the exchange respectively deploy a stream processing system and Kafka. The former is responsible for connecting with the brokerage firm and the exchange to deliver data to Kafka or reading data from the Kafka queue.

[0067] 3) After the data provided by the brokerage arrives at the Kafka queue of the server responsible for processing, the stream processing system summarizes the required information and updates the corresponding underlying nodes. Then, the stream processing system calculates the aggregation update frequency of the corresponding node = the number of aggregation updates of the node within the validity period / the cumulative duration of the validity period, and determines whether the aggregation update frequency exceeds the limit threshold given by the brokerage and the exchange.

[0068] 4) If the speed limit given by the brokerage firm and the exchange is exceeded, perform the following steps:

[0069] ① Determine the current node status on the server. If the current node is in a blocked state, the processing of the currently received data ends; if the current node is not in a blocked state, then block the current node and proceed to step ②.

[0070] ② Determine if the current node has a parent node. If the current node has a parent node, then point the pointer to the parent node of the current node; if the current node does not have a parent node, then update the depth line operation and end the processing of the current tuple.

[0071] ③ Determine the state of the currently pointed-to node (i.e., the parent node of the original node). If the currently pointed-to node is not blocked, then block the current node and execute step ②. If the currently pointed-to node is in a blocked state, then pass the blocked state upwards. If a blocked state encounters another blocked state during the upload process (the parent node is already in a blocked state), then determine whether a deep water zone is formed based on the Blockage of the intersecting nodes, and then end the processing of the current data.

[0072] 5) Determine whether the node to which the input data belongs on the server is in a blocked state or in a deep water zone. If the current node has not exceeded the speed limit and is not in a blocked state or in a deep water zone, determine whether the parent node of the current node is in a blocked state. If not, move to the parent node and update its data. Then, starting from this new current node, continue to explore the state of the upper-level nodes until the result node is uploaded.

[0073] 6) If the current node is in a blocked state, determine the source of the blockage. If no other source of blockage exists, restore the current underlying node to a free state, and then attempt to propagate the state upwards. The steps are as follows: Execute the following steps:

[0074] ① Determine if the current node has a parent node. If not, redefine the water depth line and end the processing of the current data. If a parent node exists, proceed to step ②.

[0075] ② Point the pointer to the parent node of the current node, and then determine whether the current node is in a blocked state. If the node is blocked, check if there are other child nodes that cause the node to be blocked. If there are other child nodes that cause the current node to be blocked, redefine the depth line and end the current data processing. If there are no other reasons for the current node to be blocked, proceed to step ③. If the node is not in a blocked state, switch to uploading the aggregation update brought by the current data. If blocking is encountered, stop directly and end the current data processing.

[0076] ③ Set the node state of the currently pointed-to node to open and update the aggregation result of the current node, then execute step ①.

[0077] In addition, it is important that the data from exchanges and brokerages jumps every few seconds, so the reset time of the deep water zone is consistent with the screen jump update, because the data is updated immediately when the blockage is cleared during the reset.

[0078]

Example 2

[0079] A large number of tuples t0~t k The arrival will be delayed, t i The delayed arrival time is T i For time partition P a Let's say there is a sequence of tuples that will be reached: {t1, t2, ... t} n}, take time point T m There is T m <T1,T2,...T n And P(T) i -T m >400ms)>0.95 (0≤i≤n); At this point, it can be foreseen that if the worst-case scenario is considered, it would be inappropriate unless the current stream aggregation operation is commutative, and it would prolong the system response time, because everything would be delayed in reaching partition P. a The tuples have not yet arrived, and 95% of the tuples will arrive in more than 400ms. Within 400ms, the updated aggregation results for this node should be uploaded to meet current needs, and then blocked again when the load increases, which is more in line with the current scenario. To propose a general method, we cannot only consider the commutativity problem; we need to design specifically for problems where commutativity is not possible.

[0080] 1) We can then take the time length of each underlying node as 100ms and organize the data structure as needed. Based on the required time length for providing information, we can divide the data into a sufficient number of underlying nodes according to the Cutty.

[0081] 2) Deploy this stream processing system and Kafka on each host and server.

[0082] 3) Stream processing systems deployed in various locations receive data and deliver it to Kafka. After the data provided by the producer arrives, it enters the corresponding underlying nodes and updates the corresponding underlying partitions.

[0083] 4) Calculate the aggregate update frequency of the corresponding partition and compare it with the defined threshold.

[0084] 5) When the upload speed exceeds the user-defined limit, the upload is blocked. If a blocking node is encountered, the blockage of the intersecting node is used to determine whether a deep water zone has been formed.

[0085] 6) If the speed limit is not exceeded, and the current node is not in a blocked state or deep water zone, then the update is passed up; if the current node is in a blocked state, then the node attempts to clear the blockage.

[0086] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0087] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for limiting the aggregation rate of out-of-order data streams based on Apache Flink, characterized in that, The steps include the following: Step (1): The user sets the data structure for stateful computation in the current stream processing system according to the requirements of query aggregation results; Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides clients with standardized output and input ports. Data provided by the producer is handed over to Kafka and sent to the consumer's stream processing system. The underlying nodes on the input ports are set as time-oriented aggregation sub-nodes. Step (3): Read the input tuple from Kafka through the stream processing system, find the node to which the timestamp of the input tuple belongs, and update the node; note that when the processing of each tuple is finished, start the processing of the next tuple from this step. Step (4): Calculate the aggregation update frequency of the corresponding node, and determine whether the aggregation update frequency exceeds the user-given rate limit. If the current node exceeds the rate limit, perform the operation of propagating the blocking status upwards. This operation will update the water level. Otherwise, proceed to the next step. Note that the blocking status will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval between the user's access to the result node. Otherwise, the user decides the length of the reset cycle. In addition, the blocking status has the characteristic of propagating upwards, which means that all ancestor nodes will be blocked together. That is, when any child node of a node is blocked, the node will also be blocked. Step (5): If the aggregation update frequency of the current node does not exceed the rate limit, determine whether the node to which the input tuple belongs is in a blocked state or in a deep water zone. If the current node is in a deep water zone, end the processing of the current tuple. If the current node is not in a blocked state and is not in the deep water zone, the update is passed up. If the upload encounters a node in a blocked state, the current tuple processing ends early; otherwise, the update is passed up to the result node and then the current tuple processing ends. If the current node is in a blocked state, proceed to the next step; Step (6): When the current node is in a blocked state, the source of the blockage is determined. If there is no other source of blockage, the current bottom node is restored to a free state, and the free state is passed up. It should be noted that since the blocked state is determined from bottom to top, the upload will stop when the propagation state encounters a blockage caused by other child nodes.

2. The method for limiting the aggregation rate of out-of-order data streams based on Apache Flink according to claim 1, characterized in that, The specific steps for step (1) are as follows: Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure according to the type of aggregation operation. The data structure includes bottom-level nodes, intermediate nodes, and result nodes. The information within each node includes: an update count field and a stream aggregation data structure; the number of times a record needs to be updated and the time period for which the update occurred; bottom-level nodes are segmented using a Cutty method, and to ensure availability to upper-level nodes, they are decomposed into the largest shared child nodes based on window sliding; bottom-level nodes exist in two states: propagation state and blocking state; intermediate and result nodes are also affected by these two node states, and their node states are a summary of the bottom-level nodes they encompass.

3. The method for limiting the aggregation rate of out-of-order data streams based on Apache Flink according to claim 1, characterized in that, The specific steps for the upward propagation of the blocked state in step (4) are as follows: Step 4-1: Determine the current node's state. If the current node is in a blocked state, the processing of the current tuple ends; if the current node is not in a blocked state, then the current node is blocked. Step 4-2: After blocking the current node, check if the current node has a parent node. If the current node has a parent node, set the pointer to the parent node of the current node; if the current node does not have a parent node, update the depth line operation and end the processing of the current tuple. Step 4-3: Determine the state of the currently pointed-to node. If the currently pointed-to node is not blocked, then block the current node and execute Step 4-2. If the currently pointed-to node is blocked, then execute the update depth line operation and then end the processing of the current tuple.

4. The method for limiting the aggregation rate of out-of-order data streams based on Apache Flink according to claim 1, characterized in that, Step (6) The specific steps for transmitting the smooth status upwards are as follows: Step 6-1: Determine if the current node has a parent node. If not, redefine the water depth line and end the processing of the current tuple. If a parent node exists, proceed to step 6-2. Step 6-2: Set the pointer to the parent node of the current node, and then check if the currently pointed-to node is set to a blocked state. If the node is blocked, access the other child nodes of the node to see if there are other child nodes that also cause the node to be blocked. If there are other child nodes that cause the parent node to be blocked, redefine the depth line and the processing of the tuple ends. If there are no other child nodes that cause the parent node to be blocked, proceed to step 6-3. If the node is not blocked, directly update the result of the currently pointed-to node and pass the aggregation update upwards, unless a blockage is encountered. Step 6-3: Set the node state of the currently pointed-to node to smooth and update the aggregation result of the currently pointed-to node, then execute step 6-1.

5. The method for limiting the aggregation rate of out-of-order data streams based on Apache Flink according to claim 1, characterized in that, The depth line update occurs when there is a blocking upload and the parent node is also blocked. The update steps are as follows: Step 1: When uploading in a blocked state, if the parent node is already in a blocked state, the position of that parent node is the intersection of the blocked states. If the intersection of the blocked states is the result node, no operation is performed; otherwise, proceed to Step 2. Step 2: Based on the blocking level of the intersecting nodes in the blocking state, Blockage = aggregation update frequency of the intersecting nodes in the blocking state / tree height of the intersecting nodes in the blocking state. If the Blockage is lower than the user-preset Blockage threshold, no operation is performed; otherwise, proceed to Step 3. Step 3: Starting from the intersection node, follow the parent-child relationship downwards to find the leftmost and rightmost bottom-level nodes that are in a blocked state. Mark a depth line on each of these two nodes. Define the time stamp between the two depth lines as the deep water zone. Nodes within this time range are in the deep water zone. Note that the existence of the deep water zone is determined by the intersection nodes of the blocked state that define the deep water zone. If the blocking no longer intersects, then the depth lines and deep water zones generated as a result will be discarded.

Images