Out-of-order data stream aggregation rate limiting method based on Apache Flink

Abstract

The invention discloses an out-of-order data flow aggregation rate limiting method based on Apache Flink. The stream processing system provided by the invention additionally comprises a stream aggregation rate limiter and a water depth line. The method is applied to a structure with a hierarchical structure (a hierarchical DAG graph) in aggregation, aims at a stateful distributed flow aggregation problem, and optimizes the aggregation performance in a mode of transmitting node states upwards. According to the invention, Apache Kafka is adopted to communicate with producers and consumers. In the aggregation updating process, whether the intermediate aggregation result is uploaded or the blocking state is uploaded is judged according to the aggregation updating rate, the maintenance cost for updating the meaningless aggregation intermediate result is reduced, then the water depth line is flexibly marked according to the blocking state to form a deep water area, the deep water area limits aggregation updating, and therefore the aggregation updating frequency is reduced. Therefore, the total overhead of the aggregation algorithm is reduced.

Description

Technical Field

[0001] The invention belongs to the field of data stream processing and relates to a method for limiting aggregation high-frequency overhead by transmitting node status in the stream processing problem of disordered stream. Background Art

[0002] Existing out-of-order stream processing usually works in a watermark manner, that is, a timestamp is used to divide the area for different processing, and only the data before the timestamp is processed. For example, a paper titled "CPiX: Real-Time Analytics Over Out-of-Order Data Streams by Incremental Sliding-Window Aggregation" published in TKDE in 2021 disclosed "an out-of-order stream processing method for algebraic aggregation problems". The article believes that the reason for the poor performance of the current mainstream stream aggregation algorithms in the big data field is that the stream processing system wastes too much time maintaining intermediate aggregation results in severe out-of-order conditions. CPiX speeds up the aggregation and reduces the cost of maintaining intermediate results in severe data stream out-of-order conditions by shortening the path for uploading aggregations. The way to shorten the aggregation upload path is to put the aggregation nodes in the same pool, take advantage of the interchangeability of some aggregation types, abandon the maintenance of all intermediate aggregation nodes in the near future, and only maintain the earliest and oldest nodes. Because the delay is limited, the records should be discarded after the maximum survival time (TTL). Therefore, it can be considered that the earlier the record, the less likely there is to be unreached records, so disorder will not occur. This method of aggregating disordered data streams has defects. Because it shortens the aggregation path, its processing capacity becomes single, and it has no ability to cope with aggregations with interchangeable order. Therefore, there is a problem of sacrificing the processing range in exchange for improved processing performance. When dealing with some complex overall aggregations, especially when dealing with aggregation problems that are interchangeable but not reversible, because the aggregation path is not retained, all can only be completely re-aggregated.

[0003] Data flow is a concept in computer science that describes the path of data in a system. Data flow can show the transmission process of data from the point of generation to the point of consumption, as well as the storage, transformation, query and other operations that the data may undergo during this process. Data flow is unbounded and infinite. Stream processing systems are systems that can process data streams in real time and continuously. They are often used in application scenarios that require fast response, such as real-time analysis, monitoring and decision support systems.

[0004] Partitioning algorithms are often designed in stream processing algorithms. Partitioning in stream processing is a key link, which affects the performance, scalability, and fault tolerance of the system. The number of partitions should match the parallelism of the processing resources to make full use of the computing resources. Try to ensure that related data is in the same partition to reduce cross-partition communication. The partitioning strategy should support fault tolerance, such as through data backup or multi-copy mechanism. The partitioning strategy should support dynamic expansion to adapt to changes in data volume. The partitioning algorithm used in CPiX is Cutty, which is a method of partitioning based on sliding. In a stream processing system, the user program that generates data is usually called a producer, and the user program that obtains the data provided by the system is usually called a consumer.

[0005] The watermark mechanism is an important concept used in the stream processing framework to handle event time, especially in scenarios where out-of-order events are processed or calculations based on time windows are required. It is a logical timestamp that marks that all data before a specific time point has arrived at the stream processing system, and it can be assumed that data before this time point will not appear again. In other words, the watermark is a delay boundary that allows the system to perform calculations after observing all relevant events. Watermarks can be divided into periodic watermarks and punctuated watermarks according to the type of operation. Periodic watermarks are issued periodically, for example, every time a certain number of records are received or at a certain time interval; while punctuated watermarks are issued when a specific event occurs, for example, after processing an event with specific attributes.

[0006] Apache Flink is an open source stream processing framework for stateful computing with high throughput and low latency. It can process bounded and unbounded data streams and is easy to develop and deploy large-scale data processing applications. Apache Flink provides multiple APIs to write data processing applications, including: DataSetAPI (for batch processing), DataStreamAPI (for stream processing), TableAPI (for structured data processing, similar to SQL) and SQL (query directly using SQL).

[0007] Apache Kafka is a platform for building real-time data pipelines and streaming applications. Kafka can be regarded as a super message queue with high performance, high scalability, high availability and support for persistence.

[0008] Stateful distributed stream processing is a common problem faced by stream processing platforms such as Apache Flink. Stateful means that the accumulated state is saved in the stream processing, that is, all events received in the past history; simple operations such as Sum counting and complex state accumulation such as machine learning are all stateful stream processing. Distributed means that the servers that generate and process information are decentralized and independent computers; these hosts are physically separated from each other but are considered as the same host.

[0009] The commutativity of aggregation means that if the order of the sequential elements of the aggregation is exchanged, the result of the aggregation cannot be guaranteed to be consistent with the original aggregation result. For aggregation problems where the aggregation is not commutative, it is usually necessary to maintain many intermediate results. Therefore, the nodes in this type of stream aggregation can be divided into three types: bottom-level exchange nodes, intermediate nodes, and result output nodes. The bottom-level exchange node is the lowest-level node, which can directly interact with the input data stream and obtain updates of the aggregation information directly from the input data stream. The intermediate node refers to a node that summarizes the information in some bottom-level exchange nodes. The intermediate node can be invisible to the outside and has no meaning to the outside. Each intermediate node is a summary of the information of several bottom-level exchange nodes. There are often more than one layer of intermediate nodes (bottom-level nodes rarely have only one layer). The result node is the output end of the entire aggregation, and the consumer directly obtains the result from the result output node. Usually, there is only one such result node. The present invention will use a hierarchical structure to refer to the organizational structure of data in this stream processing algorithm that is divided into three categories. It is essentially a directed hierarchical graph (DAG), but this article emphasizes its hierarchical characteristics, so it is referred to by a hierarchical structure. This hierarchical structure is necessary when the aggregation operation is not interchangeable, and the research of the present invention is carried out on this basis. Summary of the invention

[0010] In order to overcome the defect of the existing stream aggregation technology that the overhead is too high when dealing with the problem of non-interchangeability, the present invention provides a method for limiting the aggregation rate of disordered data streams based on Apache Flink. The stream processing system proposed in the present invention additionally includes a stream aggregation rate limiter and a water depth line to limit the excessive overhead of the operation of transmitting node status. The present invention is applied to the structure of the aggregation existence hierarchical structure (a layered DAG graph), facing the stateful distributed stream aggregation problem. The present invention uses Apache Kafka to communicate with producers and consumers.

[0011] The stream aggregation rate limiter requires the stream processing system to save recent records of stream aggregation updates in the corresponding nodes. When updating the aggregation, the aggregation update rate is used to determine whether to upload the intermediate aggregation results or the blocking status (the blocking status is reset regularly), thereby reducing the maintenance cost of updating meaningless intermediate aggregation results. Then, the water depth line is flexibly marked according to the blocking status to form a deep water area. The deep water area limits the aggregation update, thereby reducing the frequency of aggregation updates and reducing the overall overhead of the aggregation algorithm. The data structure is layered up and down, and the data structures at different levels cover different amounts of data represented by information; in the parent-child relationship, the parent node always covers a larger amount of data than its child node.

[0012] The stream processing system in the present invention runs in a distributed environment, and provides output ports and input ports to the clients who provide data sources. The aggregation form at the output port of the sender is unified with the format of the receiver, and the bottom-level node on the input port of the receiver is set as a time-oriented aggregation sub-node. The aggregation update starts from the bottom-level node as a sub-node, and propagates upward to a more abstract level by summarizing the hierarchical relationship of the information through the aggregation structure; the bottom-level node on the receiver that directly interacts with the input data stream, the intermediate node that aggregates the lower-level information, the node storage pool, and the user program that receives the data together constitute a complete structure for the aggregated received data.

[0013] The water depth line is an extension of the traditional water level line. Although there are many types of traditional water levels, there is only one, and the scope of influence is only after the timestamp of the water level line. The water depth line defines the deep water area. The deep water area, like the blocking state, can limit the upload of aggregate updates, and it is reset when the blocking state is reset. During data transmission, due to the complexity of the network environment, the delay in data arrival is completely unpredictable. In CPiX, it is believed that earlier records are safer, that is, fewer disorder problems will need to be dealt with in the future. This is logically correct, but the problem is that more in the future does not mean more at the moment, so the use of the water depth line can flexibly limit the scope of aggregate update uploads.

[0014] The technical solution adopted by the present invention to solve its technical problem is:

[0015] A method for limiting the aggregation rate of out-of-order data streams based on Apache Flink uses the transfer node state to limit the invalid aggregation overhead, including the following steps:

[0016] Step (1): The user sets the data structure of the stateful computation in the current stream processing system according to the requirements of the query aggregation results (only the stateful computation of Apache Flink is changed).

[0017] Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides clients with output ports and input ports in a unified format. The data provided by the producer is handed over to Kafka to be sent to the stream processing system on the consumer side. The underlying node on the input port is set as a time-oriented aggregation subnode.

[0018] Step (3): Read the input tuple from Kafka through the stream processing system, find the node to which the timestamp of the input tuple belongs, and update the node. Note that when the processing of each tuple is completed, start again from this step to process the next tuple.

[0019] Step (4): Calculate the aggregate update frequency of the corresponding node, and determine whether the aggregate update frequency exceeds the speed limit given by the user. If the current node exceeds the speed limit, perform an operation to pass the blocking state upward, which will update the watermark; otherwise, proceed to the next step. Note that the blocking state will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval for the user to access the result node (if the user periodically accesses the result node), otherwise the user determines the length of the reset cycle; in addition, the blocking state has the characteristic of propagating upward even if all ancestor nodes are blocked together, that is, when any child node of a node is blocked, this node will also be blocked.

[0020] Step (5): When the aggregation update frequency of the current node does not exceed the speed limit, determine whether the node to which the input tuple belongs is in a blocked state or in the deep water area. If the current node is in the deep water area, end the current tuple processing; if the current node is not in a blocked state and is not in the deep water area, pass the update upward. When the upload encounters a node in a blocked state, end the current tuple processing in advance; otherwise, continue to upload the update to the result node and then end the current tuple processing; if the current node is in a blocked state, proceed to the next step.

[0021] Step (6): When the current node is in a blocked state, determine the source of the blockage. If there is no other blocking source, restore the current bottom node to a smooth state and perform the operation of passing the smooth state upward. It should be noted that because the blocking state is determined from the bottom up, the upload of the propagation state can only be attempted. When encountering a blockage caused by other child nodes, the upload stops.

[0022] Furthermore, the specific operation of step (1) is as follows:

[0023] Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure (tree or network, such as binary tree, B-tree, neural network and other data structures) according to the type of aggregation operation;

[0024] The data structure includes bottom-level nodes, intermediate nodes, and result nodes. The information in the node includes: including the update count field and the stream aggregation data structure, the number of times the bottom-level node needs to update the record and the time of the update. The bottom-level nodes are cut in a Cutty manner. To ensure the availability of the upper-level nodes, the bottom-level nodes are decomposed into the largest shared child nodes according to the sliding of the window. There are two states of the bottom-level nodes: propagation state and blocking state. The intermediate nodes and result nodes are also affected by the above two node states, and their node states are the summary of the bottom-level nodes of the aggregation they cover.

[0025] Furthermore, the specific steps of the operation of transferring the blocking state upward in step (4) are as follows:

[0026] Step 4-1: Determine the current node state. If the current node is in a blocked state, the current tuple processing ends; if the current node is not in a blocked state, the current node is blocked.

[0027] Step 4-2: After blocking the current node, determine whether the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the water depth line operation and end the current tuple processing.

[0028] Step 4-3: Determine the state of the current pointing node (i.e., the parent node of the original node). If the current pointing node is not blocked, block the current node and then execute step 4-2; if the current pointing node is in a blocked state, execute the water depth line update operation and then end the processing of the current tuple.

[0029] Furthermore, the specific steps of the operation of passing the unobstructed state upward in step (6) are as follows:

[0030] Step 6-1: Determine whether the current node has a parent node. If not, redraw the water depth line and end the processing of the current tuple. If the parent node exists, go to step 6-2.

[0031] Step 6-2: Point the pointer to the parent node of the current node, and then determine whether the currently pointed node (which is already the parent node of the original node) is set to a blocked state. If the node is in a blocked state, access other child nodes of the node to see if there are other child nodes that also constitute the reason for the node to be blocked. If there are other child nodes that cause the parent node to be blocked, the tuple processing is completed after the water depth line is redrawn; if there are no other child nodes that cause the parent node to be blocked, execute step 6-3; if the node is not in a blocked state, directly update the result of the currently pointed node and pass the aggregate update upward, unless it encounters a blockage, it will stop (at this time, the blockage has other sources).

[0032] Step 6-3: Set the node status of the currently pointing node to unblocked and update the aggregation result of the currently pointing node, and then execute step 6-1.

[0033] Furthermore, the update of the water depth line occurs when the upload is blocked and the parent node is blocked, and the update steps are as follows:

[0034] Step 1: When the parent node is already in the blocked state during the blocking state upload process, the position of the parent node is the blocked state intersection position. If the blocked state intersection position is the result node, no operation is performed, otherwise, step 2 is executed.

[0035] Step 2: According to the blocking level Blockage of the blocking state intersection node = aggregation update frequency of the blocking state intersection node / tree height of the blocking state intersection node, if the Blockage is lower than the Blockage threshold preset by the user, no operation is performed, otherwise, step 3 is executed.

[0036] Step 3: Starting from the intersection node, go downward according to the parent-child relationship, find the leftmost bottom node in the blocked state (the leftmost node represents the earliest system time) and the rightmost bottom node in the blocked state (the rightmost node represents the latest system time), mark a water depth line on each of these two nodes, and mark the time stamps between the two water depth lines as the deep water area. The nodes within this time range are in the deep water area. Note: The existence of the deep water area is determined by the intersection node of the blocking state that demarcates the deep water area. If the blocking no longer intersects, the resulting water depth line and deep water area will also be abandoned.

[0037] The beneficial effects of the present invention are as follows:

[0038] The present invention uses the method of limiting the update of some intermediate aggregation results by transferring node status to limit the system overhead, effectively eliminating the problems of large processing overhead and low update utilization rate of the out-of-order stream aggregation method when facing data with a high degree of disorder. After setting the initial conditions of the restriction on the bottom node, the node blocking state can be conveniently set according to the frequency of recent aggregation updates of the node, and when the invalid overhead is too large, the upward transmission of the status is used instead of the transmission of the aggregation update. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] Figure 1 is a processing flow chart of a stream processing system according to an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the present invention on a binary tree. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 As shown in the figure, a method for limiting the aggregation rate of out-of-order data streams based on Apache Flink is shown. The operation process is as follows:

[0044] Step (1): The user sets the data structure of the stateful computation in the current stream processing system according to the requirements of the query aggregation results (only the stateful computation of Apache Flink is changed). The specific operations are as follows:

[0045] Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure (tree or network, such as binary tree, B-tree, neural network and other data structures) according to the type of aggregation operation; Figure 2 It is a schematic diagram of the structure of the present invention on a binary tree.

[0046] The data structure includes bottom-level nodes, intermediate nodes, and result nodes. The information in the node includes: including the update count field and the stream aggregation data structure, the number of times the bottom-level node needs to update the record and the time of the update. The bottom-level nodes are cut in a Cutty manner. To ensure the availability of the upper-level nodes, the bottom-level nodes are decomposed into the largest shared child nodes according to the sliding of the window. There are two states of the bottom-level nodes: propagation state and blocking state. The intermediate nodes and result nodes are also affected by the above two node states, and their node states are the summary of the bottom-level nodes of the aggregation they cover.

[0047] Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides clients with output ports and input ports in a unified format. The data provided by the producer is handed over to Kafka to be sent to the stream processing system on the consumer side. The underlying node on the input port is set as a time-oriented aggregation subnode.

[0048] The stream processing system proposed in the present invention additionally includes a stream aggregation rate limiter and a water depth line. The stream aggregation rate limiter requires the stream processing system to save recent records of stream aggregation updates in the corresponding nodes. When the aggregation is updated, the aggregation update rate is used to determine whether to upload the intermediate aggregation result or the blocking state (the blocking state is reset regularly), thereby reducing the maintenance cost of updating meaningless intermediate aggregation results, and then the water depth line is flexibly marked according to the blocking state to form a deep water area. The deep water area limits the aggregation update, thereby reducing the aggregation update frequency, thereby reducing the overall overhead of the aggregation algorithm.

[0049] Step (3): Read the input tuple from Kafka through the stream processing system, find the node to which the timestamp of the input tuple belongs, and update the node. Note that when the processing of each tuple is completed, start again from this step to process the next tuple.

[0050] Step (4): Calculate the aggregation update frequency of the corresponding node, and determine whether the aggregation update frequency exceeds the speed limit given by the user. If the current node exceeds the speed limit, perform the operation of passing the blocking state upward, which will update the waterline; otherwise, proceed to the next step. Note that the blocking state will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval for the user to access the result node (if the user periodically accesses the result node), otherwise the length of the reset cycle is determined by the user; in addition, the blocking state has the characteristic of propagating upward even if all ancestor nodes are blocked together, that is, when any child node of a node is blocked, this node will also be blocked. The specific steps for passing the blocking state upward are as follows:

[0051] Step 4-1: Determine the current node state. If the current node is in a blocked state, the current tuple processing ends; if the current node is not in a blocked state, the current node is blocked.

[0052] Step 4-2: After blocking the current node, determine whether the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the water depth line operation and end the current tuple processing.

[0053] Step 4-3: Determine the state of the current pointing node (i.e., the parent node of the original node). If the current pointing node is not blocked, block the current node and then execute step 4-2; if the current pointing node is in a blocked state, execute the water depth line update operation and then end the processing of the current tuple.

[0054] Step (5): When the aggregation update frequency of the current node does not exceed the speed limit, determine whether the node to which the input tuple belongs is in a blocked state or in the deep water area. If the current node is in the deep water area, end the current tuple processing; if the current node is not in a blocked state and is not in the deep water area, pass the update upward. When the upload encounters a node in a blocked state, end the current tuple processing in advance; otherwise, continue to upload the update to the result node and then end the current tuple processing; if the current node is in a blocked state, proceed to the next step.

[0055] Step (6): When the current node is in a blocked state, determine the source of the blockage. If there is no other source of blockage, restore the current bottom node to a state of unblocked state, and perform the operation of passing the unblocked state upward. It should be noted that because the blocked state is determined from the bottom up, the upload of the propagation state can only be attempted. When encountering a blockage caused by other child nodes, the upload stops. The specific steps for passing the unblocked state upward are as follows:

[0056] Step 6-1: Determine whether the current node has a parent node. If not, redraw the water depth line and end the processing of the current tuple. If the parent node exists, go to step 6-2.

[0057] Step 6-2: Point the pointer to the parent node of the current node, and then determine whether the currently pointed node (which is already the parent node of the original node) is set to a blocked state. If the node is in a blocked state, access other child nodes of the node to see if there are other child nodes that also constitute the reason for the node to be blocked. If there are other child nodes that cause the parent node to be blocked, the tuple processing is completed after the water depth line is redrawn; if there are no other child nodes that cause the parent node to be blocked, execute step 6-3; if the node is not in a blocked state, directly update the result of the currently pointed node and pass the aggregate update upward, unless it encounters a blockage, it will stop (at this time, the blockage has other sources).

[0058] Step 6-3: Set the node status of the currently pointing node to unblocked and update the aggregation result of the currently pointing node, and then execute step 6-1.

[0059] The update of the water depth line occurs when the upload is blocked and the parent node is blocked, and the update steps are as follows:

[0060] Step 1: When the parent node is already in the blocked state during the blocking state upload process, the position of the parent node is the blocked state intersection position. If the blocked state intersection position is the result node, no operation is performed, otherwise, step 2 is executed.

[0061] Step 2: According to the blocking level Blockage of the blocking state intersection node = aggregation update frequency of the blocking state intersection node / tree height of the blocking state intersection node, if the Blockage is lower than the Blockage threshold preset by the user, no operation is performed, otherwise, step 3 is executed.

[0062] Step 3: Starting from the intersection node, go downward according to the parent-child relationship, find the leftmost bottom node in a blocked state (the leftmost node represents the earliest system time) and the rightmost bottom node in a blocked state (the rightmost node represents the latest system time), and mark a water depth line on each of these two nodes. The present invention marks the time stamps between the two water depth lines as a deep water area, and the nodes within this time range are in the deep water area. Note: The existence of the deep water area is determined by the intersection node of the blocking state that demarcates the deep water area. If the blocking no longer intersects, the water depth line and deep water area generated thereby will also be abandoned.

[0063] [Example 1]

[0064] Summarize the stock trading data provided by brokers and provide various real-time trading data on the exchange. The steps are as follows:

[0065] 1) According to the length of time that the broker needs to provide information, a sufficient number of underlying nodes are divided according to Cutty, and then organized into a tree form. If non-single results are required, they are organized into a hierarchical structure to ensure data aggregation.

[0066] 2) The stream processing system and Kafka are deployed by the brokerage firm and the exchange respectively. The former is responsible for connecting the data with the brokerage firm and the exchange and handing it over to Kafka or reading data from the Kafka queue.

[0067] 3) After the data provided by the broker arrives at the Kafka queue of the server responsible for processing, the stream processing system summarizes the required information and updates the corresponding underlying nodes. Then the stream processing system calculates the aggregation update frequency of the corresponding node = the number of aggregation updates of the node during the validity period / the cumulative duration of the validity period, and determines whether the aggregation update frequency exceeds the limit threshold of the aggregation update frequency given by the broker and the exchange.

[0068] 4) When the speed limit set by the broker and the exchange is exceeded, perform the following steps:

[0069] ① Determine the current node status on the server. If the current node is in a blocked state, the processing of the currently received data is completed; if the current node is not in a blocked state, block the current node and then proceed to step ②.

[0070] ② Determine whether the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the water depth line operation and end the current tuple processing.

[0071] ③ Determine the state of the current pointing node (i.e., the parent node of the original node). If the current pointing node is not blocked, block the current node and then execute step ②; if the current pointing node is in a blocked state, pass the blocked state upward; if a blocked state encounter occurs during the blocked state upload process (the parent node is already in a blocked state), determine whether a deep water area is formed based on the Blockage of the intersection node, and then end the processing of the current data.

[0072] 5) Determine whether the node to which the input data belongs on the server is in a blocked state or in the deep water area. If the current node does not exceed the speed limit and the current node is not in a blocked state or in the deep water area, determine whether the parent node of the current node is in a blocked state. If not, move to the parent node and update its data. Then, starting from this new current node, continue to explore the status of the upper node until it is uploaded to the result node.

[0073] 6) If the current node is in a blocked state, determine the source of the blockage. If there is no other blocking source, restore the state of the current underlying node to unblocked, and then try to upload the propagation state upwards. The steps are as follows: Perform the following steps:

[0074] ① Determine whether the current node has a parent node. If not, redraw the water depth line and end the processing of the current data. Only if the parent node exists can step ② be entered.

[0075] ② Point the pointer to the parent node of the current node, and then determine whether the current node is in a blocked state. If the node is blocked, check whether there are other child nodes that cause the node to be blocked. If there are other child nodes that cause the current node to be blocked, redraw the water depth line and end the current data processing; if there is no other reason that causes the current node to be blocked, execute step ③. If the node is not in a blocked state, upload the aggregate update brought by the current data instead. If blocked, stop and end the processing of the current data directly.

[0076] ③Set the node status of the currently pointed node to unblocked and update the aggregation result of the current node, and then execute step ①.

[0077] In addition, it is important that the data from exchanges and brokerages jumps every few seconds, so the reset time of the deep water area is consistent with the screen jump update, because the blockage is cleared during the reset, causing the data to be updated immediately.

[0078] [Example 2]

[0079] A large number of tuples t0~t k Will be delayed, t i The delayed arrival time is T i , for time partition P a For example, there is a sequence of tuples {t1, t2, ... t n}, take time point T m , there is T m <T1,T2,...T n And P(T i -T m >400ms)>0.95(0≤i≤n); At this time, it can be foreseen that if the worst case is processed in the future, it will be inappropriate unless the current stream aggregation operation is interchangeable, and it will prolong the system response time, because at this time all will be delayed to reach partition P a tuples have not arrived and 95% of the tuples arrive in more than 400ms. The updated aggregation results of this node are uploaded within 400ms to meet the current needs, and then blocked at a later time when the load increases. To propose a general method, we cannot only consider the exchangeable problem, but also need to design a special method for the problem without exchangeability.

[0080] 1) Then we can take the bottom node time length as 100ms per node and organize the data structure as needed. According to the time length required to provide information, a sufficient number of bottom nodes are divided by Cutty.

[0081] 2) Deploy the stream processing system and Kafka on each host and server.

[0082] 3) The stream processing systems deployed in various locations receive data and hand it over to Kafka. After the data provided by the producer arrives, it enters the corresponding underlying node and updates the corresponding underlying partition.

[0083] 4) Calculate the aggregate update frequency of the corresponding partition and compare it with the defined threshold.

[0084] 5) When the speed exceeds the speed limit given by the user, the blocking status is uploaded. If a blocking node is encountered, whether a deep water area is formed is determined based on the Blockage of the intersection node.

[0085] 6) When the speed limit is not exceeded, if the current node is not in a blocked state or in a deep water area, the update is passed upward; if the current node is in a blocked state, an attempt is made to eliminate the blocked state upward.

[0086] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0087] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for limiting the aggregation rate of out-of-order data streams based on Apache Flink, characterized in that: The steps include: Step (1): The user sets the data structure of stateful computing in the current stream processing system according to the requirements of query aggregation results; Step (2): Deploy the stream processing system and Apache Kafka in a distributed environment. The stream processing system obtains data streams from Kafka. The stream processing system provides the client with output ports and input ports in a unified format. The data provided by the producer is handed over to Kafka and sent to the stream processing system on the consumer side. The underlying node on the input port is set as a time-oriented aggregation subnode. Step (3): Read the input tuple from Kafka through the stream processing system, find the node to which the timestamp of the input tuple belongs, and update the node; note that when the processing of each tuple is completed, start again from this step to process the next tuple; Step (4): Calculate the aggregation update frequency of the corresponding node, and determine whether the aggregation update frequency exceeds the speed limit given by the user. If the current node exceeds the speed limit, perform an operation to pass the blocking state upward, which will update the watermark; otherwise, proceed to the next step; note that the blocking state will be reset periodically. When there is a fixed data access cycle, the reset cycle is the same as the time interval for the user to access the result node, otherwise the user determines the length of the reset cycle; in addition, the blocking state has the characteristic of propagating upward even if all ancestor nodes are blocked together, that is, when any child node of a node is blocked, this node will also be blocked; Step (5): When the aggregation update frequency of the current node does not exceed the speed limit, determine whether the node to which the input tuple belongs is in a blocked state or in a deep water area. If the current node is in the deep water area, end the current tuple processing; If the current node is not in a blocked state and is not in the deep water area, the update is passed upward. When the upload encounters a node in a blocked state, the current tuple processing is terminated in advance. Otherwise, the update is uploaded to the result node and then the current tuple processing is terminated. If the current node is in a blocked state, proceed to the next step; Step (6): When the current node is in a blocked state, determine the source of the blockage. If there is no other source of blockage, restore the state of the current underlying node to unblocked, and perform the operation of passing the unblocked state upward. It should be noted that because the blocking state is determined from the bottom up, when the upload of the propagation state encounters a blockage caused by other child nodes, the upload stops.

2. According to claim 1, a method for limiting the aggregation rate of disordered data streams based on Apache Flink is characterized in that: The specific operations of step (1) are as follows: Set the blocking status update cycle according to the user query cycle, determine the number of nodes and time length according to the timeliness of the data and the display requirements, and select the appropriate data structure according to the type of aggregation operation; The data structure includes bottom-level nodes, intermediate nodes and result nodes; the information in the node includes: including the update count field and the stream aggregation data structure, the number of times the bottom-level node needs to update the record and the time of the update; the bottom-level nodes are cut in a Cutty manner, and the bottom-level nodes are decomposed into the largest common child nodes according to the sliding of the window to ensure the availability of the upper-level nodes; the bottom-level nodes have two states: propagation state and blocking state; the intermediate nodes and result nodes are also affected by the above two node states, and their node states are the summary of the aggregated bottom-level nodes they cover.

3. According to claim 1, a method for limiting the aggregation rate of disordered data streams based on Apache Flink is characterized in that: The specific steps of the upward transmission blocking state operation of step (4) are as follows: Step 4-1: Determine the current node state. If the current node is in a blocked state, the current tuple processing ends; if the current node is not in a blocked state, the current node is blocked; Step 4-2: After blocking the current node, determine whether the current node has a parent node. If the current node has a parent node, point the pointer to the parent node of the current node; if the current node does not have a parent node, update the water depth line operation and end the current tuple processing; Step 4-3: Determine the state of the current pointing node. If the current pointing node is not blocked, block the current node and then execute step 4-2; if the current pointing node is in a blocked state, execute the water depth line update operation and then end the processing of the current tuple.

4. According to the method of claim 1, the method is characterized in that: Step (6) The specific steps of transmitting the unobstructed state upward are as follows: Step 6-1: Determine whether the current node has a parent node. If not, redraw the water depth line and end the processing of the current tuple. If the parent node exists, proceed to step 6-2; Step 6-2: Point the pointer to the parent node of the current node, and then determine whether the currently pointed node is set to a blocked state. If the node is in a blocked state, access other child nodes of the node to check whether there are other child nodes that also constitute the reason for the node to be blocked. If there are other child nodes that cause the parent node to be blocked, the tuple processing is completed after the water depth line is redrawn; if there are no other child nodes that cause the parent node to be blocked, execute step 6-3; if the node is not in a blocked state, directly update the result of the currently pointed node and pass the aggregate update upward, unless it encounters a blockage, it will stop; Step 6-3: Set the node status of the currently pointing node to unblocked and update the aggregation result of the currently pointing node, and then execute step 6-1.

5. According to the method of claim 1, the method is characterized in that: The update of the water depth line occurs when the upload is blocked and the parent node is blocked. The update steps are as follows: Step 1: When the parent node is already in the blocked state during the blocked state upload process, the position of the parent node is the blocked state intersection position; if the blocked state intersection position is the result node, no operation is performed, otherwise, step 2 is executed; Step 2: According to the blocking level of the blocked state intersection node Blockage = aggregation update frequency of the blocked state intersection node / tree height of the blocked state intersection node, if the Blockage is lower than the Blockage threshold preset by the user, no operation is performed, otherwise, step 3 is executed; Step 3: Starting from the intersection node, go downward according to the parent-child relationship to find the leftmost bottom node in a blocked state and the rightmost bottom node in a blocked state, mark a water depth line on each of the two nodes, and mark the time stamps represented by the two water depth lines as the deep water area. The nodes within this time range are in the deep water area. It should be noted that the existence of the deep water area is determined by the intersection node of the blocked state that demarcates the deep water area. If the blockage no longer intersects, the water depth line and deep water area generated thereby will also be abandoned.

Images