Method and system for data streaming aggregation, and streaming processing node

Abstract

The embodiment of the present application provides a method and system for data streaming aggregation, and a streaming processing node. The method for data streaming aggregation includes: obtaining the streaming data to be aggregated, and abstracting the streaming data to be aggregated into the required metadata; when the time window is opened, integrating the data attributes of the metadata; and when the time window is closed, submitting the integrated metadata to the Redis cache. The embodiment of the present invention solves the problem of the cluster requiring a large amount of network IO overhead through the abstract processing of metadata and the way in which each node independently calculates metadata, greatly reduces the requirements of the streaming aggregation system for computing resources, and reduces operating costs.

Description

Technical Field

[0001] The present application relates to the fields of computers and streaming data technology, and in particular to a method and system for data streaming aggregation, and a streaming processing node. Background Art

[0002] In the traditional data processing process, data is always collected first and then stored in the database. When needed, the data is queried through the database. Although this seems very reasonable, the results are very compact, especially for some specific problems in some real-time search application environments. This leads to a new data computing structure - stream computing. It can well analyze large-scale flow data in real time during the ever-changing movement process, capture potentially useful information, and send the results to the next computing node.

[0003] Currently, the industry generally uses technical frameworks such as spark / flink to implement streaming computing. This method has framework characteristics, so it is widely used, but this method also has some problems when aggregating indicators in streaming. For example, when aggregating indicators according to time windows, the streaming data to be aggregated needs to be redistributed within the cluster according to tasks. This operation will occupy a large amount of network IO, as well as a large amount of CPU and memory. Therefore, the framework cluster has high requirements for host configuration and high operating costs. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and system for data streaming aggregation, and a streaming processing node, so as to solve the defects of current streaming computing.

[0005] In order to achieve the above-mentioned purpose, the present application provides a method for data streaming aggregation, which includes: obtaining streaming data to be aggregated, and abstracting the streaming data to be aggregated into required metadata; when a time window is opened, integrating data attributes of the metadata; and when the time window is closed, submitting the integrated metadata to the Redis cache.

[0006] Optionally, abstracting the streaming data to be aggregated into required metadata includes: extracting data indicators according to the type of the streaming data to be aggregated; abstracting data attributes of the metadata based on the extracted data indicators; and generating the metadata based on the data attributes.

[0007] Optionally, before integrating the metadata when the time window is opened, the method for data streaming aggregation also includes: determining a time window interval based on the event time when the streaming task event occurs, the time window interval being the time between the opening of the time window and the closing of the time window.

[0008] Optionally, when the time window is opened, the data attributes of the metadata are integrated, including: if the metadata is the first data in the time window interval, registering the node in the Redis cache; and opening the window timer, and accumulating the data attributes of the metadata accordingly.

[0009] Optionally, integrating the data attributes of the metadata also includes: if the time window interval expires, determining whether all window data has arrived based on the arrive time of the event arriving at the stream processor; if not all arrived, controlling the window timer to automatically extend.

[0010] Optionally, the method for data streaming aggregation also includes: checking whether a node commit value commit key is equal to a node registration value regist key, wherein the commit key is generated by submitting the integrated metadata to the Redis database, and the regist key is generated by registering the node in the Redis cache; if they are equal, obtaining all metadata of the Redis cache; merging all metadata to generate final metadata; and performing aggregation calculation on the final metadata for the data indicators to obtain an output result.

[0011] An embodiment of the present invention also provides a stream processing node, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for data stream aggregation according to any one of the above items.

[0012] An embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute the method for data streaming aggregation according to any one of the above items.

[0013] The present embodiment also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for data streaming aggregation according to any one of the above items.

[0014] An embodiment of the present invention further provides a system for data streaming aggregation, the system for data streaming aggregation comprising a streaming processing cluster and a Redis high-speed cache, wherein the streaming processing cluster comprises one or more of the above-mentioned streaming processing nodes. Optionally, the one or more streaming processing nodes are divided into: streaming processing intermediate nodes and streaming processing terminal nodes.

[0015] Through the above technical solution, the embodiment of the present invention solves the problem that the cluster requires a large amount of network IO overhead by abstracting the metadata and independently calculating the metadata by each node, greatly reducing the requirements of the streaming aggregation system for computing resources and reducing operating costs.

[0016] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0018] Figure 1 It is a schematic diagram of a method flow for data stream aggregation provided by an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the system structure for data streaming aggregation provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of an example process provided by an embodiment of the present invention.

[0021] Description of Reference Numerals

[0022] 10 Stream Processing Cluster 11 Stream Processing Node

[0023] 21 Redis Cache DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0026] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] Before explaining the embodiments of the present invention in detail, the defects of the prior art are summarized first, taking a common application monitoring system as an example, mainly focusing on the following aspects:

[0028] 1) When aggregating indicators according to time windows, the streaming data to be aggregated needs to be redistributed within the cluster according to tasks. This operation will occupy a large amount of network IO, as well as a large amount of CPU and memory. Therefore, the framework cluster has high requirements for host configuration and high operating costs.

[0029] 2) Due to the extra network transmission (actually: user state -> kernel state -> network card -> kernel state -> user state, a total of 4 copies), the time cost will increase significantly;

[0030] 3) In the framework cluster, tasks are assigned based on hash calculations, so the nodes in the cluster cannot be freely increased or decreased in a planned manner during operation;

[0031] 4) Watermark is not flexible enough. It can only be judged and aggregated according to a certain time attribute and cannot perform composite operations.

[0032] 5) The cluster framework has strict requirements on the host clock. The time difference cannot be too large, otherwise data errors will occur;

[0033] 6) Limitation of technology stack. Since most frameworks are mainly based on the Java technology stack, they are not friendly to other language technology stacks. In addition, Java's resource consumption is obviously high when it is static, so it is not suitable for application in lightweight scenarios.

[0034] Figure 1is a flow chart of a method for data stream aggregation provided by an embodiment of the present invention. It should be understood that Figure 1 The steps in the flowchart are not necessarily executed in the order indicated. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0035] Please refer to Figure 1 , the method for data streaming aggregation may include the following steps:

[0036] Step S110: Obtain the streaming data to be aggregated, and abstract the streaming data to be aggregated into required metadata.

[0037] Streaming aggregation can be understood as real-time processing of data streams, without waiting for all data to be acquired, to aggregate data, calculate and obtain multiple data indicators.

[0038] Preferably, step S110 may include: extracting data indicators for the type of the streaming data to be aggregated; abstracting data attributes of the metadata based on the extracted data indicators; and generating the metadata based on the data attributes.

[0039] To illustrate with an example, in application monitoring, for the data type of the streaming data to be aggregated, for example, seven data indicators, such as transaction volume, system success rate, business success rate, average response time, average processing time, long transaction rate, and Apdex, need to be aggregated and calculated. For these seven data indicators, abstract data attributes may include: transaction volume, long transaction volume, satisfactory transaction volume, tolerable transaction volume, number of system failures, number of business failures, total response time, and total processing time, which constitute the data attributes of the metadata. Among them, for some data indicators, one or some data attributes of the metadata may not be needed. When the metadata is integrated, the data attribute may be filled with zeros or averages, etc. The specific situation may depend on the actual application scenario and the streaming data to be aggregated, and the embodiments of the present invention do not limit this.

[0040] Step S120: When the time window is opened, the data attributes of the metadata are integrated.

[0041] Before step S120, the method for data streaming aggregation may further include: determining a time window interval according to the event time when the streaming task event occurs, wherein the time window interval is the time between the opening of the time window and the closing of the time window.

[0042] The streaming task event can be understood as a single piece of data to be processed. The event can include multiple attribute fields, including one or more time attributes, such as the event time when it occurs.

[0043] In the embodiment of the present invention, when processing data streams, within a time window, on a single computing node, there is one and only one piece of metadata, and the streaming data to be aggregated (original event data) is the accumulation of the metadata.

[0044] Step S120 may include: when the time window is opened, integrating the metadata may include: if the metadata is the first data in the time window interval, registering the node in the Redis cache; and opening the window timer, and accumulating the data attributes of the metadata accordingly.

[0045] In response to the aforementioned problems 1)-3), the large amount of data secondary distribution and network IO consumption problems encountered by the big data streaming processing framework (for example, the spark / flink technical framework), the embodiment of the present invention uses a third party to aggregate data. For example, in an application monitoring project, the embodiment of the present invention can preferably use the hash type of the Redis database as this third-party Redis cache. Redis supports atomic operations and multiple data types.

[0046] For example, when a single node time window is opened, the node running in the time window needs to be registered in the Redis cache (obtaining the node registration value regist key), which can be operated through the Redis Incr command, which is an atomic accumulation of a single key. After registration, each node performs metadata accumulation calculation within the node for the received metadata (accumulation is performed for the data attributes of each metadata), and no longer communicates with the outside world.

[0047] Following the above application monitoring example, the data attributes of the metadata can be integrated as shown in Table 1:

[0048]

[0049]

[0050] Table 1

[0051] Preferably, the step S120 further includes: if the time window interval expires, determining whether all window data has arrived according to the arrive time of the event arriving at the stream processor; if not all arrived, controlling the window timer to automatically extend.

[0052] The arrive time is also the time attribute of the event.

[0053] In response to the above problem 4), the embodiment of the present invention preferably uses two event time attributes to calculate the watermark, uses the event time to calculate the data shard to which the time window belongs, and uses the arrival time to determine the delay of the event data to determine whether the data window can trigger submission. Based on this, the aggregated data can be made more accurate, and the aggregated data will not change due to unexpected blocking. At the same time, it can also accurately determine whether the data is delayed, which is not affected by the host clock. There is no mandatory requirement for the host clock time difference of the nodes in the cluster.

[0054] Step S130: When the time window is closed, submit the integrated metadata to the Redis cache.

[0055] The integrated metadata can be submitted to the Redis database (Redis cache) through the commit command. The commit command is used to save the changes made by the transaction to the database, and save all transactions after the last commit or rollback command to the database.

[0056] Taking an example, when the time window in the node ends, the node submits metadata to the Redis cache for caching, and performs a commit operation on the node. The Redis Incr command can be used to operate the node commit value commitkey.

[0057] Preferably, the method for data streaming aggregation also includes: checking whether a node commit value commit key is equal to a node registration value regist key, wherein the commit key is generated by submitting the integrated metadata to the Redis database, and the regist key is generated by registering the node in the Redis cache; if they are equal, obtaining all metadata of the Redis cache; merging all metadata to generate final metadata; and performing aggregation calculation on the final metadata for the data indicators to obtain an output result.

[0058] Continuing with the above example, after commit is completed, each node will check whether the value of the current commit key is equal to the value of the regist key. If they are not equal, the node will "realize" that it is not the final node and only caches data. If they are equal, the node determines that it is the final execution node. The node will retrieve all the cached data stored by all nodes at once, merge the data, generate the final metadata, and then use the metadata to calculate the data indicators to be calculated (for example, the 7 indicators mentioned above), and output the results and store them in the database.

[0059] Following the above application monitoring example, for the data indicators to be processed, the output results can be shown in Table 2:

[0060] Data indicators Calculation formula of data indicators Trading Volume Total metadata transaction volume System success rate (Transaction volume - Number of system failures)*100 / Transaction volume Business success rate (Transaction volume - Number of failed transactions)*100 / Transaction volume Average response time Total response time / transaction volume Average processing time Total processing time / transaction volume Long Transaction Rate Long transaction volume*100 / transaction volume APDEX (Satisfactory trading volume + tolerable trading volume / 2) / trading volume

[0061] Table 2

[0062] Accordingly, the embodiments of the present invention solve the problem that the cluster requires a large amount of network IO overhead by abstracting the metadata and independently calculating the metadata by each node, which greatly reduces the requirements of the streaming aggregation system for computing resources and reduces operating costs.

[0063] An embodiment of the present invention also provides a stream processing node, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for data stream aggregation according to steps S110-S130.

[0064] An embodiment of the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute the method for data streaming aggregation described in steps S110-S130.

[0065] The present embodiment also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for data streaming aggregation described in steps S110-S130.

[0066] Figure 2 Schematic diagram of the system structure for data stream aggregation provided by an embodiment of the present invention. Those skilled in the art can understand that Figure 2 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] Please refer to Figure 2The system for data streaming aggregation includes a streaming processing cluster 10 and a Redis cache 21, wherein the streaming processing cluster 10 includes one or more of the above-mentioned streaming processing nodes 11.

[0068] Regarding the above problem 5), since the cluster nodes of the streaming processing cluster 10 in the embodiment of the present invention perform independent registration and submission operations for each window period, there is no specific task node, and the final execution node will be determined only after submission. Therefore, the cluster can arbitrarily increase or decrease nodes when running, realize flexible use of resources, and data security is more guaranteed.

[0069] In response to the above problem 6), the computing cluster concept of the embodiment of the present invention does not need to use a specific framework, can remove the restriction on a specific language technology stack, and is no longer limited to the JAVA technology stack. Preferably, the one or more stream processing nodes 11 are divided into: a stream processing intermediate node and a stream processing terminal node.

[0070] Figure 3 Please refer to the example flow chart provided by the embodiment of the present invention. Figure 2 and Figure 3 The process of the system for data streaming aggregation provided by an embodiment of the present invention is explained by an example, which may include the following steps:

[0071] Step S10: the stream processing node 11 obtains the stream data to be aggregated.

[0072] Step S11, abstracting the streaming data to be aggregated into metadata. Please refer to the above for the abstraction process, which will not be repeated here.

[0073] Step S12, determining whether the abstracted metadata is abnormal.

[0074] Step S13: If the metadata is abnormal, the metadata is discarded.

[0075] Step S14: If the metadata is normal, determine the time window interval according to the event time.

[0076] Step S15, determining whether the metadata is the first data in the time window interval.

[0077] Step S16 / Step S17, if yes, submit redis registration; open the window timer.

[0078] Step S18: if not, perform metadata accumulation.

[0079] Step S19: If the window timer expires, execute step S20.

[0080] Step S20, determining whether all window data has arrived according to the arrival time.

[0081] Step S21: If not all of them have arrived, the window timer can be automatically extended and step S19 is executed.

[0082] Step S22: If all are reached, submit to redis cache.

[0083] Step S23, determining whether the current stream processing node 11 is a stream processing terminal node.

[0084] Step S24: If it is not the end node of the streaming processing, the current window process ends.

[0085] Step S25: If it is a streaming processing terminal node, merge the data to generate final metadata.

[0086] Step S26, use the metadata to calculate the data indicators to be calculated, and output the results and store them in the database.

[0087] Accordingly, the embodiments of the present invention, through the abstract processing of metadata and the method of independently calculating metadata by each node, can at least achieve the following technical effects: 1) Solving the problem that the cluster requires a large amount of network IO overhead, greatly reducing the requirements of the streaming aggregation system for computing resources, and reducing operating costs; 2) Eliminating the requirements of the streaming aggregation cluster for the host clock, reducing the occurrence rate of failures during production operation; 3) Cluster nodes can be freely increased or decreased during operation, reducing the pressure of production operation and maintenance, and there is no need to perform subsequent number-filling operations, making production operation more stable; 4) The technology stack is more flexible, and the technology stack can be selected more freely during development.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0093] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0094] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0095] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0096] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for data streaming aggregation, characterized in that: The method for data streaming aggregation comprises: Obtain the streaming data to be aggregated, and abstract the streaming data to be aggregated into required metadata; When the time window opens, integrating the data attributes of the metadata; and When the time window is closed, the integrated metadata is submitted to the Redis cache. The step of abstracting the streaming data to be aggregated into required metadata includes: Extracting data indicators according to the type of the streaming data to be aggregated; Abstracting data attributes of the metadata based on the extracted data indicators; generating the metadata according to the data attributes, Before integrating the metadata when the time window is opened, the method for data streaming aggregation further includes: Determine a time window interval according to the event time when the streaming task event occurs, wherein the time window interval is the time between the opening of the time window and the closing of the time window. When the time window is opened, integrate the data attributes of the metadata including: If the metadata is the first data in the time window interval, register the node in the Redis cache; and Opening a window timer and accumulating the data attributes of the metadata accordingly, wherein when the time window is opened, integrating the data attributes of the metadata further comprises: If the time window interval expires, determine whether all window data has arrived based on the arrival time of the event at the stream processor; If not all of them have arrived, the window timer is controlled to automatically extend. The method for data streaming aggregation also includes: Verify whether the node commit key and the node registration key are equal, wherein the commit key is generated by submitting the integrated metadata to the Redis database, and the regist key is generated by registering the node in the Redis cache; If they are equal, obtain all metadata of the Redis cache; Merging all the metadata to generate final metadata; and For the data indicators, the final metadata is aggregated and calculated to obtain an output result.

2. A stream processing node, characterized in that: The stream processing node comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for data stream aggregation according to claim 1.

3. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which enable the machine to execute the method for data streaming aggregation according to claim 1.

4. A computer program product, comprising a computer program, characterized in that The computer program implements the method for data streaming aggregation according to claim 1 when executed by a processor.

5. A system for data streaming aggregation, characterized in that: The system for data streaming aggregation includes a streaming processing cluster and a Redis cache, wherein the streaming processing cluster includes one or more streaming processing nodes as described in claim 2.

6. The system for data streaming aggregation according to claim 5, characterized in that: The one or more stream processing nodes are divided into: stream processing intermediate nodes and stream processing terminal nodes.

Images