Streaming data processing method and system based on column-stored data

Abstract

The present invention proposes a streaming data processing method and system based on column-stored data, including: obtaining streaming data to be processed and corresponding processing tasks, dividing the streaming data into batch data blocks based on the time dimension, each data in the batch data block contains the timestamp of the window to which it belongs; compressing the timestamp according to the time type of the timestamp, assigning a window number to each data in the batch data block according to the compression result, dividing the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window number, performing pre-aggregation calculation on the data of each intermediate data block according to the processing task, and generating a pre-aggregation intermediate state; according to a preset streaming data time processing mode, extracting the pre-aggregation intermediate state of the corresponding window number from the internal storage and executing the corresponding processing task, and outputting the task processing result of each window number as the streaming data processing result.

Description

Technical Field

[0001] The present invention relates to the field of distributed streaming data computing, and in particular to a streaming data processing method and system based on column-stored data. Background Art

[0002] Streaming data computing engines are gradually emerging and penetrating into various industries. Currently, almost all cloud service providers provide streaming data computing engines, which can be used for scenarios such as data aggregation, data association, data monitoring, and data analysis. The current mainstream streaming data computing engines are represented by systems such as Apache Flink, Apache Spark Streaming, and Storm. They use directed acyclic graphs to represent user jobs, and their programming model is more flexible than MapReduce. Contemporary streaming data computing engines use window technology to achieve data aggregation in the time dimension, and support out-of-order message processing through event messages.

[0003] Introduction to window technology. The streaming data computing engine uses window technology to aggregate data in the time dimension. Common windows include rolling and sliding windows. A rolling time window is also called a fixed time window, which aggregates data at fixed time intervals, such as summarizing data every day. A sliding time window is also called a jump window, which defines a window with a fixed size that slides at fixed time intervals. For example, it can be used to generate a statistical data table for the past week every day. When the time window size and the sliding interval are equal, the sliding time window degenerates into a rolling time window. When the sliding interval is smaller than the time window size, the sliding windows overlap, and a record may belong to multiple different windows. The essence of window operation is the aggregation operation of data in the time dimension.

[0004] Introduction to time semantics. The streaming data computing engine processes data in the time dimension, and usually supports two types of time semantics: processing time and event time. Processing time is the time when the message enters the computing engine. The data is bound to an increasing timestamp in the order in which it enters the system. Since the physical time of the machine is used under the processing time semantics, the window can be triggered according to the machine time. In this mode, data processing is relatively simple. Event time refers to the time when the data actually occurs. However, after the data is generated, it may arrive at the server out of order due to network instability or other reasons, or it may not arrive at the server due to network failure. Therefore, in the event time processing mode, the system cannot use machine time to determine whether all the data in the window has arrived. One of the current mainstream practices is to use the watermark mechanism to determine whether the data has arrived. The watermark is a flag bit estimated by the system using a specific algorithm to mark that all the data in a window has arrived at that moment. It is usually derived using a heuristic algorithm. However, since the data is unknown, the system cannot predict the exact watermark position, so late data may still arrive after the watermark, and this part of the data is processed separately. Since data may arrive delayed by several hours or even days, and data discard is unacceptable in some fields such as finance, how to cache a large amount of window data in the late data processing scenario is a major challenge for streaming data computing systems.

[0005] Introduction to storage and computing modes. The storage and computing modes of the streaming computing engine are divided into two categories: row-based storage and column-based storage. The row storage mode means that the system stores data and performs calculations in units of rows in the data table. Figure 1 As shown in the figure. The row-based storage mode is a very intuitive storage mode, and its storage mode is similar to the table storage mode that humans are accustomed to. Its advantage is that it can efficiently operate on various data attributes of the same record and is friendly to transaction operations. However, since the row-based storage mode needs to read all the data of each record by row when reading data, if a query only needs to use some attributes in the data record, this mode will cause irrelevant read and write overhead. When there are many data record attributes, this overhead may seriously affect system performance. In addition, in scenarios where data aggregation is required for the entire data set according to a certain attribute, since the row-based storage mode needs to read all the data in the data record, it is not memory-friendly and results in poor performance. Mainstream streaming data computing engines such as Apache Flink and Apache Spark Streaming use the row-based storage mode, which brings lower latency to the system in scenarios such as data cleaning, filtering, and conversion.

[0006] Column storage mode means that the system maintains data records and performs calculations according to the columns of the data table. Each column of the data table represents an attribute of the data record, and all data records are sorted by attribute and stored in memory, such as Figure 2As shown. Its storage mode is not as intuitive as the row storage mode. The column storage mode was born to solve data analysis scenarios. Since the attributes of each data record are not stored continuously, the operation speed of a single data record is slower than the row storage mode, and it is not friendly to transaction operations. However, since the column storage mode can only retrieve specified data attributes without reading all data, it can greatly reduce data reading and writing overhead in scenarios where data filtering is required. At the same time, it is memory-friendly in data aggregation scenarios. Therefore, the column storage mode has its unique advantages in data analysis scenarios. The column storage mode is widely used in data analysis engines, such as HBase, ClickHouse, etc.

[0007] (1) Memory cache cannot store very large window data

[0008] In streaming data computing scenarios, data may arrive late due to unpredictable reasons such as network failure or device offline, resulting in a large delay between data generation and processing in data analysis tasks, which may be several hours or days. For example, a user generates a log before the plane takes off, and the log may not arrive at the server until several hours or a day later. The current mainstream streaming data computing engines use window technology to cache data. Users often need to set a relatively large window to aggregate all late data, but a larger window interval also increases system processing latency. Modern streaming data computing engines such as Google MillWheel, Apache Flink, or Spark Streaming use a watermark mechanism to define the lower bound of the time window. Reaching the watermark means that all data in the current time window has arrived at the system. Unfortunately, for a real application scenario, since the data stream is unknown, the system cannot have enough knowledge to establish a 100% accurate watermark, and data arriving after the watermark is still at risk of being lost. In addition, a larger window interval also brings greater memory pressure. Mainstream streaming data computing engines use memory to cache window data, and limited memory cannot cache infinite data streams. Summary of the invention

[0009] The purpose of the present invention is to improve the computing efficiency and window data caching capability of a streaming data computing system, and proposes a design method for a dedicated storage engine for streaming data based on column storage.

[0010] Specifically, the present invention proposes a streaming data processing method based on column-stored data, which includes:

[0011] Step 1: Obtain the streaming data to be processed and its corresponding processing tasks, and divide the streaming data into batch data blocks based on the time dimension. Each data in the batch data block contains the timestamp of the window to which it belongs;

[0012] Step 2: compress the timestamp according to the time type of the timestamp, assign a window sequence number to each data in the batch data block according to the compression result, divide the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window sequence number, and perform pre-aggregation calculation on the data of each intermediate data block according to the processing task to generate a pre-aggregation intermediate state;

[0013] Step 3: According to the preset streaming data time processing mode, extract the pre-aggregated intermediate state of the corresponding window number from the internal storage and execute the corresponding processing task, and output the task processing result of each window number as the streaming data processing result.

[0014] The streaming data processing method based on column-stored data, wherein step 2 comprises:

[0015] Determine whether the time type of the timestamp is seconds. If so, use t / w as the compression result;

[0016] Otherwise, determine whether the time type of the timestamp is minutes. If so, use t / 60 / w as the compression result.

[0017] Otherwise, determine whether the time type of the timestamp is hour. If so, use t / 3600 / w as the compression result;

[0018] Otherwise, determine whether the time type of the timestamp is day. If so, use (t+28800) / 86400 / w as the compression result;

[0019] Otherwise, determine whether the time type of the timestamp is week. If so, use (d–4) / 7 / w as the compression result.

[0020] Otherwise, determine whether the time type of the timestamp is month. If so, use to_month(d) / w as the compression result;

[0021] Otherwise, determine whether the time type of the timestamp is quarterly. If so, use to_month(d) / 3 / w as the compression result;

[0022] Otherwise, whether the time type of the timestamp is year, to_yead(d) / w is used as the compression result;

[0023] Where t is the window timestamp, w is the window size, and d is the number of days from the window timestamp to 1970-01-01.

[0024] The streaming data processing method based on column-stored data, wherein step 2 includes: when the window sequence number of the intermediate data block is greater than the watermark, the intermediate data block belongs to untriggered active data, and the intermediate data block is stored in memory, and the process includes:

[0025] Step 21, traverse the memory linked list to determine whether the window number has a corresponding linked list node. If so, execute step 22. Otherwise, add linked list nodes at the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number, and execute step 22.

[0026] Step 22: Use a tree structure to store pre-aggregation intermediate states with the same window sequence number, use the linked list node corresponding to the window sequence number as the root node, and append the pre-aggregation intermediate state to the leaf node of the tree;

[0027] Step 23: Periodically merge the tree structure, find the leaf node with the largest depth each time, and merge it with all sibling nodes and the parent node;

[0028] Step 24: Repeat step 23 until only one node is left in the tree structure except the root node. The root node represents the window number of the tree, and the other nodes are pre-aggregation intermediate states.

[0029] The streaming data processing method based on column-stored data, wherein step 2 includes: when the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum lateness allowed interval, the intermediate data block belongs to the triggered late data, and the intermediate data block is stored on a disk, and the process includes:

[0030] Step 25: Find the pre-aggregated intermediate state of the window sequence number corresponding to the late data from the disk and load it into the memory, and mark the corresponding intermediate state data in the disk as a state to be deleted;

[0031] Step 26: merge the pre-aggregation intermediate state obtained in step 25 with the late data to generate a new pre-aggregation intermediate state;

[0032] Step 27: Delete the pre-aggregation intermediate state marked as a to-be-deleted state in the disk, and append the new pre-aggregation intermediate state generated in step 26 to the disk.

[0033] The streaming data processing method based on column-stored data, wherein the streaming data is physiological data, image data or log text data collected by a sensor in real time; and the processing task corresponding to the streaming data is a database statistical task.

[0034] The present invention also proposes a streaming data processing system based on column-stored data, which includes:

[0035] The segmentation module is used to obtain the streaming data to be processed and its corresponding processing tasks, and segment the streaming data into batch data blocks based on the time dimension. Each data in the batch data block contains the timestamp of the window to which it belongs;

[0036] A compression module is used to compress the timestamp according to the time type of the timestamp, assign a window sequence number to each data in the batch data block according to the compression result, divide the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window sequence number, and perform pre-aggregation calculation on the data of each intermediate data block according to the processing task to generate a pre-aggregation intermediate state;

[0037] The output module is used to extract the pre-aggregated intermediate state of the corresponding window number from the internal storage according to the preset streaming data time processing mode and execute the corresponding processing tasks, and output the task processing results of each window number as the streaming data processing results.

[0038] The said streaming data processing system based on column-stored data, wherein the compression module is used to:

[0039] Determine whether the time type of the timestamp is seconds. If so, use t / w as the compression result;

[0040] Otherwise, determine whether the time type of the timestamp is minutes. If so, use t / 60 / w as the compression result.

[0041] Otherwise, determine whether the time type of the timestamp is hour. If so, use t / 3600 / w as the compression result;

[0042] Otherwise, determine whether the time type of the timestamp is day. If so, use (t+28800) / 86400 / w as the compression result;

[0043] Otherwise, determine whether the time type of the timestamp is week. If so, use (d–4) / 7 / w as the compression result.

[0044] Otherwise, determine whether the time type of the timestamp is month. If so, use to_month(d) / w as the compression result;

[0045] Otherwise, determine whether the time type of the timestamp is quarterly. If so, use to_month(d) / 3 / w as the compression result;

[0046] Otherwise, whether the time type of the timestamp is year, to_yead(d) / w is used as the compression result;

[0047] Where t is the window timestamp, w is the window size, and d is the number of days from the window timestamp to 1970-01-01.

[0048] The said streaming data processing system based on column-stored data, wherein the compression module is used for storing the intermediate data block in memory when the window sequence number of the intermediate data block is greater than the watermark, and the intermediate data block belongs to untriggered active data;

[0049] The compression module includes:

[0050] Module 21 is used to traverse the memory linked list to determine whether the window number has a corresponding linked list node. If so, module 22 is executed. Otherwise, linked list nodes are added to the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number, and module 22 is called.

[0051] Module 22, used to store pre-aggregation intermediate states with the same window sequence number using a tree structure, with the linked list node corresponding to the window sequence number as the root node, and append the pre-aggregation intermediate state to the leaf node of the tree;

[0052] Module 23 is used to periodically merge the tree structure, and each time a merge is performed, a leaf node with the largest depth is searched, and the leaf node is merged with all sibling nodes and the parent node;

[0053] Module 24 is used to repeatedly call module 23 until only one node is left in the tree structure except the root node. The root node represents the window number of the tree, and the other nodes are pre-aggregation intermediate states.

[0054] The said streaming data processing system based on column-stored data, wherein the compression module is used for storing the intermediate data block on disk when the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum lateness allowed interval, the intermediate data block belongs to the triggered late data;

[0055] The compression module includes:

[0056] Module 25, used to search the pre-aggregated intermediate state of the window sequence number corresponding to the late data from the disk and load it into the memory, and mark the corresponding intermediate state data as a state to be deleted in the disk;

[0057] Module 26, used to merge the pre-aggregation intermediate state obtained by module 25 with the late data to generate a new pre-aggregation intermediate state;

[0058] Module 27 is used to delete the pre-aggregation intermediate state marked as a to-be-deleted state in the disk, and append the new pre-aggregation intermediate state generated by module 26 to the disk.

[0059] The streaming data processing system based on column-stored data, wherein the streaming data is physiological data, image data or log text data collected by sensors in real time; the processing task corresponding to the streaming data is a database statistical task.

[0060] It can be seen from the above scheme that the advantages of the present invention are:

[0061] This invention proposes a column-based streaming data storage engine design method. Compared with the prior art, this engine solves the contradiction between the limited storage capacity of traditional memory and the high latency of disk storage through a memory + disk secondary storage structure, and supports caching of ultra-large windows. This engine reduces the latency of window calculation output by storing pre-aggregated intermediate states in memory and using background tasks to merge them regularly. In addition, the performance of window operation data aggregation is improved by using window ID compression technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the row storage mode;

[0063] Figure 2 This is a schematic diagram of the column storage mode;

[0064] Figure 3 It is a schematic diagram of the secondary storage structure;

[0065] Figure 4 This is a schematic diagram of the storage engine data structure;

[0066] Figure 5 Updates to the WindowView disk storage engine status. DETAILED DESCRIPTION

[0067] When the inventor was conducting research on streaming computing engines, he found that in many event time processing scenarios, user messages are seriously late. For example, in many scenarios with high data value such as finance and advertising, the window size is often set to several days to deal with late messages. The mainstream streaming data computing system is mainly based on memory storage, and caching large time window data puts a lot of pressure on the memory. In addition, the mainstream streaming data computing systems are all row storage computing and storage modes. The row storage mode processes single data as a unit, with low latency, but the window operation is a data aggregation operation, and the data aggregation operation performance of the row storage mode is not as good as the column storage mode. In the window computing mode, all window data must be in place before the calculation output can be performed, so the row storage mode cannot fully play the low latency advantage. After studying the prior art, the inventor found that the traditional storage engine operation mode is not fixed and needs to deal with random additions, deletions, modifications and queries, but the streaming data operation mode is fixed, such as the window size, sliding interval, etc., which are fixed when the streaming query is created. Therefore, the inventors proposed a column-based streaming data storage engine based on the characteristics of streaming data computing. This engine uses a memory + disk secondary storage structure to solve the problem that traditional memory storage engines are difficult to cache ultra-large window data while ensuring low-latency processing. By using a column-based storage structure, the window aggregation computing performance is improved. In addition, the processing delay of the column-based engine is further reduced through technologies such as window ID compression and window computing state pre-aggregation.

[0068] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0069] In streaming data processing, whether in event time or processing time mode, rolling windows and sliding windows are triggered sequentially. Therefore, the storage engine uses a disk + memory secondary storage structure, such as Figure 3 As shown in the figure, the memory is responsible for storing window data that has not been triggered. This part of data is updated frequently and requires higher IO performance. After the window is triggered, the update frequency of the window data becomes very low. Therefore, the system saves the triggered window data on the disk to deal with late data. When the disk cache data exceeds the maximum allowed late time, the cache data that exceeds the maximum allowed late time will be marked as data to be discarded and will be cleaned up regularly by the system. The storage engine data structure is as follows Figure 4 shown.

[0070] Process 1: After the data enters the storage engine, the storage engine first filters out the data whose timestamp is less than the watermark (m) - the maximum allowed late interval (t), for example Figure 4 The window containing 1234 in the data has timed out, so it is meaningless to process this data.

[0071] Process 2: Each piece of data entering the storage engine contains a timestamp used to mark the window (for example, a Unix timestamp, which is a number representing a specific date and time). The system compresses the timestamp and generates a continuous window number, as shown in Table 1, where t is the window timestamp, w is the window size, and d is the number of days from 1970-01-01 to the date of the window timestamp.

[0072] For example, the window start time is used as the timestamp, the window size is 2s, and the three window Unix timestamps are respectively: 1636012243, 1636012245, 1636012247. Through the compression of the present invention, they can be converted into continuous window numbers 1, 2, 3.

[0073] This step can bring two major performance improvements. First, the window sequence number occupies fewer bits than the 32-bit unsigned integer of the window timestamp, which means it is more cache-friendly and can bring more efficient aggregation performance. When the window sequence number has less than 16 bits, the storage engine uses a lookup table instead of a hash table for indexing, which can further improve aggregation performance; second, continuous window sequence numbers are more friendly to the storage engine and can reduce the number of times the storage engine searches for data when the window is triggered.

[0074] Table 1 Window sequence number generation method

[0075] Time Type Compressed size (bit) Window timestamp(s) SECOND 32 t / w MINUTE 27 t / 60 / w HOUR 21 t / 3600 / w DAY 16 (t+28800) / 86400 / w WEEK 14 (d–4) / 7 / w MONTH 12 to_month(d) / w QUARTER 10 to_month(d) / 3 / w YEAR 08 to_yead(d) / w

[0076] Process 3: The system classifies the data in units of the window numbers assigned in process 2, and performs pre-aggregation calculations on the data under each classification to generate pre-aggregation intermediate states. Multiple pre-aggregation intermediate states can be merged into one pre-aggregation intermediate state through a merge operation, or calculated as the final calculation result through a calculation output operation. The pre-aggregation operation can reduce the amount of calculation when the window is triggered. In this step, each pre-aggregation intermediate state corresponds to a window number.

[0077] Process 4: In the data generated by process 3, for the data whose window number is greater than the watermark (m), this part of the data belongs to the active window that has not been triggered. The processing process is as follows. The watermark is the maximum time observed by the system. For example, if the arrival data sequence is 1, 2, 5, 3, 4, 6, then the corresponding current watermark is 1, 2, 5, 5, 5, 6.

[0078] Process 4.1: The system uses a linked list structure in memory to index data. The linked list consists of continuously increasing nodes, and the value of the node represents the window number. The system traverses the linked list and searches for the node corresponding to the window number. If the maximum node number is less than the window number, the linked list nodes are added to the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number.

[0079] Process 4.2: The system uses a tree structure to store pre-aggregated intermediate states with the same window sequence number, with the linked list node found in process 4.1 as the root node of the tree, and appends the pre-aggregated intermediate state to the leaf node of the tree.

[0080] Process 4.3: The system periodically merges the tree structures generated by process 4.2. Each time a merge occurs, it searches for the leaf node with the largest depth and merges it with all sibling nodes and the parent node.

[0081] Process 4.4: Repeat process 4.3 until there is only one node left in the tree structure except the root node.

[0082] Process 5: In the data generated by process 3, for data whose window number is less than or equal to the waterline (m) but greater than wt, this part of the data is late data, and the processing process is as follows:

[0083] Process 5.1: Find the pre-aggregated intermediate state of the corresponding window sequence number from the disk (log storage) and load it into the memory, and mark the corresponding intermediate state data as "to be deleted" on the disk.

[0084] Process 5.2: Merge the pre-aggregated intermediate state obtained in process 5.1 with the newly inserted data to generate a new pre-aggregated intermediate state, such as Figure 5 shown.

[0085] Process 5.3: Append the pre-aggregated intermediate state generated by process 5.2 to the end of the disk (log storage) queue.

[0086] Process 5.4: Calculate the pre-aggregated intermediate state generated in process 5.2 as the final calculation state and output it.

[0087] Process 6: In the data generated in process 3, for the data whose window sequence number is less than or equal to wt, this part of the data has exceeded the maximum allowable late arrival interval, so it is directly discarded.

[0088] Process 7: The system provides a "window trigger (window number)" operation. The parameter of the operation is "window number" wid. When the operation is called, the system automatically triggers all windows that have not been triggered and whose window numbers are less than wid. The window trigger operation is called by the calculation engine when a trigger is needed. The process is described as follows:

[0089] Process 7.1: Since the rolling window and sliding window triggers under processing time and event events are calculated based on the window number increment, the system uses a sliding window on the memory linked list to extract data, extracting a pre-aggregated intermediate state tree of a window number each time, assuming that the window number corresponding to the current sliding window is wid_now.

[0090] Process 7.2: Use the methods described in Processes 4.3 and 4.4 to merge the pre-aggregated intermediate state trees extracted in Process 7.1 into one node.

[0091] Process 7.3: Calculate the pre-aggregated intermediate state generated in process 7.3 as the final calculation state and output it.

[0092] Process 7.4: Write the pre-aggregated intermediate state generated in process 7.3 to the end of the disk (log storage) queue.

[0093] Procedure 7.5: Use the temporary variable n to point to the location of the sliding window.

[0094] Procedure 7.6: Slide the sliding window one unit in the direction of increasing window number.

[0095] Procedure 7.7: Delete the linked list node pointed to by the temporary variable n described in Procedure 7.5, as well as the memory of the tree structure pointed to by the linked list node.

[0096] Process 7.8: Mark the data blocks in the disk (log storage) with window numbers less than wid_now-t as "to be deleted".

[0097] Process 7.8: Repeat process 7.1 to process 7.7 until the window number pointed to by the sliding window is greater than wid.

[0098] Process 8: The system periodically scans the disk (log storage) and deletes data blocks marked as "pending deletion".

[0099] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0100] The present invention also proposes a streaming data processing system based on column-stored data, which includes:

[0101] The segmentation module is used to obtain the streaming data to be processed and its corresponding processing tasks, and segment the streaming data into batch data blocks based on the time dimension. Each data in the batch data block contains the timestamp of the window to which it belongs;

[0102] A compression module is used to compress the timestamp according to the time type of the timestamp, assign a window sequence number to each data in the batch data block according to the compression result, divide the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window sequence number, and perform pre-aggregation calculation on the data of each intermediate data block according to the processing task to generate a pre-aggregation intermediate state;

[0103] The output module is used to extract the pre-aggregated intermediate state of the corresponding window number from the internal storage according to the preset streaming data time processing mode and execute the corresponding processing tasks, and output the task processing results of each window number as the streaming data processing results.

[0104] The said streaming data processing system based on column-stored data, wherein the compression module is used to:

[0105] Determine whether the time type of the timestamp is seconds. If so, use t / w as the compression result;

[0106] Otherwise, determine whether the time type of the timestamp is minutes. If so, use t / 60 / w as the compression result.

[0107] Otherwise, determine whether the time type of the timestamp is hour. If so, use t / 3600 / w as the compression result;

[0108] Otherwise, determine whether the time type of the timestamp is day. If so, use (t+28800) / 86400 / w as the compression result;

[0109] Otherwise, determine whether the time type of the timestamp is week. If so, use (d–4) / 7 / w as the compression result.

[0110] Otherwise, determine whether the time type of the timestamp is month. If so, use to_month(d) / w as the compression result;

[0111] Otherwise, determine whether the time type of the timestamp is quarterly. If so, use to_month(d) / 3 / w as the compression result;

[0112] Otherwise, whether the time type of the timestamp is year, to_yead(d) / w is used as the compression result;

[0113] Where t is the window timestamp, w is the window size, and d is the number of days from the window timestamp to 1970-01-01.

[0114] The said streaming data processing system based on column-stored data, wherein the compression module is used for storing the intermediate data block in memory when the window sequence number of the intermediate data block is greater than the watermark, and the intermediate data block belongs to untriggered active data;

[0115] The compression module includes:

[0116] Module 21 is used to traverse the memory linked list to determine whether the window number has a corresponding linked list node. If so, module 22 is executed. Otherwise, linked list nodes are added to the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number, and module 22 is called.

[0117] Module 22, used to store pre-aggregation intermediate states with the same window sequence number using a tree structure, with the linked list node corresponding to the window sequence number as the root node, and append the pre-aggregation intermediate state to the leaf node of the tree;

[0118] Module 23 is used to periodically merge the tree structure, and each time a merge is performed, a leaf node with the largest depth is searched, and the leaf node is merged with all sibling nodes and the parent node;

[0119] Module 24 is used to repeatedly call module 23 until only one node is left in the tree structure except the root node. The root node represents the window number of the tree, and the other nodes are pre-aggregation intermediate states.

[0120] The said streaming data processing system based on column-stored data, wherein the compression module is used for storing the intermediate data block on disk when the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum lateness allowed interval, the intermediate data block belongs to the triggered late data;

[0121] The compression module includes:

[0122] Module 25, used to search the pre-aggregated intermediate state of the window sequence number corresponding to the late data from the disk and load it into the memory, and mark the corresponding intermediate state data as a state to be deleted in the disk;

[0123] Module 26, used to merge the pre-aggregation intermediate state obtained by module 25 with the late data to generate a new pre-aggregation intermediate state;

[0124] Module 27 is used to delete the pre-aggregation intermediate state marked as a to-be-deleted state in the disk, and append the new pre-aggregation intermediate state generated by module 26 to the disk.

[0125] The streaming data processing system based on column-stored data, wherein the streaming data is physiological data, image data or log text data collected by sensors in real time; the processing task corresponding to the streaming data is a database statistical task.

Claims

1. A streaming data processing method based on column-stored data, characterized in that: include: Step 1: Obtain the streaming data to be processed and its corresponding processing task, and divide the streaming data into batch data blocks based on the time dimension, where each data in the batch data block contains the timestamp of the window to which it belongs; the streaming data is physiological data, image data or log text data collected by the sensor in real time; the processing task corresponding to the streaming data is a database statistics task; Step 2: compress the timestamp according to the time type of the timestamp, assign a window sequence number to each data in the batch data block according to the compression result, divide the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window sequence number, and perform pre-aggregation calculation on the data of each intermediate data block according to the processing task to generate a pre-aggregation intermediate state; When the window sequence number of the intermediate data block is greater than the watermark, the intermediate data block belongs to untriggered active data, and the intermediate data block is stored in the memory; When the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum late arrival interval, the intermediate data block belongs to the triggered late arrival data, and the intermediate data block is stored on disk; Step 3: According to the preset streaming data time processing mode, extract the pre-aggregated intermediate state of the corresponding window number from the internal storage and execute the corresponding processing task, and output the task processing result of each window number as the streaming data processing result.

2. The method for processing streaming data based on column-stored data according to claim 1, characterized in that: This step 2 includes: Determine whether the time type of the timestamp is seconds. If so, use t / w as the compression result; Otherwise, determine whether the time type of the timestamp is minutes. If so, use t / 60 / w as the compression result. Otherwise, determine whether the time type of the timestamp is hour. If so, use t / 3600 / w as the compression result; Otherwise, determine whether the time type of the timestamp is day. If so, use (t+28800) / 86400 / w as the compression result; Otherwise, determine whether the time type of the timestamp is week. If so, use (d–4) / 7 / w as the compression result. Otherwise, determine whether the time type of the timestamp is month. If so, use to_month(d) / w as the compression result; Otherwise, determine whether the time type of the timestamp is quarterly. If so, use to_month(d) / 3 / w as the compression result. Otherwise, whether the time type of the timestamp is year, to_yead(d) / w is used as the compression result; Where t is the window timestamp, w is the window size, and d is the number of days between the window timestamp date and the preset date.

3. The method for processing streaming data based on column-stored data according to claim 1, characterized in that: When the window sequence number of the intermediate data block is greater than the watermark, step 2 includes: Step 21, traverse the memory linked list to determine whether the window number has a corresponding linked list node. If so, execute step 22. Otherwise, add linked list nodes at the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number, and execute step 22. Step 22: Use a tree structure to store pre-aggregation intermediate states with the same window sequence number, use the linked list node corresponding to the window sequence number as the root node, and append the pre-aggregation intermediate state to the leaf node of the tree; Step 23: Periodically merge the tree structure, find the leaf node with the largest depth each time, and merge it with all sibling nodes and the parent node; Step 24: Repeat step 23 until only one node is left in the tree structure except the root node. The root node represents the window number of the tree, and the other nodes are pre-aggregation intermediate states.

4. The method for processing streaming data based on column-stored data according to claim 1, characterized in that: When the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum late arrival interval, step 2 includes: Step 25: Find the pre-aggregated intermediate state of the window sequence number corresponding to the late data from the disk and load it into the memory, and mark the corresponding intermediate state data in the disk as a state to be deleted; Step 26: merge the pre-aggregation intermediate state obtained in step 25 with the late data to generate a new pre-aggregation intermediate state; Step 27: Delete the pre-aggregation intermediate state marked as a to-be-deleted state in the disk, and append the new pre-aggregation intermediate state generated in step 26 to the disk.

5. A streaming data processing system based on column-stored data, characterized in that: include: The segmentation module is used to obtain the streaming data to be processed and its corresponding processing tasks, and segment the streaming data into batch data blocks based on the time dimension. Each data in the batch data block contains the timestamp of the window to which it belongs; The streaming data is physiological data, image data or log text data collected by the sensor in real time; the processing task corresponding to the streaming data is a database statistics task; A compression module is used to compress the timestamp according to the time type of the timestamp, assign a window sequence number to each data in the batch data block according to the compression result, divide the batch data block into multiple intermediate data blocks, each intermediate data block only contains data with the same window sequence number, and perform pre-aggregation calculation on the data of each intermediate data block according to the processing task to generate a pre-aggregation intermediate state; When the window sequence number of the intermediate data block is greater than the watermark, the intermediate data block belongs to untriggered active data, and the intermediate data block is stored in the memory; When the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum late arrival interval, the intermediate data block belongs to the triggered late arrival data, and the intermediate data block is stored on disk; The output module is used to extract the pre-aggregated intermediate state of the corresponding window number from the internal storage according to the preset streaming data time processing mode and execute the corresponding processing tasks, and output the task processing results of each window number as the streaming data processing results.

6. The stream data processing system based on column-stored data according to claim 5, characterized in that: The compression module is used to Determine whether the time type of the timestamp is seconds. If so, use t / w as the compression result; Otherwise, determine whether the time type of the timestamp is minutes. If so, use t / 60 / w as the compression result. Otherwise, determine whether the time type of the timestamp is hour. If so, use t / 3600 / w as the compression result; Otherwise, determine whether the time type of the timestamp is day. If so, use (t+28800) / 86400 / w as the compression result; Otherwise, determine whether the time type of the timestamp is week. If so, use (d–4) / 7 / w as the compression result. Otherwise, determine whether the time type of the timestamp is month. If so, use to_month(d) / w as the compression result; Otherwise, determine whether the time type of the timestamp is quarterly. If so, use to_month(d) / 3 / w as the compression result. Otherwise, whether the time type of the timestamp is year, to_yead(d) / w is used as the compression result; Where t is the window timestamp, w is the window size, and d is the number of days between the window timestamp date and the preset date.

7. The stream data processing system based on column-stored data according to claim 5, characterized in that: The compression module is used for storing the intermediate data block in memory when the window sequence number of the intermediate data block is greater than the watermark, and the intermediate data block belongs to untriggered active data; The compression module includes: Module 21 is used to traverse the memory linked list to determine whether the window number has a corresponding linked list node. If so, module 22 is executed. Otherwise, linked list nodes are added to the end of the linked list in sequence, and each node number is one greater than the previous node number until the linked list number is equal to the window number, and module 22 is called. Module 22, used to store pre-aggregation intermediate states with the same window sequence number using a tree structure, with the linked list node corresponding to the window sequence number as the root node, and append the pre-aggregation intermediate state to the leaf node of the tree; Module 23 is used to periodically merge the tree structure, and each time a merge is performed, a leaf node with the largest depth is searched, and the leaf node is merged with all sibling nodes and the parent node; Module 24 is used to repeatedly call module 23 until only one node is left in the tree structure except the root node. The root node represents the window number of the tree, and the other nodes are pre-aggregation intermediate states.

8. The stream data processing system based on column-stored data according to claim 5, characterized in that: The compression module is used for storing the intermediate data block on a disk when the window sequence number of the intermediate data block is less than the watermark but greater than the watermark minus the maximum late arrival allowable interval, the intermediate data block belongs to the triggered late arrival data; The compression module includes: Module 25, used to search the pre-aggregated intermediate state of the window sequence number corresponding to the late data from the disk and load it into the memory, and mark the corresponding intermediate state data as a state to be deleted in the disk; Module 26, used to merge the pre-aggregation intermediate state obtained by module 25 with the late data to generate a new pre-aggregation intermediate state; Module 27 is used to delete the pre-aggregation intermediate state marked as a to-be-deleted state in the disk, and append the new pre-aggregation intermediate state generated by module 26 to the disk.

Images