Streaming computing method and device for distributed database

Abstract

The invention relates to the technical field of databases, and particularly provides a streaming computing method and device for a distributed database, and the method comprises the following steps: S1, receiving time sequence data change captured by a CDC component in real time, and storing the time sequence data change in a local cache; s2, performing stream aggregation calculation of stream calculation; s3, receiving a calculation result of stream calculation, converting a result set into a target table compatible format, and writing data into a target table by using an internal actuator; s4, processing existing historical data during stream calculation creation, breakpoint data generated during a stream calculation pause period and expired data exceeding a time sequence data out-of-order window; and S5, managing the life cycle of the flow calculation in the distributed database cluster. Compared with the prior art, the method has the advantages that the streaming calculation of the time series data of the database can be efficiently completed, the recalculation of the window split data is efficiently completed by using the streaming calculation recalculation module, and the performance attenuation is within 15% of the original storage performance.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and specifically provides a streaming computing method and device for a distributed database. Background Art

[0002] With the development of IoT technology, the number of IoT devices has grown rapidly, and their application scenarios have become increasingly complex, placing higher demands on the storage and query efficiency of their massive data. Compared with the data processed by traditional databases, the data generated by IoT devices has obvious time series data processing characteristics, such as:

[0003] Data always contains a timestamp (data generation time / data storage time) and is generated at a high frequency and fixed time intervals in chronological order;

[0004] The amount of data is enormous. For example, a sensor generates approximately 1.4M data per day. A factory with tens of millions of sensors can generate 500TB of data per year. In emerging industries like the Internet of Vehicles and digital energy, managing hundreds of millions or even billions of devices is becoming a trend.

[0005] Query filter conditions generally include timestamp columns, and aggregate analysis is usually performed on the device and time dimensions.

[0006] Traditional database processing technologies struggle to efficiently store and analyze such large amounts of data. To improve the storage efficiency of IoT device data, two common approaches are used: lossless data compression and lossy downsampling. Users can choose one or both based on their application scenario. Furthermore, to improve the efficiency of analyzing massive amounts of IoT device data, especially costly aggregate queries, typical database systems employ pre-computation techniques. These pre-run / calculate important, repetitive queries and store the query results in a new database table. Subsequent queries can then quickly retrieve the aggregated results from this pre-computation table.

[0007] The current mainstream solutions are continuous query and streaming computing.

[0008] Continuous queries submit ordinary queries at fixed time intervals, resulting in a large amount of repeated scanning and calculations, low efficiency, and inability to process incoming data in real time;

[0009] Although existing stream computing can process incoming data in real time, it cannot efficiently handle the interruption and splitting of aggregation windows (when creating or starting stream computing, if there is historical data, there may be incomplete aggregation windows at the start and end positions of the aggregated data, called split windows. That is, the end time of the breakpoint data is in the middle of the stream computing aggregation window, and the aggregation results of the window need to be recalculated). Summary of the Invention

[0010] The present invention aims to address the above-mentioned deficiencies in the prior art and provides a highly practical streaming computing method for distributed databases.

[0011] A further technical task of the present invention is to provide a stream computing device for distributed databases that is rationally designed, safe and applicable.

[0012] The technical solution adopted by the present invention to solve its technical problem is:

[0013] A stream computing method for a distributed database has the following steps:

[0014] S1. The StreamCompute Reader module receives the time series data changes captured by the CDC component in real time and saves them to the local cache.

[0015] S2, the StreamCompute Aggregator module performs stream computing aggregation calculations;

[0016] S3: The StreamCompute Receive module receives the calculation results of StreamCompute, converts the result set into a format compatible with the target table, and then uses the internal executor to write the data to the target table.

[0017] S4. The stream computing recalculation module needs to process the historical data that existed when the stream computing was created, the breakpoint data generated during the stream computing pause, and the expired data that exceeds the time series data disorder window;

[0018] S5. The stream computing task management module manages the life cycle of stream computing in the distributed database cluster.

[0019] Furthermore, in step S1, every time data is updated, the StreamCompute Reader module checks whether there is data in the local cache that exceeds the out-of-order window of the time series data;

[0020] If there is, this part of the data is sorted according to the timestamp and Group BY column and returned to the upper-level operator. At the same time, the StreamCompute Reader module also uses a scheduled task to periodically check and output the time series data in the cache that is larger than the out-of-order window.

[0021] Furthermore, in step S2, the database system aggregation operator needs to read all the data before completing the aggregation calculation. The stream computing Aggregator module introduces an internal stream computing mechanism to aggregate time series data in real time according to user-defined rules.

[0022] At the same time, the StreamCompute Aggregator module supports a forced closing mechanism for aggregation windows.

[0023] Furthermore, in step S3, the StreamCompute Receiver module uses a scheduled task to periodically write the StreamCompute lowest watermark into the system table. When StreamCompute is shut down and restarted, the StreamCompute Reader module will load the judgment and load the unprocessed data based on this watermark, completing the breakpoint resumption of the StreamCompute task.

[0024] Furthermore, in step S4, the StreamCompute data recalculation module determines the range of historical data by using the latest timestamp of the source time series table when StreamCompute was created;

[0025] The range of the breakpoint data is determined by the lowest water mark calculated by the stream and the latest timestamp of the source time series table;

[0026] Expired data is single-point data and serves as the input of the StreamCompute data recalculation module.

[0027] Based on the characteristics of the three types of data, the StreamCompute data recalculation module calculates the precise range of the recalculated data, that is, the precise start and end timestamps of the aggregation window.

[0028] Furthermore, the processing of historical data is divided into two stages. First, the "maximum timestamp of existing data" in the source time series table is confirmed through internal query, and a time filter condition is added to the original StreamCompute query to process this part of the data.

[0029] Then, use the "maximum timestamp of existing data" to confirm the start time of the last aggregation window and use it as the input of the "aggregation window splitting" data processing module. If it is a non-aggregation query, it will be skipped.

[0030] Furthermore, the breakpoint data is also divided into two stages. The first step is to use the lowest watermark of StreamCompute to search the target table to confirm the data recalculation start time. If it is an aggregation StreamCompute query, the start time of the last aggregation window is used. Then, an internal query is performed to confirm the "maximum timestamp of existing data" in the source time series table. The data between these two timestamps is the breakpoint data. Time filter conditions can be added to the original StreamCompute query to process this part of data.

[0031] The second step is to record the start and end time of each aggregation window in the target table. The system searches for records whose aggregation window start time is less than the maximum timestamp of existing data. The maximum timestamp is used as the start timestamp for splitting the window data. The end timestamp of the window split data is determined by the stream computing aggregation function.

[0032] The expired data of time series data in the expired data generally arrives in batches. The expired data will wait for multiple checkpoint cycles before entering the data recalculation process:

[0033] (1) Sort expired data and confirm the start and end timestamps;

[0034] (2) Use the start timestamp to confirm the start time of data recalculation, and use the end timestamp to confirm the end time of data recalculation.

[0035] Furthermore, the aggregation window splitting data input is the window splitting timestamp, and the processing flow is:

[0036] (1) Search the target table and confirm the start time of the latest aggregation window that is less than the "window split timestamp" as the data recalculation start time;

[0037] (2) Search the target table and confirm the end time of the first aggregation window that is greater than the "window split timestamp" as the data recalculation end time; if it does not exist, the aggregation window is not closed, and this "split window" is added to the waiting list and waits for the next calculation cycle;

[0038] (3) Before persisting and recalculating data, invalid data existing in the target table must be deleted based on the data unique identifier;

[0039] (4) If there are data with the same window end time in the recalculation result, only the first one will be persisted and the rest will be discarded.

[0040] (5) The checkpoint task of the StreamCompute Reader component periodically activates the data recalculation logic to check and update the end time of the split window.

[0041] Furthermore, in step (4), it includes:

[0042] (4-1) The end time of the aggregation window is usually monotonically increasing;

[0043] (4-2) If a sliding window is used when defining stream computing, multiple data items may appear with the same aggregation window end time. Except for the first item, the other aggregation results do not fall within the data range of this recalculation and are invalid data. They need to be discarded. Otherwise, the correct aggregation results will be overwritten.

[0044] A stream computing device for a distributed database, comprising: at least one memory and at least one processor;

[0045] The at least one memory is configured to store a machine-readable program;

[0046] The at least one processor is configured to call the machine-readable program to execute a stream computing method for a distributed database.

[0047] Compared with the prior art, the stream computing method and device for distributed databases of the present invention have the following outstanding beneficial effects:

[0048] The present invention can efficiently complete the streaming calculation of database time series data, and use the streaming calculation recalculation module to efficiently complete the recalculation of window-split data, with its performance degradation within 15% of the original storage performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow diagram of a streaming computing method for distributed databases. Figure 1 ;

[0051] Figure 2 The present invention is a flow chart of a stream computing recalculation module in a stream computing method for distributed databases. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] A best embodiment is given below:

[0054] like Figure 1 、 2 As shown, a stream computing method for a distributed database in this embodiment has the following steps:

[0055] S1. The StreamCompute Reader module receives time series data changes captured by the CDC (data change capture) component in real time and saves them to the local cache.

[0056] Each time data is updated, the StreamCompute Reader module checks whether there is data in the local cache that exceeds the out-of-order window for the time series data. If so, it sorts this data based on the timestamp and Group By columns and returns it to the upper-level operator. At the same time, the StreamCompute Reader module also uses a scheduled task to periodically check and output time series data in the cache that is larger than the out-of-order window.

[0057] S2, the StreamCompute Aggregator module performs stream computing aggregation calculations;

[0058] Aggregation operators in typical database systems must read all data before completing aggregation calculations. The StreamCompute Aggregator module introduces an efficient internal stream computation mechanism that aggregates time series data in real time according to user-defined rules. Furthermore, the StreamCompute Aggregator module supports a forced closing mechanism for aggregation windows to prevent them from being permanently closed.

[0059] S3: The StreamCompute Receive module receives the calculation results of StreamCompute, converts the result set into a format compatible with the target table, and then uses the internal executor to write the data to the target table.

[0060] The StreamCompute Receiver module also uses a scheduled task to periodically write the lowest watermark of StreamCompute to the system table. When StreamCompute is shut down and restarted, the StreamCompute Reader module loads unprocessed data based on this watermark, completing the breakpoint-resume transfer of the StreamCompute task.

[0061] S4. The stream computing recalculation module needs to process the historical data that existed when the stream computing was created, the breakpoint data generated during the stream computing pause, and the expired data that exceeds the time series data disorder window;

[0062] like Figure 2 As shown, these three types of data share commonalities and can be uniformly processed by the StreamCompute data recalculation module. Furthermore, when processing historical and breakpoint data, "aggregation window fragmentation" (i.e., aggregation results generated by incomplete aggregation windows) may occur, and this data is also uniformly processed by this module.

[0063] The StreamCompute data recalculation module determines the range of historical data using the latest timestamp of the source time series table at the time of StreamCompute creation. It also determines the range of breakpoint data using the StreamCompute minimum watermark and the latest timestamp of the source time series table. Expired data is single-point data and can be used as input to the StreamCompute data recalculation module.

[0064] Based on the characteristics of the above three types of data, the StreamCompute data recalculation module can calculate the precise range of the recalculated data, that is, the precise start and end timestamps of the aggregation window.

[0065] The processing of historical data is divided into two stages. The first stage is to confirm the "maximum timestamp of existing data" in the source time series table through internal queries, and add time filter conditions in the original stream computing query to process this part of the data.

[0066] In the second step, use the "maximum timestamp of existing data" to confirm the start time of the last aggregation window and use it as the input of the "aggregation window splitting" data processing module. If it is a non-aggregation query, the second step can be skipped.

[0067] Processing breakpoint data is similar to historical data and is divided into two stages. The first step is to use the lowest watermark in StreamCompute to determine the data recalculation start time by searching the target table. For aggregate StreamCompute queries, this is the start time of the last aggregation window. Then, an internal query is performed to determine the maximum timestamp of existing data in the source time series table.

[0068] The data between these two timestamps is the breakpoint data. You can add a time filter condition to the original stream computing query to process this part of the data.

[0069] The second step is to record the start and end time of each aggregation window in the target table. By searching for records whose aggregation window start time is less than the "maximum timestamp of existing data", the largest timestamp is used as the starting timestamp of the window split data. The end timestamp of the window split data is determined by the aggregation function of the stream calculation.

[0070] Expired data, such as time series data, generally arrives in batches. To reduce the frequency of recalculation, expired data will wait for multiple checkpoint cycles. Then, it will enter the data recalculation process uniformly:

[0071] (1) Sort expired data and confirm the start and end timestamps;

[0072] (2) Similar to historical data, the start timestamp is used to confirm the start time of data recalculation, and the end timestamp is used to confirm the end time of data recalculation.

[0073] The processing of historical data, breakpoint data, and expired data will generate "aggregation window split" data. Its input is the "window split timestamp" (the timestamp of the last data of historical data, breakpoint data, and expired data). The processing flow is as follows:

[0074] (1) Search the target table and confirm the start time of the latest aggregation window that is less than the "window split timestamp" as the data recalculation start time;

[0075] (2) Search the target table and confirm the end time of the first aggregation window that is greater than the "window split timestamp" as the data recalculation end time; if it does not exist, the aggregation window is not closed, and this "split window" is added to the waiting list and waits for the next calculation cycle;

[0076] (3) Before persisting and recalculating data, invalid data existing in the target table must be deleted based on the data's unique identifier (data timestamp and possible device identifier);

[0077] (4) If there are data with the same window end time in the recalculation result, only the first one will be persisted and the rest will be discarded.

[0078] include:

[0079] 4-1. The end time of the aggregation window is usually monotonically increasing;

[0080] 4-2. If a sliding window is used when defining a stream calculation, multiple data entries may appear with the same aggregation window end time. Except for the first entry, the remaining aggregation results are not within the data range of this recalculation and are invalid data (aggregation windows that were not closed normally). These data must be discarded to prevent the correct aggregation results from being overwritten.

[0081] (5) The checkpoint task of the StreamCompute Reader component periodically activates the data recalculation logic to check and update the end time of the split window.

[0082] S5, the stream computing task management module manages the life cycle of stream computing in the distributed database cluster;

[0083] At the same time, it also constructs a stream computing physical execution plan based on user input, initializes and runs the stream computing reader module, stream computing aggregator module and stream computing receiver module.

[0084] Finally, it is also responsible for completing the error retry function of the stream computing task.

[0085] Based on the above method, a stream computing device for a distributed database in this embodiment includes: at least one memory and at least one processor;

[0086] The at least one memory is configured to store a machine-readable program;

[0087] The at least one processor is configured to call the machine-readable program to execute a stream computing method for a distributed database.

[0088] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any technical solutions that conform to the above-mentioned specific implementation methods of the present invention and any appropriate changes or substitutions made thereto by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0089] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A streaming computing method for a distributed database, characterized in that: The following steps are involved: S1. The StreamCompute Reader module receives the time series data changes captured by the CDC component in real time and saves them to the local cache. S2, the StreamCompute Aggregator module performs stream computing aggregation calculations; S3: The StreamCompute Receive module receives the calculation results of StreamCompute, converts the result set into a format compatible with the target table, and then uses the internal executor to write the data to the target table. S4. The stream computing recalculation module needs to process the historical data that existed when the stream computing was created, the breakpoint data generated during the stream computing pause, and the expired data that exceeds the time series data disorder window; S5. The stream computing task management module manages the life cycle of stream computing in the distributed database cluster.

2. A stream computing method for a distributed database according to claim 1, characterized in that: In step S1, every time data is updated, the StreamCompute Reader module checks whether there is data in the local cache that exceeds the out-of-order window of the time series data; If there is, this part of the data is sorted according to the timestamp and Group BY column and returned to the upper-level operator. At the same time, the StreamCompute Reader module also uses a scheduled task to periodically check and output the time series data in the cache that is larger than the out-of-order window.

3. A stream computing method for a distributed database according to claim 2, characterized in that: In step S2, the database system aggregation operator needs to read all the data before completing the aggregation calculation. The stream computing Aggregator module introduces an internal stream computing mechanism to aggregate time series data in real time according to user-defined rules. At the same time, the StreamCompute Aggregator module supports a forced closing mechanism for aggregation windows.

4. A stream computing method for a distributed database according to claim 3, characterized in that: In step S3, the StreamCompute Receiver module uses a scheduled task to periodically write the StreamCompute lowest watermark to the system table. When StreamCompute is shut down and restarted, the StreamCompute Reader module will load the unprocessed data based on this watermark, completing the breakpoint-resume transfer of the StreamCompute task.

5. A stream computing method for a distributed database according to claim 4, characterized in that: In step S4, the StreamCompute data recalculation module determines the scope of historical data based on the latest timestamp of the source time series table when StreamCompute was created; The range of the breakpoint data is determined by the lowest water mark calculated by the stream and the latest timestamp of the source time series table; Expired data is single-point data and serves as the input of the StreamCompute data recalculation module. Based on the characteristics of the three types of data, the StreamCompute data recalculation module calculates the precise range of the recalculated data, that is, the precise start and end timestamps of the aggregation window.

6. A stream computing method for a distributed database according to claim 5, characterized in that: The processing of historical data is divided into two stages. First, the "maximum timestamp of existing data" in the source time series table is confirmed through internal query, and a time filter condition is added to the original StreamCompute query to process this data. Then, use the "Maximum Timestamp of Existing Data" to determine the start time of the last aggregation window and use this as the input for the "Aggregation Window Splitting" data processing module. If it is a non-aggregation query, it is skipped.

7. A stream computing method for a distributed database according to claim 6, characterized in that: The processing of breakpoint data is also divided into two stages. The first step is to use the lowest watermark of StreamCompute to search the target table to confirm the data recalculation start time. If it is an aggregation StreamCompute query, the start time of the last aggregation window is used. Then, an internal query is performed to confirm the "maximum timestamp of existing data" in the source time series table. The data between these two timestamps is the breakpoint data. Time filter conditions can be added to the original StreamCompute query to process this data. The second step is to record the start and end time of each aggregation window in the target table. The target table searches for records whose aggregation window start time is less than the maximum timestamp of existing data. The maximum timestamp is used as the start timestamp for splitting the window data. The end timestamp of the window split data is determined by the stream computing aggregation function. The expired data of time series data in the expired data generally arrives in batches. The expired data will wait for multiple checkpoint cycles before entering the data recalculation process: (1) Sort expired data and confirm the start and end timestamps; (2) Use the start timestamp to confirm the start time of data recalculation, and use the end timestamp to confirm the end time of data recalculation.

8. A stream computing method for a distributed database according to claim 7, characterized in that: The aggregation window split data input is the window split timestamp, and the processing flow is as follows: (1) Search the target table and confirm the start time of the latest aggregation window that is less than the "window split timestamp" as the data recalculation start time; (2) Search the target table and confirm the end time of the first aggregation window that is greater than the "window split timestamp" as the data recalculation end time; if it does not exist, the aggregation window is not closed, and this "split window" is added to the waiting list and waits for the next calculation cycle; (3) Before persisting and recalculating data, invalid data existing in the target table must be deleted based on the data unique identifier; (4) If there are data with the same window end time in the recalculation result, only the first one will be persisted and the rest will be discarded. (5) The checkpoint task of the StreamCompute Reader component periodically activates the data recalculation logic to check and update the end time of the split window.

9. A stream computing method for a distributed database according to claim 8, characterized in that: In step (4), it includes: (4-1) The end time of the aggregation window is usually monotonically increasing; (4-2) If a sliding window is used when defining stream computing, multiple data items may appear with the same aggregation window end time. Except for the first item, the other aggregation results do not fall within the data range of this recalculation and are invalid data. They need to be discarded. Otherwise, the correct aggregation results will be overwritten.

10. A stream computing device for a distributed database, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 9.

Images