A time series data synchronizer

Abstract

The application discloses a kind of time series data synchronizer, belong to time series database technical field, the technical problem to be solved is how to provide a kind of high performance high throughput real-time synchronizer, meet the demand of a large number of time series data persistence in edge end. Including: synchronization record storage component, for reading and maintaining the key to be read and corresponding reading time stamp;RedisTsSource data source component, for reading time series data from RedisTimeSeries time series database in parallel according to the key to be read and corresponding reading start time stamp provided by synchronization record storage component;Data converter component, for converting time series data into data table structure time series data;TaosSink data sinking component, for converting data table structure time series data into SQL statement, and sending SQL statement to TDEngine.

Description

Technical Field

[0001] This invention relates to the field of time-series database technology, specifically a time-series data synchronizer. Background Technology

[0002] Edge computing refers to processing, analyzing, and storing data closer to where it is generated, enabling rapid and near real-time analysis and response. With the development of IoT technology, more and more things (devices or sensors, etc.) are interconnected, generating a large amount of time-series data. Time-series data refers to a series of numerical data points for a specific indicator over a period of time, uniquely identified by the indicator, label, and timestamp. This indicator data is immutable. To meet the needs of edge computing, an edge data platform was designed and implemented.

[0003] As attached Figure 1 The edge data platform shown primarily consists of a collector, message queue, synchronizer, RedisTimeSeries in-memory database, and TDEngine time-series database. The RedisTimeSeries in-memory database stores recent time-series data, while the TDEngine time-series database stores the full dataset. These components not only serve as edge storage for the edge service platform but also support rapid and near real-time edge computing. RedisTimeSeries provides a real-time computing engine for recent time-series data, while TDEngine provides a computing engine for large-scale data processing and analysis.

[0004] The time-series data stored in the Redis TimeSeries in-memory database is synchronized and moved down to the persistent storage of TDEngine. Due to the key-value data storage characteristics of the Redis TimeSeries time-series database, which differ from the various data write interfaces provided by TDEngine, and considering the large amounts of time-series data collected at the edge, providing a high-performance, high-throughput real-time synchronizer to meet the persistence requirements of large amounts of time-series data at the edge is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing a time-series data synchronizer to solve the technical problem of how to provide a high-performance, high-throughput real-time synchronizer that meets the need for persistent storage of large amounts of time-series data at the edge.

[0006] This invention provides a time-series data synchronizer for synchronizing time-series data stored in a Redis TimeSeries in-memory database to persistent storage in a TDEngine. The synchronizer includes:

[0007] A synchronous recording memory component, which is used to read and maintain the key to be read and the corresponding reading timestamp;

[0008] The RedisTsSource data source component is used to read time-series data in parallel from the Redis TimeSeries time-series database based on the key to be read and the corresponding read start timestamp provided by the synchronous record storage component.

[0009] A data converter component, wherein the data converter is used to receive time-series data sent by the RedisTsSource data source component and convert the time-series data into time-series data with a data table structure;

[0010] The TaosSink data sinking component is used to receive time-series data of a data table structure sent by the data converter component, convert the time-series data of the data table structure into SQL statements, and send the SQL statements to TDEngine.

[0011] Preferably, the synchronous recording memory includes:

[0012] Keys reader, the Keys reader is used to periodically read the Keys to be read and the corresponding read start timestamp from the metadata storage;

[0013] Keys grouper, which is used to group Keys according to the sampling frequency of the sampling point corresponding to each Key;

[0014] The Keys updater is used to listen to the event listening channel, synchronously update the Keys groups in memory, and asynchronously update the Keys read timestamps and the number of synchronized data points in the metadata storage.

[0015] Preferably, the RedisTsSource data source component includes:

[0016] The data reader is used to periodically acquire Keys groups through the memory handle of the synchronous recording memory, to create timer groups and start timers according to the acquisition frequency of different groups, and to group Keys with the same acquisition frequency according to different reading start timestamps through timers to obtain different Keys groups, and to read time series data from the Redis TimeSeries time series database in parallel based on Keys groups.

[0017] A data channel is used to send time-series data to a data converter component.

[0018] Preferably, Redis TimeSeries extends and implements a time-series data reading interface for multiple key groups within the same time range, and the data reader is used to read the corresponding time-series data through the interface.

[0019] Preferably, the data converter component includes:

[0020] A table grouper is used to aggregate time-series data from the same device into the same group to obtain at least one table group.

[0021] A row grouper is used to aggregate time-series data in the same table group into the same group based on timestamps, resulting in at least one row group.

[0022] A column set grouper is used to aggregate time-series data from different row groups into the same group based on the same set of collection points, resulting in multiple data tables.

[0023] Preferably, the rows field in the data table is of map type, the key of the rows field is a string formed by sorting and concatenating the column set, the value of the rows field is a set of time series data with the same column as the key, and the number field in the data table is the number of time series data entries.

[0024] Preferably, the TaosSink data sinking component includes:

[0025] An SQL generator is used to obtain time-series data in the form of a database table from the data converter component and generate an SQL statement from the time-series data in the database table structure.

[0026] The writer is used to read SQL statements from the SQL generator and send the SQL statements to TDEngine. By executing the SQL statements in TDEngine, the time-series data is written to the persistent storage of the TDEngine time-series database. The writer is also used to pass the read keys and timestamps to the event listening channel of the synchronous record memory through the memory handle of the synchronous record memory. The synchronous record memory updates the read keys and timestamps to the memory and metadata storage through its key updater.

[0027] Preferably, when the cumulative value of the number of time series data entries reaches a threshold, or when it reaches the timestamp value of the timer in the writer, the writer is used to write the time series data in the form of SQL statements into the persistent storage of the TDEngine time series database.

[0028] The number of time-series data entries is recorded in the number field of the data table.

[0029] The timing data synchronizer of the present invention has the following advantages:

[0030] 1. Quickly synchronize time-series data from the Redis TimeSeries in-memory time-series database to the TDEngine persistent time-series database to meet the needs of persistent large amounts of time-series data at the edge.

[0031] 2. In IoT application scenarios, massive amounts of time-series data need to be collected and stored by edge devices. This synchronizer can reduce the hardware cost of edge devices. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] The invention will be further described below with reference to the accompanying drawings.

[0034] Figure 1 This is a diagram of the edge data platform architecture.

[0035] Figure 2 This is an overall architecture diagram of a timing data synchronizer as an example.

[0036] Figure 3 Here is a synchronizer data flow diagram for an embodiment of a timing data synchronizer;

[0037] Figure 4 This is a data table class diagram of a data converter in a timing data synchronizer, as shown in an embodiment. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0039] This invention provides a time-series data synchronizer to address the technical problem of how to provide a high-performance, high-throughput real-time synchronizer to meet the need for persistent storage of large amounts of time-series data at the edge.

[0040] Example:

[0041] The present invention provides a time-series data synchronizer, comprising a synchronization record storage component, a RedisTsSource data source component, a data converter component, and a TaosSink data sinking component. The four components work together to synchronize and sink time-series data stored in the Redis TimeSeries memory database to the persistent storage of TDEngine.

[0042] The synchronous record storage component is used to read and maintain the key to be read and its corresponding read timestamp (start_timestamp).

[0043] As attached Figure 3 As shown, this component consists of three sub-components: a Keys reader, a Keys grouper, and a Keys updater. The Keys reader is responsible for periodically retrieving the keys to be read and their corresponding read timestamps (start_timestamp) from the metadata storage. The Keys grouper is responsible for grouping the keys according to the collection frequency of each key's corresponding collection point. The Keys updater is responsible for listening to the event listening channel, synchronously updating the Keys groups in memory in real time, and asynchronously updating the Keys read timestamps and the number of synchronized data points in the metadata storage.

[0044] The RedisTsSource data source component is used to read time-series data in parallel from the Redis TimeSeries time-series database based on the key to be read and the corresponding read start timestamp provided by the synchronous record storage component, and at the same time, it sends the time-series data to the data converter component.

[0045] As attached Figure 3 As shown, this component includes a data reader and a data channel. The data reader is configured with a metadata timer and a memory handle for a synchronous recording memory. The metadata timer triggers the memory handle to periodically acquire Keys groups. It also creates timer groups and starts timers based on the acquisition frequency of different Keys groups. For Keys with the same acquisition frequency, the timer groups them according to different read start timestamps, resulting in different Keys groups. These groups are then used to read time-series data from the Redis TimeSeries database in parallel based on the Keys groups. In other words, its workflow is as follows: the data reader acquires Keys groups through the memory handle of the synchronous recording memory (metadata timer), then creates timer groups based on the acquisition frequency of different groups and starts these timers. Within the timers, Keys with the same acquisition frequency are further grouped according to different start timestamps.

[0046] The Redis TimeSeries extension implements an interface for reading time-series data under multiple keys with the same time range, so the data reader can read the corresponding time-series data through this interface;

[0047] This component passes the read timing data address into the data channel.

[0048] The data converter is used to receive time-series data sent by the RedisTsSource data source component and convert the time-series data into time-series data with a data table (TaosTable) structure.

[0049] like Figure 3 As shown, the data converter component includes a table grouper, a row grouper, and a column set grouper. The table grouper is used to aggregate time-series data from the same device into the same group to obtain at least one table group. The row grouper is used to aggregate time-series data from the same table group into the same group based on timestamps to obtain at least one row group. The column set grouper is used to aggregate time-series data from different row groups into the same group based on time-series data from the same collection point set to obtain multiple data tables.

[0050] Dataset structure as follows Figure 4 As shown, the rows field is of map type, its key is a string of sorted and concatenated columns, and its value is a set of time series data with the same columns as the key; the number field is the number of entries in the time series data.

[0051] The TaosSink data sinking component is used to receive time-series data of the data table structure sent by the data converter component, convert the time-series data of the data table structure into SQL statements, and send the SQL statements to TDEngine.

[0052] like Figure 3 As shown, the TaosSink data sinking component includes an SQL generator and a writer. The SQL generator is used to obtain time-series data in the form of a database table from the data converter component and generate an SQL statement from the time-series data in the database table structure. The writer is used to read the SQL statement from the SQL generator, send the SQL statement to TDEngine for execution, and write the time-series data into the persistent storage of the TDEngine time-series database by executing the SQL statement shown. It is also used to pass the read keys and timestamps to the event listening channel of the synchronous record memory through the memory handle held by the synchronous record memory. The synchronous record memory updates the read keys and timestamps to the memory and metadata storage through its key updater.

[0053] The SQL generator produces an SQL statement from TaosTable type data. Operations within the data converter component can reduce the length of the SQL statement, thereby reducing the amount of data transmitted to TDEngine via the network interface. Specifically, when the accumulated number of time-series data entries (the sum of the number fields in the TaosTable class) reaches the threshold set by the parameters, the writer writes the time-series data into TDEngine.

[0054] Considering that the number of cached time-series data entries may be small over a long period and may not reach the set length in a short time, a timer is configured in the writer. If the accumulated number of time-series data entries (summed from the number field in the TaosTable class) reaches the threshold set by the parameters or reaches the timestamp value of the timer in the writer, the writer will write the time-series data to TDEngine. That is, even if the time-series data has not reached the required length, if the timing requirement is met, the time-series data will still be sent to TDEngine.

[0055] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A time-series data synchronizer, comprising: A synchronizer for sinking time series data stored in a Redis TimeSeries in-memory database into a persistent storage of a TDEngine, the synchronizer comprising: a synchronization record storage component for reading and maintaining keys to be read and corresponding read timestamps; a RedisTsSource data source component for reading time series data from a Redis TimeSeries time series database in parallel according to the keys to be read and the corresponding read start timestamps provided by the synchronization record storage component; a data converter component for receiving the time series data sent by the RedisTsSource data source component and converting the time series data into table-structured time series data; a TaosSink data sinking component for receiving the table-structured time series data sent by the data converter component, converting the table-structured time series data into SQL statements, and sending the SQL statements to the TDEngine; wherein the RedisTsSource data source component comprises: a data reader in which a metadata timer and a memory handle of a synchronization record storage are configured, for triggering the memory handle to acquire Keys groups in time through the metadata timer, for creating timer groups according to the collection frequencies of different Keys groups and starting the timers, for grouping the Keys according to different read start timestamps through the timers for the same collection frequency of Keys, obtaining different Keys groups, and for reading time series data from the Redis TimeSeries time series database in parallel based on the Keys groups, the Redis TimeSeries implements a time series data reading interface under multiple Key groups in the same time range, and the data reader reads corresponding time series data through the interface; a data channel for sending the time series data to the data converter component; the synchronization record storage comprises: a Keys reader for reading the Keys to be read and the corresponding read start timestamps from the metadata storage in time; a Keys grouper for grouping the Keys according to the collection frequencies of the collection points corresponding to each Key; a Keys updater for listening to an event listening channel, synchronously updating the Keys groups in the memory, and asynchronously updating the Keys read timestamps and the number of synchronized data points in the metadata storage.

2. The timing data synchronizer of claim 1, wherein, the data converter component comprises: a table grouper for aggregating time series data of the same device into the same group to obtain at least one table group; a row grouper for aggregating time series data in the same table group according to timestamps into the same group to obtain at least one row group; A column set grouper is configured to aggregate time series data in different row groups into the same group according to time series data of the same collection point set, to obtain a plurality of data tables.

3. The timing data synchronizer of claim 2, wherein, The rows field in the data table is of a map type, the key of the rows field is a string after column set sorting and connection, and the value of the rows field is a time series data set having the same columns as the key.

4. The timing data synchronizer of claim 2, wherein, The TaosSink data sinking component includes: An SQL generator is configured to obtain time series data in a database table format from the data converter component, and to generate an SQL statement from the time series data in the database table structure. A writer is configured to read the SQL statement from the SQL generator, and to send the SQL statement to a TDengine, to write the time series data into a persistent storage of a time series database of the TDengine by executing the SQL statement in the TDengine, and to transmit the read keys and timestamps into an event listening channel of a synchronization record storage through a memory handle of the synchronization record storage, and the synchronization record storage updates the read keys and timestamps to the memory and the metadata storage through a keys updater.

5. The timing data synchronizer of claim 4, wherein, When the cumulative value of the time series data entry quantity reaches a threshold value, the writer is configured to write the time series data in the form of the SQL statement into the persistent storage of the time series database of the TDengine. The time series data entry quantity is recorded in the number field of the data table.

6. The timing data synchronizer of claim 5, wherein, The writer is configured with a timer, and when the cumulative value of the time series data entry quantity reaches the threshold value, or when a timestamp value of the timer in the writer is reached, the writer is configured to write the time series data in the form of the SQL statement into the persistent storage of the time series database of the TDengine.

Images