Data synchronization method, device and system of time sequence database and storage medium

Abstract

The invention provides a data synchronization method, device and system of a time sequence database and a storage medium, events generated by the time sequence database are monitored, and the events comprise an operation log writing event and a data file writing event; distributing a file life cycle for each data file, obtaining the current state of the file life cycle, and selecting to add the operation log write-in event or the data file write-in event into a waiting queue according to the current state of the file life cycle; the event is obtained from the waiting queue, and the operation log writing event or the data file writing event is selected to be sent to the receiving end according to the state of the file life cycle corresponding to the event. The log and the file are dynamically and mixedly transmitted, so that the problems of large network flow and high synchronization delay of a traditional scheme are effectively solved; the optimal balance between high throughput and low delay is realized, and in addition, the storage structure of the time sequence database is adapted, so that the transmission pressure and the calculation load of the receiving end in the edge weak network environment are reduced.

Description

Technical Field

[0001] This invention relates to the field of computer data management technology, and in particular to a data synchronization method, apparatus, system, and storage medium for a time-series database. Background Technology

[0002] In IoT applications, time-series databases need to synchronize massive amounts of time-series data generated by edge devices to the cloud. Traditional synchronization solutions borrow from the master-slave replication model of relational databases (such as MySQL), whose core idea is to synchronize based on user-triggered transactions. Taking MySQL as an example, MySQL master-slave replication is based on binary log files (Binlog), recording transactions formed by Structured Query Language (SQL) statements and re-executing them on the slave database to complete data synchronization. Specifically, when MySQL master-slave replication is started, three threads run simultaneously: a read thread runs on the master database, collecting data and writing it to the Binlog; a transmit thread continuously reads the master database's Binlog and synchronizes it to the slave database's replay log; and the slave database's replay thread continuously reads the replay log and re-executes the data on the slave database. However, this log-based synchronization method is problematic when dealing with time-series databases. Since time-series databases are built on a Log-Structured Merge tree (LSM) structure for their write and storage engines, they require batch writing to data files. Replaying a large number of single operation logs cannot leverage this advantage; instead, it consumes a lot of computing resources and affects database performance. Transmitting logs one by one generates huge network traffic, making it unsuitable for edge environments with limited bandwidth and unstable networks. Furthermore, during initial synchronization or when a large amount of data is backlogged, replaying logs one by one is very slow, and the latency is unacceptable. Summary of the Invention

[0003] This invention provides a data synchronization method, apparatus, system, and storage medium for time-series databases, which addresses the shortcomings of traditional log-based synchronization methods in time-series database scenarios, such as high network traffic, high synchronization latency, and inability to fully utilize the advantages of batch writing.

[0004] This invention provides a data synchronization method for a time-series database, applied at the sending end, comprising: Monitor events generated by the time-series database, including operation log write events and data file write events; Assign a file lifecycle to each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. The event is retrieved from the waiting queue, and based on the state of the file lifecycle corresponding to the event, the operation log write event or the data file write event is selected and sent to the receiving end.

[0005] According to the data synchronization method for a time-series database provided by the present invention, the file lifecycle states include an empty state, a file usage state, a log usage state, and a mixed usage state. The step of selecting to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle includes: If the current state of the file lifecycle is empty, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to the file usage state. Otherwise, if the event is a data file write event, the current state is set to file usage state; if the event is an operation log write event, the current state is set to log usage state.

[0006] According to the data synchronization method for a time-series database provided by the present invention, the step of selecting to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle further includes: If the current state of the file lifecycle is log usage state or mixed usage state, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to file usage state. If the system load does not exceed the preset threshold, then if the current state of the file lifecycle is the log usage state, then it remains unchanged; if the current state is the file usage state, then the current state is set to the file usage state.

[0007] According to the data synchronization method for time-series databases provided by the present invention, the current system load exceeding a preset threshold includes: the logs accumulated at the operation log source being written to memory exceeding a preset threshold. According to the data synchronization method for a time-series database provided by the present invention, the method selects to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle, including: If the current state of the file lifecycle is empty, then the operation log write event and data file write event corresponding to the file lifecycle are allowed to be added to the waiting queue; If the current state of the file lifecycle is the log usage state, then only the operation log write event corresponding to that file lifecycle is allowed to be added to the waiting queue; If the current state of the file lifecycle is the file usage state, then only the data file write event corresponding to that file lifecycle is allowed to be added to the waiting queue.

[0008] If the current state of the file lifecycle is a mixed-use state, then any write event corresponding to that file lifecycle is allowed to be added to the waiting queue.

[0009] According to the data synchronization method for time-series databases provided by the present invention, write events are sent to the receiving end.

[0010] The present invention also provides a data synchronization device for a time-series database, comprising: The monitoring module is used to monitor events generated by the time-series database, including operation log write events and data file write events. The module is used to assign a file lifecycle to each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. The sending module is used to retrieve events from the waiting queue and send the write events to the receiving end.

[0011] The present invention also provides a data synchronization system for a time-series database, comprising: The sending end is equipped with a data synchronization device for the time-series database as described above; The receiving end is used to receive events sent by the sending end, wherein: If the received event is an operation log write event, then the operation log write event is replayed as a write statement and executed in the receiving database; If the received event is a data file write event, then the data file write event is directly loaded into the storage structure of the receiving end database.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data synchronization method for the time-series database described in any of the preceding claims.

[0013] The present invention provides a data synchronization method, apparatus, system, and storage medium for time-series databases. By monitoring events generated by the time-series database, including operation log write events and data file write events; allocating a file lifecycle for each data file; obtaining the current state of the file lifecycle; and selecting to add either the operation log write event or the data file write event to a waiting queue based on the current state of the file lifecycle; retrieving events from the waiting queue; and sending the write events to the receiving end, the present invention effectively solves the problems of high network traffic and high synchronization latency in traditional solutions by dynamically mixing and transmitting logs and files. It achieves an optimal balance between high throughput and low latency. Furthermore, it adapts to the LSM-Tree storage structure of time-series databases, reducing transmission pressure and receiving end computational load in weak network environments at the edge, and improving data synchronization efficiency. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the data synchronization method for a time-series database provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the file lifecycle provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the event generation process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the functional structure of the data synchronization device for a time-series database provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the event receiving process provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0017] Figure 1 A flowchart of a data synchronization method for a time-series database provided in an embodiment of the present invention is shown below. Figure 1As shown, the data synchronization method for a time-series database provided in this embodiment of the invention includes: Step 101: Monitor the events generated by the time-series database, including operation log write events and data file write events; In this embodiment of the invention, an Event represents the generation of a new log or data file by the time-series database. It includes two types of Events: LogInsertionEvent: Operation log write event, that is, the log entry produced by a DML command executed by the user.

[0018] FileInsertionEvent: A data file write event, which is the event in which the LSM storage model writes temporary data in memory to a new data file generated on the disk.

[0019] In this embodiment of the invention, the event is generated by the Event Source and comprises two important components: Log Source: Operation log stream, which is the source of operation logs in a time-series database, typically the Write-Ahead Log (WAL). File Source: A file stream, which is a file stream consisting of new level 0 files generated by the LSM-Tree model in the time series database.

[0020] Step 102: Assign a file lifecycle to each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. In this embodiment of the invention, the waiting queue PendingQueue is a cached FIFO queue containing Events.

[0021] Step 103: Obtain the event from the waiting queue, and according to the state of the file lifecycle corresponding to the event, select to send the operation log write event or the data file write event to the receiving end.

[0022] In this embodiment of the invention, the sending end transmits the Event to the receiving end via the network through the sending connector.

[0023] Traditional time-series database synchronization schemes borrow from the master-slave replication model of relational databases, with three threads running concurrently: a read thread runs on the master database, collecting data and writing it to the Binlog; a transmission thread continuously reads the master database's Binlog and synchronizes it to the slave database's replay log; and the slave database's replay thread continuously reads the Replay Log and re-executes the process on the slave database. However, this log-based synchronization method is problematic when dealing with time-series databases. Because time-series databases are built on LSM (Local Standard Model) for writing and storage engines, they require batch writing to data files. Replaying a large number of single operation logs cannot leverage this approach; instead, it consumes significant computing resources and negatively impacts database performance. Transmitting logs one by one generates huge network traffic, making it unsuitable for edge environments with limited bandwidth and unstable networks. Furthermore, during initial synchronization or when large amounts of data are backlogged, replaying logs one by one is extremely slow, with unacceptable latency.

[0024] The data synchronization method for time-series databases provided in this invention monitors events generated by the time-series database, including operation log write events and data file write events; allocates a file lifecycle for each data file, obtains the current state of the file lifecycle, and selects to add either the operation log write event or the data file write event to a waiting queue based on the current state of the file lifecycle; retrieves the event from the waiting queue and sends the write event to the receiving end. This invention effectively solves the problems of high network traffic and high synchronization latency in traditional solutions by dynamically mixing and transmitting logs and files, achieving an optimal balance between high throughput and low latency. Furthermore, it adapts to the LSM-Tree storage structure of time-series databases, reducing transmission pressure and receiving end computing load in edge weak network environments, and improving data synchronization efficiency.

[0025] Based on any of the above embodiments, the file lifecycle states include an empty state, a file usage state, and a log usage state, such as... Figure 2 As shown, the FileEpoch is a collection of all events generated during the lifecycle of a data file. Each Event also has its own corresponding FileEpoch, which includes the following parts: This data file contains all LogInsertionEvents; The FileInsertionEvent corresponding to this file; The current state of FileEpoch.

[0026] FileEpoch has four states: EMPTY, USING_FILE, USING_LOG, and USING_BOTH. These states indicate that the current FileEpoch has not been accessed, the data within the current FileEpoch is synchronized using file synchronization, and the data within the current FileEpoch is synchronized using log synchronization.

[0027] The step of selecting to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle includes: If the current state of the file lifecycle is empty, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to the file usage state. Otherwise, if the event is a data file write event, the current state is set to file usage state; if the event is an operation log write event, the current state is set to log usage state.

[0028] Based on any of the above embodiments, the step of selecting to add the operation log write event or the data file write event to the waiting queue according to the current state of the file lifecycle further includes: If the current state of the file lifecycle is log usage state or mixed usage state, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to file usage state. If the system load does not exceed the preset threshold, then if the current state of the file lifecycle is log usage state, it remains unchanged; if the current state is file usage state, it switches to mixed usage state.

[0029] According to the data synchronization method for time-series databases provided by the present invention, the current system load exceeding a preset threshold only includes: the logs accumulated at the operation log source being written to memory exceeding a preset threshold. In embodiments of the present invention, such as Figure 3 As shown, the following state transition rules are executed for the FileEpoch corresponding to each newly generated Event: If the FileEpoch state is EMPTY, the state will be set to USING_FILE if the following conditions are met: too many write operations have accumulated at the Log Source, exceeding the user-defined memory limit; If the state of FileEpoch is not EMPTY: If the current Event is FileInsertionEvent, set the state to USING_FILE; If the current Event is LogInsertionEvent, set the status to USING_LOG; If the FileEpoch state is USING_LOG or USING_BOTH, the state will be set to USING_FILE if the following condition is met: too many write operations have accumulated at the Log Source, exceeding the user-defined memory limit.

[0030] Otherwise, the current state of FileEpoch will not be changed.

[0031] If the FileEpoch state is USING_FILE, the state will be set to USING_BOTH if the following condition is not met: too many write operations have accumulated at the Log Source, exceeding the user-defined memory limit.

[0032] Otherwise, the current state of FileEpoch will not be changed.

[0033] Based on any of the above embodiments, selecting to add the operation log write event or the data file write event to the waiting queue according to the current state of the file lifecycle includes: If the current state of the file lifecycle is empty, then the operation log write event and data file write event corresponding to the file lifecycle are allowed to be added to the waiting queue; If the current state of the file lifecycle is the log usage state, then only the operation log write event corresponding to that file lifecycle is allowed to be added to the waiting queue; If the current state of the file lifecycle is the file usage state, then only the data file write event corresponding to that file lifecycle is allowed to be added to the waiting queue.

[0034] If the current state of the file lifecycle is a mixed-use state, then all write events are allowed to be added to the waiting queue.

[0035] In this embodiment of the invention, the following enqueue rule is executed according to the state of the FileEpoch corresponding to the current Event: If the FileEpoch state corresponding to the current Event is EMPTY: all Events of this FileEpoch are allowed to be added to the PendingQueue; If the FileEpoch state corresponding to the current Event is USING_LOG: only the LogInsertionEvent of this FileEpoch is allowed to be added to the PendingQueue; If the FileEpoch state corresponding to the current Event is USING_FILE: only FileInsertionEvents of this FileEpoch are allowed to be added to the PendingQueue; Based on any of the above embodiments, the write event is sent to the receiving end.

[0036] In an IoT environment, databases are typically deployed in multiple locations, such as devices, edge gateways, and cloud data centers. This requires edge databases to synchronize data with cloud databases in real time. Traditional relational databases, such as MySQL, use master-slave replication for data synchronization, but this method faces many limitations in IoT scenarios, such as over-reliance on a single server and high data transmission traffic.

[0037] The data synchronization method for time-series databases provided in this invention combines log-based transmission (reducing latency) and data file-based transmission (improving throughput). This method dynamically selects these two modes to increase data throughput while maintaining low latency. Key data structures include: Events, Event Sources, PendingQueue, Connectors, and FileEpochs. Events are divided into two types: operation log writes (LogInsertionEvent) and data file writes (FileInsertionEvent). Event sources generate these events, the connector is responsible for transmitting these events over the network, and the file epoch is the collection of all events generated within the data file's lifecycle. Event sources monitor new events generated by the database, execute state transition rules for the file epoch of each new event, and determine how to add the event to the queue based on the current state. The connector retrieves events from the queue and executes state transition and dequeue rules based on the file epoch's state. The receiver is responsible for receiving events from the network and performing corresponding processing based on the event type. This invention provides a flexible and efficient data synchronization solution for time-series databases in an IoT environment, taking into account both the challenges of the network environment and the needs of data processing.

[0038] The data synchronization device for time-series databases provided by the present invention is described below. The data synchronization device for time-series databases described below can be referred to in correspondence with the data synchronization method for time-series databases described above.

[0039] Figure 4 A functional structure diagram of the data synchronization device for a time-series database provided in an embodiment of the present invention is shown below. Figure 4As shown, the data synchronization device for a time-series database provided in this embodiment of the invention includes: Monitoring module 401 is used to monitor events generated by the time-series database, including operation log write events and data file write events; The module 402 is used to allocate a file lifecycle for each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. The sending module 403 is used to obtain events from the waiting queue and, based on the state of the file lifecycle corresponding to the event, select to send the operation log write event or the data file write event to the receiving end.

[0040] The data synchronization device for time-series databases provided in this invention monitors events generated by the time-series database, including operation log write events and data file write events. It allocates a file lifecycle to each data file, obtains the current state of the file lifecycle, and selects to add either the operation log write event or the data file write event to a waiting queue based on the current state of the file lifecycle. It retrieves events from the waiting queue and, based on the state of the file lifecycle corresponding to the event, selects to send either the operation log write event or the data file write event to the receiving end. This invention effectively solves the problems of high network traffic and high synchronization latency in traditional solutions by dynamically mixing and transmitting logs and files, achieving an optimal balance between high throughput and low latency. Furthermore, it adapts to the LSM-Tree storage structure of time-series databases, reducing transmission pressure and receiving end computational load in weak network environments at the edge, and improving data synchronization efficiency.

[0041] This invention also provides a data synchronization system for a time-series database, comprising: The sending end is equipped with a data synchronization device for the time-series database as described in the above embodiments; The receiving end is used to receive events sent by the sending end, wherein: If the received event is an operation log write event, then the operation log write event is replayed as a write statement and executed in the receiving database; If the received event is a data file write event, then the data file write event is directly loaded into the storage structure of the receiving end database.

[0042] In embodiments of the present invention, such as Figure 5 As shown, the receiving end's working process includes: It continuously receives Events from the network. For each Event, if it is a LogInsertionEvent, the operation log is reproduced as a write statement and a standard write is performed on the receiving end time series database; if it is a FileInsertionEvent, the data file is directly placed into layer 0 of the LSM-Tree storage model of the receiving end time series database.

[0043] The data synchronization system for time-series databases provided in this invention differentiates the processing of two event types at the receiving end. When an operation log write event is received, it is replayed as a write statement and executed, ensuring the atomicity and real-time nature of data operations and guaranteeing accurate data synchronization. When a data file write event is received, the underlying storage characteristics of the time-series database are fully utilized to directly load the entire data file into layer 0 of the LSM-Tree storage structure. This processing method skips the traditional cumbersome process of parsing and executing SQL statements one by one, enabling batch and high-speed data import, greatly improving the throughput efficiency of massive data synchronization, and significantly reducing the CPU and I / O computational load on the receiving end. Ultimately, the system achieves synergistic optimization of high real-time performance and high throughput, perfectly meeting the stringent requirements of data synchronization efficiency and reliability in the Internet of Things environment.

[0044] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a data synchronization method for a time-series database provided by the methods described above. The method includes: monitoring events generated by the time-series database, the events including operation log write events and data file write events; allocating a file lifecycle for each data file, obtaining the current state of the file lifecycle, and selecting to add the operation log write event or the data file write event to a waiting queue based on the current state of the file lifecycle; retrieving an event from the waiting queue, and selecting to send the operation log write event or the data file write event to a receiving end based on the state of the file lifecycle corresponding to the event.

[0045] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0046] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data synchronization method for a time-series database, characterized in that, Applied to the sending end, including: Monitor events generated by the time-series database, including operation log write events and data file write events; Assign a file lifecycle to each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. The event is retrieved from the waiting queue, and based on the state of the file lifecycle corresponding to the event, the operation log write event or the data file write event is selected and sent to the receiving end.

2. The data synchronization method for a time-series database according to claim 1, characterized in that, The file lifecycle states include an empty state, a file usage state, a log usage state, and a mixed usage state. The step of adding either the operation log write event or the data file write event to the waiting queue based on the current file lifecycle state includes: If the current state of the file lifecycle is empty, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to the file usage state. Otherwise, if the event is a data file write event, the current state is set to file usage state; if the event is an operation log write event, the current state is set to log usage state.

3. The data synchronization method for a time-series database according to claim 2, characterized in that, The step of adding the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle also includes: If the current state of the file lifecycle is log usage state or mixed usage state, determine whether the current system load exceeds a preset threshold. If so, set the current state of the file lifecycle to file usage state. If the system load does not exceed the preset threshold and the current state is log usage state, then maintain it; if it is file usage state, then switch to mixed usage state.

4. The data synchronization method for a time-series database according to claim 2 or 3, characterized in that, The current system load exceeding a preset threshold includes: the amount of logs accumulated at the operation log source being written to memory exceeding a preset threshold.

5. The data synchronization method for a time-series database according to claim 1, characterized in that, The step of selecting to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle includes: If the current state of the file lifecycle is empty, then the operation log write event and data file write event corresponding to the file lifecycle are allowed to be added to the waiting queue; If the current state of the file lifecycle is the log usage state, then only the operation log write event corresponding to that file lifecycle is allowed to be added to the waiting queue; If the current state of the file lifecycle is the file usage state, then only the data file write event corresponding to that file lifecycle is allowed to be added to the waiting queue; If the current state of the file lifecycle is a mixed-use state, then all write events are allowed to be added to the waiting queue.

6. The data synchronization method for a time-series database according to claim 1, characterized in that, After retrieving an event from the waiting queue, if the file lifecycle state corresponding to the event is empty, and if the event is a data file write event, then the current file lifecycle state is set to file usage state; if the event is an operation log write event, then the current file lifecycle state is set to log usage state.

7. A data synchronization device for a time-series database, characterized in that, Applied to the sending end, including: The monitoring module is used to monitor events generated by the time-series database, including operation log write events and data file write events. The module is used to assign a file lifecycle to each data file, obtain the current state of the file lifecycle, and select to add the operation log write event or the data file write event to the waiting queue based on the current state of the file lifecycle. The sending module is used to retrieve events from the waiting queue and send the operation log write event or the data file write event to the receiving end.

8. A data synchronization system for a time-series database, characterized in that, include: The sending end is equipped with a data synchronization device for the time-series database as described in claim 7; The receiving end is used to receive events sent by the sending end, wherein: If the received event is an operation log write event, then the operation log write event is replayed as a write statement and executed in the receiving database; If the received event is a data file write event, then the data file write event is directly loaded into the storage structure of the receiving end database.

9. A non-transitory readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the data synchronization method for a time-series database as described in any one of claims 1 to 6.

Images