A WAL component mechanism design method suitable for distributed time series database

Abstract

The invention discloses a WAL component mechanism design method applicable to a distributed time series database, belongs to the technical field of distributed databases, and the technical problem to be solved is how to implement the WAL component mechanism design of a distributed database. For a distributed database integrated with a time series data engine, the public interface of the time series engine is opened as an upper-layer functional component, and the time series data engine is changed; the time series data engine calls the Checkpoint operation and Flush LogBuffer operation of the WAL component, and calls the recover method of the WAL component Mgr in each Entity of the time series table to recover data; and calls the Mini-transaction operation of the WAL component to realize the state synchronization of all steps in each transaction.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to a WAL component mechanism design method suitable for distributed time series databases. Background Art

[0002] Write-ahead logging (WAL component) is a series of technologies used to provide atomicity and persistence in database systems. In a system using WAL components, all data modifications must be written to the log file before being committed. The implementation principle of the WAL component mechanism is that the modification is not written directly to the database file, but to another file called the WAL component. If the transaction fails, the record in the WAL component will be ignored and the modification will be undone. If the transaction succeeds, it will be written back to the database file at a later time to commit the modification.

[0003] The previous application scenarios of WAL component mechanism implementation include stand-alone relational databases and distributed relational databases. There are not many WAL component mechanism implementations for distributed time series databases.

[0004] How to implement the WAL component mechanism design of a distributed database is a technical problem that needs to be solved. Summary of the invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a WAL component mechanism design method suitable for distributed time series databases to solve the technical problem of how to implement the WAL component mechanism design of distributed databases.

[0006] In a first aspect, the present invention provides a WAL component mechanism design method applicable to a distributed time series database, comprising the following steps:

[0007] For distributed databases that integrate a time series data engine, the public interface of the time series engine is opened as an upper-layer functional component, and the time series data engine is modified to add the Checkpoint method, FlushBuffer method, Recover function, and Mini-transaction method to the time series data engine;

[0008] The time series data engine calls the Checkpoint operation of the WAL component to refresh the data in the time series data engine MMap to the disk;

[0009] The time series data engine calls the Flush LogBuffer operation of the WAL component to flush the contents of the time series data engine log buffer to the disk;

[0010] When the distributed database is started, in the process of starting the time series engine, all time series tables in the storage directory of this node are traversed, and the recover method of the WAL component Mgr in each Entity of the time series table is called to recover data;

[0011] The insertion and update of time series data in a distributed database consists of multiple steps and involves the update of data files. The time series data engine calls the Mini-transaction operation of the WAL component to achieve state synchronization of all steps in each transaction.

[0012] Preferably, the upper-layer functional components include a distribution layer, an execution layer and a replica layer.

[0013] Preferably, the time series data engine calls the Checkpoint operation at a fixed interval, and the time series engine triggers the Checkpoint operation before the node stops normally and before data migration.

[0014] Preferably, when a scheduled task is triggered in a distributed database, the time series data engine traverses all EntityGroups and calls the Flush LogBuffer operation.

[0015] Preferably, the recover method of the WAL component Mgr in each Entity of the time series table is called to recover the data, including the following steps:

[0016] L100, obtain the list of the current node time series table ID and traverse it;

[0017] L200, traverse the EntityGroup of each time series table, and determine the LAST_CHECKPOINT_LSN for each EntityGroup;

[0018] L300, traverse the WAL component log file group of EntityGroup, confirm the starting log file where LAST_CHECKPOINT_LSN is located, and locate the LAST_CHECKPOINT_LSN position;

[0019] L400, starting from the starting log file where LAST_CHECKPOINT_LSN is located, traverse the subsequent log files:

[0020] L410, read the Log Block of all logs to be restored in the current log file and check the Checksum; L420, apply the WAL component logs in sequence:

[0021] L421. If it is a normal log, directly call the apply method of the relevant log type to apply the log; L422. If a BEGIN log of Mini-Transaction is encountered, record the xID of this Mini-Transaction to the unfinished transaction list, and apply subsequent logs according to normal logic;

[0022] L423. If a COMMIT log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared; L424. If a ROLLBACK log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared, and the rollback process is entered according to the xID; L425. After the WAL component is read, if the unfinished transaction list is not empty, the unfinished transactions are traversed, and according to the xID:

[0023] Add ROLLBACK log;

[0024] Roll back this incomplete transaction.

[0025] L426, call the storage layer interface to clean up data larger than the latest LSN;

[0026] The offsets of L430, data tables, and tag tables will change as the log application process progresses. If there is data in the data file that is written to the disk before the WAL component log, it will be overwritten by the log playback logic.

[0027] Preferably, the time series data engine calls the Mini-transaction operation of the WAL component to synchronize the status of all steps in each transaction, including the following steps:

[0028] Record a unique transaction ID, using the LSN of the BEGIN log as the transaction ID;

[0029] Before the replica layer starts applying a raftlog, it calls the TSMtrBegin method of the time series data engine to start Mini-Transaction;

[0030] The time series data engine generates and inserts the BEGIN log and returns its LSN as the transaction ID;

[0031] When the TSEngine interface is called for subsequent operations on the same raftlog, the transaction ID needs to be provided and recorded by the WAL component in the relevant WAL component log entry;

[0032] When the Raftlog application ends, the TSMtrCommit method of the time series data engine is called to commit the Mini-Transaction. The time series data engine generates and inserts the COMMIT log and records the transaction ID.

[0033] If the replica layer needs to roll back the current Mini-Transaction, it needs to call the TSMtrRollback method of the timing engine.

[0034] In a second aspect, the present invention provides a distributed database, wherein a time series data engine is integrated in the distributed database, and a WAL component is deployed in the time series data engine, which is used for a WAL component mechanism design method suitable for a distributed time series database as described in any one of the first aspects.

[0035] In a third aspect, the present invention provides a computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the processor executes any of the methods described in the first aspect.

[0036] The WAL component mechanism design method suitable for a distributed time series database of the present invention has the following advantages: the function of the WAL component of the distributed database is realized, and the atomicity and consistency of the database operation are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] The present invention is further described below in conjunction with the accompanying drawings.

[0039] Figure 1 This is a flowchart of a WAL component mechanism design method suitable for a distributed time series database in Example 1. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0041] The embodiment of the present invention provides a WAL component mechanism design method applicable to a distributed time series database, which is used to solve the technical problem of how to implement the WAL component mechanism design of a distributed database.

[0042] Embodiment 1:

[0043] In this embodiment, time series data refers to a group of data points collected at time intervals. A data point is usually based on a timestamp primary key and carries one or more data.

[0044] Entity refers to the basic unit for collecting time series data. In the IoT scenario, a device with a sensor is an Entity. In the development scenario, a monitored software application is also an Entity.

[0045] A tag exists based on an Entity and is used to identify an Entity or describe the information of an Entity. For example, "location" is a tag for a sensor and "ip_address" is a tag for an application. Tags are rarely modified in Entities.

[0046] KaiwuDB provides two levels of WAL logs: instance level and time series table level:

[0047] The instance-level WAL log is used to record changes to the schema of each time series table, including data type changes, column additions and subtractions, and tag additions and subtractions.

[0048] The WAL log at the time series table level is used to record all data changes related to this time series table, including data INSERT, UPDATE, and DELETE as well as transaction information. The time series table data is divided into different EntityGroups in the storage structure, so the storage granularity of WAL is also consistent with it, that is, each EntityGroup maintains an independent log file group.

[0049] The WAL log of a traditional relational database is usually a physical log for a specific data page. KaiwuDB's WAL is different. It uses a logical log to record changes in the time series table and adds log LSN information to the data record to determine the correspondence between the log entry and the data record.

[0050] The present invention provides a WAL component mechanism design method applicable to a distributed time series database, comprising the following steps:

[0051] S100. For a distributed database integrated with a time series data engine, the public interface of the time series engine is opened as an upper-layer functional component, and the time series data engine is changed to add a Checkpoint method, a FlushBuffer method, a Recover function, and a Mini-transaction method to the time series data engine;

[0052] S200, the time series data engine calls the Checkpoint operation of the WAL component to refresh the data in the time series data engine MMap to the disk;

[0053] S300: The time series data engine calls the Flush LogBuffer operation of the WAL component to flush the content of the time series data engine log buffer to the disk.

[0054] S400, when the distributed database is started, in the process of starting the time series engine, all time series tables in the storage directory of this node are traversed, and the recover method of the WAL component Mgr in each Entity of the time series table is called to recover data;

[0055] S500. The insertion and update of time series data in a distributed database consists of multiple steps and involves the update of data files. The time series data engine calls the Mini-transaction operation of the WAL component to achieve state synchronization of all steps in each transaction.

[0056] In step S100 of this embodiment, the upper-layer functional components include a distribution layer, an execution layer, and a replica layer. The WAL component requires the multi-time series data engine to make the following changes:

[0057] 1. Since the WAL log will periodically perform Checkpoint and Flush LogBuffer operations, it is necessary to add Checkpoint and FlushBuffer methods to TSEngine;

[0058] 2. Since TSEngine will call the Recover function of each time series table to recover the system when it starts, it is necessary to add a Recover method that supports WAL to TSEngine;

[0059] 3. Since the initiators and submitters of Mini-transaction and TSx are the replica layer and distributed layer of KaiwuDB, TSEngine, as a unified external access interface, needs to add methods related to time series table transactions.

[0060] The CHECKPOINT operation refreshes the data in the time series data engine MMap to the disk to ensure the persistence of user data.

[0061] The purpose of Checkpoint is to shorten the time required for database recovery when the database crashes. In theory, since WAL records all operations on time series data, it is not necessary to manually refresh MMap to disk, but to rely entirely on the synchronization mechanism of the operating system. When the database crashes, all WAL logs can be restored. Since KaiwuDB is an application environment that needs to ensure high availability, the speed of recovery greatly affects external applications. When Checkpoint is triggered, the time series data is first refreshed to disk and the current Checkpoint_LSN is recorded. In this way, when the system crashes and recovers, the refreshed data does not need to be restored, and only the operations after the checkpoint need to be restored.

[0062] The time series storage engine WAL will call the Checkpoint operation at a fixed interval to ensure that the MMap data is not overstocked. At the same time, the Checkpoint operation will also be triggered before the node is normally stopped and the data is migrated.

[0063] The KaiwuDB timing engine will periodically flush LogBuffer to the WAL log file. This operation is triggered by a KaiwuDB scheduled task, just like Checkpoint. Its process and Checkpoint type require traversing all EntityGroups and calling their FlushBuffer methods.

[0064] When a KaiwuDB node is started, in the process of starting the time series engine, it is necessary to traverse all time series tables in the storage directory of this node and call the recover method of WALMgr in each Entity of the time series table to recover data:

[0065] L100, obtain the list of the current node time series table ID and traverse it;

[0066] L200, traverse the EntityGroup of each time series table, and determine the LAST_CHECKPOINT_LSN for each EntityGroup;

[0067] L300, traverse the WAL component log file group of EntityGroup, confirm the starting log file where LAST_CHECKPOINT_LSN is located, and locate the LAST_CHECKPOINT_LSN position;

[0068] L400, starting from the starting log file where LAST_CHECKPOINT_LSN is located, traverse the subsequent log files:

[0069] L410, read the Log Block of all logs to be restored in the current log file and check the Checksum; L420, apply the WAL component logs in sequence:

[0070] L421. If it is a normal log, directly call the apply method of the relevant log type to apply the log; L422. If a BEGIN log of Mini-Transaction is encountered, record the xID of this Mini-Transaction to the unfinished transaction list, and apply subsequent logs according to normal logic;

[0071] L423. If a COMMIT log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared; L424. If a ROLLBACK log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared, and the rollback process is entered according to the xID; L425. After the WAL component is read, if the unfinished transaction list is not empty, the unfinished transactions are traversed, and according to the xID:

[0072] Add ROLLBACK log;

[0073] Roll back this incomplete transaction.

[0074] L426, call the storage layer interface to clean up data larger than the latest LSN;

[0075] The offsets of L430, data tables, and tag tables will change as the log application process progresses. If there is data in the data file that is written to the disk before the WAL component log, it will be overwritten by the log playback logic.

[0076] The insertion and update operations of KaiwuDB's time series data generally consist of multiple steps and involve the update of multiple data files. If an error occurs in any of the steps, or the system crashes during the operation for any reason, the consistency of the time series data may be destroyed if there is no atomicity guarantee between the multiple steps of the operation. The transaction engine of the time series storage engine can ensure the atomicity of data insertion and update operations and provide distributed transaction support.

[0077] In this embodiment, a unique transaction ID needs to be recorded at the beginning of Mini-Transaction. Currently, the LSN of the BEGIN log is used as the transaction ID. Before the replica layer starts applying a raftlog, it needs to call the TSMtrBegin method of TSEngine to start Mini-Transaction. The time series transaction engine will generate and insert the BEGIN log, and return its LSN as the transaction ID. Subsequent related operations on the same raftlog need to provide this transaction ID when calling the TSEngine interface, and the WAL component will record it in the relevant WAL log entry. At the end of the Raftlog application, it is necessary to call the TSMtrCommit method of TSEngine to commit the Mini-Transaction. The time series transaction engine will generate and insert the COMMIT log and record the transaction ID. If the replica layer needs to roll back the current Mini-Transaction, it is necessary to call the TSMtrRollback method of TSEngine.

[0078] Based on the method of the embodiment, the operation flow involved in the WAL module is described using a data writing scenario.

[0079] The INSERT operation generates and writes the BEGIN WAL log at the beginning, and writes the related INSERT WAL log before each subsequent operation, and then performs the data writing operation. After all operations are successful, write the COMMIT WAL log; if any intermediate step fails, call the rollback method of the WAL component to roll back, and its input is xID. At the beginning of the rollback, the rollback method writes the ROLLBACK WAL log, and traces back the WAL log involved in this xID, and then performs the reverse operation. If the rollback fails, the system error log is recorded, and then the error is reported and exited. After receiving an error message at this stage of the process, the replica layer will first enter the rollback process.

[0080] Embodiment 2:

[0081] The present invention provides a distributed database, in which a time series data engine is integrated. A WAL component is deployed in the time series data engine for executing the method disclosed in Example 1.

[0082] Embodiment 3:

[0083] The present invention provides a computer-readable medium on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method disclosed in Example 1.

[0084] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A WAL component mechanism design method suitable for distributed time series database, characterized in that: The steps include: For distributed databases that integrate a time series data engine, the public interface of the time series engine is opened as an upper-layer functional component, and the time series data engine is modified to add the Checkpoint method, FlushBuffer method, Recover function, and Mini-transaction method to the time series data engine; The time series data engine calls the Checkpoint operation of the WAL component to refresh the data in the time series data engine MMap to the disk; The time series data engine calls the Flush LogBuffer operation of the WAL component to flush the contents of the time series data engine log buffer to the disk; When the distributed database is started, in the process of starting the time series engine, all time series tables in the storage directory of this node are traversed, and the recover method of the WAL component Mgr in each Entity of the time series table is called to recover data; The insertion and update of time series data in a distributed database consists of multiple steps and involves the update of data files. The time series data engine calls the Mini-transaction operation of the WAL component to achieve state synchronization of all steps in each transaction.

2. According to claim 1, the WAL component mechanism design method suitable for distributed time series database is characterized in that: The upper-level functional components include the distribution layer, execution layer, and replica layer.

3. The WAL component mechanism design method applicable to a distributed time series database according to claim 1, characterized in that: The time series data engine calls the Checkpoint operation at a fixed interval. The time series engine triggers the Checkpoint operation when the node stops normally and before data migration.

4. The WAL component mechanism design method applicable to a distributed time series database according to claim 1, characterized in that: When a scheduled task is triggered in a distributed database, the time series data engine traverses all EntityGroups and calls the FlushLogBuffer operation.

5. The WAL component mechanism design method applicable to a distributed time series database according to claim 1, characterized in that: Call the recover method of the WAL component Mgr in each Entity of the time series table to recover data, including the following steps: L100, obtain the list of the current node time series table ID and traverse it; L200, traverse the EntityGroup of each time series table, and determine the LAST_CHECKPOINT_LSN for each EntityGroup; L300, traverse the WAL component log file group of EntityGroup, confirm the starting log file where LAST_CHECKPOINT_LSN is located, and locate the LAST_CHECKPOINT_LSN position; L400, starting from the starting log file where LAST_CHECKPOINT_LSN is located, traverse the subsequent log files: L410, read the Log Block of all logs to be restored in the current log file and check the Checksum; L420, apply the WAL component logs in sequence: L421. If it is a normal log, directly call the apply method of the relevant log type to apply the log; L422. If a BEGIN log of Mini-Transaction is encountered, record the xID of this Mini-Transaction to the unfinished transaction list, and apply subsequent logs according to normal logic; L423. If a COMMIT log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared; L424. If a ROLLBACK log of Mini-Transaction is encountered, the corresponding record in the unfinished transaction list is cleared, and the rollback process is entered according to the xID; L425. After the WAL component is read, if the unfinished transaction list is not empty, the unfinished transactions are traversed, and according to the xID: Add ROLLBACK log; Roll back this incomplete transaction. L426, call the storage layer interface to clean up data larger than the latest LSN; The offsets of L430, data tables, and tag tables will change as the log application process progresses. If there is data in the data file that is written to the disk before the WAL component log, it will be overwritten by the log playback logic.

6. The WAL component mechanism design method applicable to a distributed time series database according to claim 1, characterized in that: The time series data engine calls the Mini-transaction operation of the WAL component to synchronize the status of all steps in each transaction, including the following steps: Record a unique transaction ID, using the LSN of the BEGIN log as the transaction ID; Before the replica layer starts applying a raftlog, it calls the TSMtrBegin method of the time series data engine to start Mini-Transaction; The time series data engine generates and inserts the BEGIN log and returns its LSN as the transaction ID; When the TSEngine interface is called for subsequent operations on the same raftlog, the transaction ID needs to be provided and recorded by the WAL component in the relevant WAL component log entry; When the Raftlog application ends, the TSMtrCommit method of the time series data engine is called to commit the Mini-Transaction. The time series data engine generates and inserts the COMMIT log and records the transaction ID. If the replica layer needs to roll back the current Mini-Transaction, it needs to call the TSMtrRollback method of the timing engine.

7. A distributed database, characterized in that: The distributed database is integrated with a time series data engine, and a WAL component is deployed in the time series data engine, which is used to execute a WAL component mechanism design method applicable to a distributed time series database as described in any one of claims 1 to 6.

8. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.

Images