Time sequence database log storage method and computer readable medium
By establishing a log system and a circular linked list buffer for timing database operations, the write and recovery performance problems of timing databases in massive data scenarios are solved, efficient data recovery and rollback are achieved, and the database read and write throughput is improved.
Patent Information
- Application Number
- CN202411834053.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-15
AI Technical Summary
In the massive data scenario, it is difficult to efficiently write and restore operations on the timing database, which affects the performance of data recovery and rollback.
The time-series database log storage method is used to establish a log system for DDL and DML in database operations, create a buffer in ring-linked list format, format the database operations into log format, and write logs to disk according to the log security level, creating checkpoints to support data recovery and rollback.
It improves the write and recovery performance of timing databases in massive data scenarios, supports data recovery and rollback, and greatly increases the read and write throughput of the database by running multiple log systems in parallel.
Smart Images
Figure CN119829329A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database technology, and in particular to a time series database log storage method and a computer-readable medium. Background Art
[0002] The database WAL log (Write Ahead Log) refers to the operation of the database, which is pre-written to the disk in the form of a log before being written to the storage. It is used for recovery and transaction rollback when the database fails to ensure data security and consistency. With the development of Internet of Things technology, the application of time series databases has become more extensive. Compared with the original relational database, it has the characteristics of each row having a timestamp and device ID, high production frequency, and large data volume. This puts higher requirements on the log system of the time series database, and meets the high-speed reading and writing requirements in massive data scenarios while supporting functions such as data recovery and data rollback.
[0003] For time series database WAL logs and devices, how to improve the writing and recovery performance of time series databases in massive data scenarios is a technical problem that needs to be solved. Summary of the invention
[0004] The technical task of the present invention is to address the above shortcomings and provide a time series database log storage method and a computer-readable medium to solve the technical problem of how to improve the writing and recovery performance of the time series database WAL log and equipment in the scenario of massive data.
[0005] In a first aspect, the present invention provides a time series database log storage method, comprising the following steps:
[0006] Establish a logging system for DDL and DML operations in database operations;
[0007] Create n block ring chain table buffers for each log system;
[0008] Format database operations into log format;
[0009] Write the log into the log system buffer and return the total number of bytes written into the log as the current LSN.
[0010] According to the log security level, the buffer is written into the log file in blocks;
[0011] Before the DDL operation, a checkpoint is created at a predetermined time point, the data in the buffer is forced to be written to the disk, and the current LSN is recorded as the checkpoint LSN;
[0012] When the database is restarted abnormally, the logs between the checkpoint LSN and the current LSN are read for playback. During playback, the log LSN is checked. If it is greater than the LSN in the database, redo is executed, otherwise it is skipped.
[0013] When performing a rollback operation, read the LSN of the specified begin operation, use other transaction IDs as all logs of the LSN, sort them in reverse order, and perform undo one by one.
[0014] Preferably, a log system is established for the DML according to the device group, and the log system runs in parallel, and its device group is calculated by the device ID using a predetermined algorithm.
[0015] Preferably, the database operation is formatted into a log format to obtain insert operation logs, update operation logs and delete log records. The insert operation log is used to insert content, the update operation log is used to record updated content and marks of overwritten content, and the delete log record is used to record marks of deleted content.
[0016] Preferably, the buffer is written into the log file block by block in a regular or immediate manner according to the log security level.
[0017] Preferably, before the DDL operation, a checkpoint is created when the database is closed and at regular time points.
[0018] Preferably, when redo is executed, the DDL log is restored and the DML log is restored in parallel according to the device group.
[0019] In a second aspect, the present invention provides a computer-readable medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.
[0020] The time series database log storage method and computer-readable medium of the present invention have the following advantages: support data recovery and data rollback, and are divided into multiple independent log systems according to device groups. Multiple log systems can run in parallel, greatly increasing the database read and write throughput. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] The present invention is further described below in conjunction with the accompanying drawings.
[0023] Figure 1 This is a flowchart of a time series database log storage method according to Example 1. DETAILED DESCRIPTION
[0024] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.
[0025] The embodiments of the present invention provide a time series database log storage method and a computer-readable medium, which are used to solve the technical problems of time series database WAL logs and devices, and how to improve the writing and recovery performance of time series databases in massive data scenarios.
[0026] Embodiment 1:
[0027] The present invention provides a time series database log storage method, comprising the following steps:
[0028] S100, establish a log system for DDL operations in database operations, and establish a log system for DML operations;
[0029] S200, creating n block ring chain table buffers for each log system;
[0030] S300, formatting the database operation into a log format;
[0031] S400, write the log into the buffer of the log system, and return the total number of bytes currently written into the log as the current LSN;
[0032] S500, writing the buffer into the log file according to the block according to the log security level;
[0033] S600, before the DDL operation, create a checkpoint at a predetermined time point, force the data in the buffer to be written to the disk, and record the current LSN as the checkpoint LSN;
[0034] S700, when the database is restarted abnormally after shutdown, the logs between the checkpoint LSN and the current LSN are read for playback. During playback, the log LSN is checked. If it is greater than the LSN in the database, redo is executed, otherwise it is skipped.
[0035] S800, when performing a rollback operation, read the LSN of the specified begin operation, use other transaction IDs as all logs of the LSN, sort them in reverse order, and perform undo one by one.
[0036] In step S100, a log system is established for the DML according to the device group, and the log system runs in parallel, and its device group is calculated by the device ID using a predetermined algorithm.
[0037] In step S300, the database operation is formatted into a log format to obtain an insert operation log, an update operation log and a delete log record. The insert operation log is used to insert content, the update operation log is used to record the updated content and the mark of the overwritten content, and the delete log record is used to record the mark of the deleted content.
[0038] In step S500, the buffer is written into the log file in blocks in a regular or immediate manner according to the log security level.
[0039] In step S600 , before the DDL operation, checkpoints are created at the time when the database is closed and at regular time points.
[0040] When redo is executed in step S700, the DDL log is restored, and the DML log is restored in parallel according to the device group.
[0041] The method of this embodiment establishes a log system for DDL in database operations, establishes a log system according to device groups for DML operations, and the log systems run in parallel; formats data operations into WAL log format according to operation types, establishes a circular buffer to write to the cache of the corresponding log system, and writes the cache WAL log to the disk of the corresponding log system according to the security level; after the database is abnormally shut down and restarted, the DDL log is restored first, and then the DML log is restored in parallel according to the device group; the database transaction module rolls back data through the WAL log.
[0042] Embodiment 2:
[0043] The present invention provides a computer-readable medium on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method disclosed in Example 1.
[0044] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.
Claims
1. A time series database log storage method, characterized in that: The steps include: Establish a logging system for DDL and DML operations in database operations; Create n block ring chain table buffers for each log system; Format database operations into log format; Write the log into the log system buffer and return the total number of bytes written into the log as the current LSN. According to the log security level, the buffer is written into the log file in blocks; Before the DDL operation, a checkpoint is created at a predetermined time point, the data in the buffer is forced to be written to the disk, and the current LSN is recorded as the checkpoint LSN; When the database is restarted abnormally, the logs between the checkpoint LSN and the current LSN are read for playback. During playback, the log LSN is checked. If it is greater than the LSN in the database, redo is executed, otherwise it is skipped. When performing a rollback operation, read the LSN of the specified begin operation, use other transaction IDs as all logs of the LSN, sort them in reverse order, and perform undo one by one.
2. The time series database log storage method according to claim 1, characterized in that: A logging system is established for DML according to the device group. The logging system runs in parallel and calculates the device group through the device ID using a predetermined algorithm.
3. The time series database log storage method according to claim 1, characterized in that: The database operation is formatted into a log format to obtain insert operation logs, update operation logs, and delete log records. The insert operation log is used to insert content, the update operation log is used to record updated content and marks of overwritten content, and the delete log record is used to record marks of deleted content.
4. The time series database log storage method according to claim 1, characterized in that: The buffer is written to the log file block by block, either periodically or immediately, depending on the log security level.
5. The time series database log storage method according to claim 1, characterized in that: Before DDL operations, checkpoints are created when the database is shut down and at regular intervals.
6. The time series database log storage method according to claim 1, characterized in that: When redo is executed, DDL logs are restored and DML logs are restored in parallel according to the device group.
7. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 6.