Distributed database transaction data storage optimization method

Abstract

The invention relates to the technical field of distributed databases, and particularly provides a distributed database transaction data storage optimization method. The method comprises the followingsteps of S1, adding a semi-solidified memory table; S2, the semi-solidified memory table is composed of a memory Block and an SST file; S3, merging the Block data of the memory, and executing Compaction operation; S4, when the memory Block of the semi-solidified memory table is empty, removing the memory table from the storage engine; S5, setting a longest life cycle time threshold value for thesemi-solidified memory table; S6, after the system breaks down and is restarted, recovering the data of the read-only memory table according to the WAL; S7, providing a read interface the same as thatof the read-only memory table for the outside by the semi-solidified memory table; S8, after receiving the reading request, ensuring that the latest written data is preferentially read. Compared withthe prior art, the method has the advantages that the writing intention data can reside in the memory until the transaction is ended or overtime, so that the persistent data volume of the disk is reduced, and the data storage performance is optimized.

Description

technical field [0001] The invention relates to the technical field of distributed databases, and specifically provides a method for optimizing transactional data storage of distributed databases. Background technique [0002] CockroachDB (CRDB) is an open source distributed database system based on the idea of ​​Google Spanner. It has the ability of NoSQL to store and manage massive data, and maintains the characteristics of ACID and SQL supported by traditional databases. [0003] CockroachDB's storage layer RocksDB engine adopts the storage structure of LSM (log structure tree). The data of the write request is first written into the memory table. After the memory table data is full of 64MB (default), it will be persisted to the SST (SortedSequence Table- Sort Sequence List) file. The SST files on the disk are organized in a hierarchical manner, and there are background threads to perform layer-by-layer merge operations. [0004] In the transaction processing of Cockroa...

AI Technical Summary

Problems solved by technology

However, due to the layer-by-layer merging process of the LSM structure, DELETE cleaning may need to go to the last layer to ensure that the writing intent data is deleted from the underlying file, thus occupying a large amount of unnecessary storage space, and the data generated during the multiple merging of writing intent data write amplification problem

Images