sstable file storage method and device

Abstract

The invention relates to an SSTable file storage method and device and is used for solving the problem of larger disk space for storing SSTable files. The method comprises the following steps: reading a plurality of SSTable files before combination, wherein each SSTable file before combination includes key value-marked row data; determining key value intervals corresponding to the SSTable files before combination, wherein the key value intervals corresponding to the SSTable files before combination are defined by a maximum key value and a minimal key value included in the SSTable files before combination; dividing the key value intervals corresponding to the SSTable files before combination into a plurality of subintervals, wherein each subinterval corresponds to a preset subdirectory; combining the row data with the same key values in the SSTable files before combination, storing the combined row data as a plurality of combined SSTable files according to the corresponding relations between the key values and the subintervals, and storing the combined SSTable files into the subdirectories of the corresponding subintervals respectively.

Description

technical field [0001] The invention relates to the technical field of network communication, in particular to an SSTable file storage method and device. Background technique [0002] SSTable (Sorted String Table) files can efficiently store data in memory to disk in the form of KEY-VALUE. Specifically, whenever the data written in the memory reaches a certain size, it will be flushed to the disk at one time to form one or more SSTable files. Among them, the situation of generating multiple SSTable files at one time mainly occurs when the amount of data to be stored exceeds the preset size of a standard SSTable file, otherwise, only one SSTable file is generated at a time. Among them, each SSTable file contains multiple rows of data. Each row of data is identified by a key value, which is equivalent to KEY in the KEY-VALUE structure; the data content stored in each row of data is equivalent to VALUE in the KEY-VALUE structure. Moreover, each line of data in each SSTable f...

AI Technical Summary

Problems solved by technology

[0005] However, the above-mentioned method has the following obvious defects: all SSTable files in the disk need to be opened at the same time when merging storage each time, so when the SSTable files in the disk occupy more than 1 / 2 of the entire disk space, the above-mentioned cannot be performed. Merge Storage Jobs

For example, assuming that the total size of the disk space is 15T, and 10T of SSTable files have been stored in the current disk space, at this time, if these SSTable files are to be opened at the same time, the disk needs to provide an additional 10T of cache space, which only It can only be completed when the total size of the disk is not less than 20T. It can be seen that the existing combined storage method of SSTable files has great limitations, and the requirements for disk space are relatively high.

Moreover, even if the disk space is sufficient, since the above operations need to occupy many system processes at the same time, the risk of operation failure or crash is high

[0006] In addition, in the existing merged storage method, each merged SSTable file still contains all the key values. Therefore, the number of key values ​​of the merged SSTable files has not been simplified, resulting in the above-mentioned problems in each merge process. many flaws

Moreover, just because the number of key values ​​of the SSTable files has not been simplified after merging, all SSTable files must be opened each time the SSTable files are scanned or read, so the cost of scanning or reading operations cannot be significantly reduced. workload

Images