File merging method and device based on cassandra database

Abstract

The invention relates to the field of databases, in particular to a file merging method and device based on a cassandra database. The method mainly comprises the steps of receiving a data file generated by a database, and generating a merged file list of each disk; obtaining a merged file list of the corresponding disk by the merging process of each disk, and obtaining the size of a data file needing to be merged in the merged file list of each disk; starting; and in the parallel merging process of the database, when the sum of the sizes of the data files acquired by the merging process of each disk reaches a merging file threshold value, merging the data files needing to be merged in all the disks at one time by the parallel merging process. According to the method, the small files can bemerged in time under the condition that few merging layers and temporary files are used, the merging frequency is reduced, the occupied space of the files in the disk is reduced, the disk IO frequency and disk IO competition are reduced, the file merging performance is improved, and the read-write stability of the database is improved.

Description

Technical field

[0001] The invention relates to the field of databases, in particular to a method and device for merging files based on a cassandra database.

Background technique

[0002] The Cassandra database is an open source distributed hybrid storage solution with features such as decentralization, scalability, high availability, fault tolerance, and configurable consistency. When cassandra flushes the cached data to the disk sequentially, it will generate multiple data files (Sorted String Table, abbreviated as: sstable) of about 1-10MB. When the cassandra database processes a large amount of data, the huge number of files will seriously affect the stability of the database and slow down the query speed.

[0003] In order to reduce the number of files to be processed, the cassandra database provides a file merging (Compaction) mechanism to merge multiple large files into a small number of small files. Currently commonly used file merging strategies are:

[0004] ...

Examples

Embodiment 1

[0034] In the cassandra database, when the client writes data, the client program determines the server node to which the data should be sent according to the token range on the cluster, and the server accepts data in parallel with multiple threads, and each thread will do the data received by itself. Sort and generate data files smaller than 10M. In some implementation scenarios, when the amount of data processed by a single machine of the Cassandra database reaches 4TB / day, the number of data files that the database process needs to open will exceed 200,000. When data storage is required for 7 days, the number of data files that the process needs to open will reach 1.4 million. The data files need to be merged to reduce the number of files. Therefore, this embodiment provides a new dynamic small file merging method, which avoids the defects existing in the existing file merging strategy.

[0035]Cassandra is a NoSQL distributed database that adopts the Log Structured Merge ...

Embodiment 2

[0076] On the basis of the method for merging files based on the cassandra database provided in the above-mentioned embodiment 1, the present invention also provides a device for merging files based on the cassandra database that can be used to implement the above method, such as Figure 5 Shown is a schematic diagram of the device architecture of the embodiment of the present invention. The apparatus for merging files based on a cassandra database in this embodiment includes one or more processors 21 and a memory 22 . in, Figure 5 A processor 21 is taken as an example.

[0077] Processor 21 and memory 22 can be connected by bus or other means, Figure 5 Take connection via bus as an example.

[0078] Memory 22, as a non-volatile computer-readable storage medium based on the cassandra database-based file merging method, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the one based on File merging method f...

Technical field

[0001] The invention relates to the field of databases, in particular to a method and device for merging files based on a cassandra database.

Background technique

[0002] The Cassandra database is an open source distributed hybrid storage solution with features such as decentralization, scalability, high availability, fault tolerance, and configurable consistency. When cassandra flushes the cached data to the disk sequentially, it will generate multiple data files (Sorted String Table, abbreviated as: sstable) of about 1-10MB. When the cassandra database processes a large amount of data, the huge number of files will seriously affect the stability of the database and slow down the query speed.

[0003] In order to reduce the number of files to be processed, the cassandra database provides a file merging (Compaction) mechanism to merge multiple large files into a small number of small files. Currently commonly used file merging strategies are:

[0004] ...

Examples

Embodiment 1

[0034] In the cassandra database, when the client writes data, the client program determines the server node to which the data should be sent according to the token range on the cluster, and the server accepts data in parallel with multiple threads, and each thread will do the data received by itself. Sort and generate data files smaller than 10M. In some implementation scenarios, when the amount of data processed by a single machine of the Cassandra database reaches 4TB / day, the number of data files that the database process needs to open will exceed 200,000. When data storage is required for 7 days, the number of data files that the process needs to open will reach 1.4 million. The data files need to be merged to reduce the number of files. Therefore, this embodiment provides a new dynamic small file merging method, which avoids the defects existing in the existing file merging strategy.

[0035]Cassandra is a NoSQL distributed database that adopts the Log Structured Merge ...

Embodiment 2

[0076] On the basis of the method for merging files based on the cassandra database provided in the above-mentioned embodiment 1, the present invention also provides a device for merging files based on the cassandra database that can be used to implement the above method, such as Figure 5 Shown is a schematic diagram of the device architecture of the embodiment of the present invention. The apparatus for merging files based on a cassandra database in this embodiment includes one or more processors 21 and a memory 22 . in, Figure 5 A processor 21 is taken as an example.

[0077] Processor 21 and memory 22 can be connected by bus or other means, Figure 5 Take connection via bus as an example.

[0078] Memory 22, as a non-volatile computer-readable storage medium based on the cassandra database-based file merging method, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the one based on File merging method f...

Images