File merging method and device

Abstract

The embodiment of the invention provides a file merging method and device. The file merging method comprises the steps of carrying out partitioning on sections in the charge of nodes in a cluster system according to key value information of user data, determining that each node meets a first triggering condition, reading at least two first files from disks of the nodes to caches corresponding to the nodes, respectively determining the target partition which each key value belongs to according to the key values of the data, corresponding to users, stored in the first files, merging the data with the same key value, and storing the merged data corresponding to the key values into the corresponding target partitions. Partitioning is carried out on the sections in the nodes, the data are merged once, and the data with the same key value are stored in the same partition. In the inquiry process, the partitions where the data are located are determined according to the key values, and then the files in the partitions are scanned. As the number of the files in each partition is small, data screening only needs to be carried out on fewer files, and therefore the read performance is improved.

Description

technical field [0001] The embodiment of the present invention relates to data communication technology, in particular to a method and device for merging files. Background technique [0002] With the continuous development of the Internet, the scale of Internet applications continues to grow, and the database storage that these applications rely on is facing increasing challenges. Traditional relational data has been difficult to meet the storage requirements of massive data, non-relational database NoSql applications, such as: BigTable developed by Google, Facebook's Cassandra are all non-relational databases. Usually, a non-relational database is a distributed system, and the data it stores is distributed on each node. At present, most non-relational databases are implemented through consistent hashing (hash). The so-called consistent hash refers to the hash function All hash values ​​form an end-to-end ring (the maximum value is connected to the minimum value), and each ...

AI Technical Summary

Problems solved by technology

In the prior art, even through the method of file merging, the number of files stored on the node is still relatively large, and each data read needs to filter data from multiple files, and the read performance of the entire database is still not high

Images