Data splitting method and device for a distributed storage system

Abstract

The embodiment of the invention provides a data splitting method and device for a distributed storage system. The method comprises the steps that whether query per second (QPS) of all data fragments is greater than a first preset threshold value is monitored; the data fragment with the QPS greater than the first preset threshold value is determined as a target data fragment, access requests of the target data fragment are sampled, and a sampled data stream of index key values corresponding to the sampled access requests is formed; whether the QPS of the target data fragment is greater than a second preset threshold value is judged; if yes, a median of the index key values in the sampled data stream in each preset time period is determined; a median of the index key values in a next preset time period corresponding to the current moment is obtained through prediction according to the determined median, and the median obtained through prediction is determined as a splitting point; and the target data fragment is split according to the splitting point. When the data fragments on access request hotspots are split through the scheme, load balance of the access requests can be realized.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a data splitting method and device for a distributed storage system. Background technique [0002] In the era of big data, the storage of massive data is a key technology. With the increase of data scale, when the QPS (Query Per Second, query rate per second) of a single storage node is too large to bear the pressure of access, it is necessary to adopt a distributed storage solution and use data sharding technology to Data is distributed to different storage nodes to eliminate hotspots of access requests and achieve load balancing. Wherein, the hotspot of the access request refers to the storage node corresponding to the data fragment whose QPS is too large, and the storage node refers to the storage server, which may be a physical server or a virtual server. [0003] Among them, the traditional relational database and various Nosql (Not Only SQL, non-relational database) ...

AI Technical Summary

Problems solved by technology

[0005] However, in practical applications, access to data is not uniform. That is to say, even if the index-based splitting strategy can ensure an even division of the data volume, the requested access may still be concentrated on some of the data. The distribution of on-chip data volume may not be consistent with the distribution of access requests, which makes the index-based splitting strategy invalid and cannot solve the hotspot problem, resulting in unbalanced load of access requests

Images