A method and device for managing data sets

Abstract

The invention aims to provide a method and a device for managing a data set. The method comprises the steps of determining an optimal storage device corresponding to a data subset by a management device according to subset identification information corresponding to the data subset when a trigger condition for carrying out sharding processing on the data set is met; updating storage information corresponding to the subset identification information according to storage information, which corresponds to the data subset, in a current storage device and by combining related information of the optimal storage device. Compared with the prior art, the storage information corresponding to the subset identification information is updated through the subset identification information corresponding to the data subset by combining the related information of the optimal storage device, thus segmental multi-stage database sharding on a target data subset can be realized, the problem that regular user data are subjected to in-place expansion can be solved, the uniform distribution of data is ensured, sub databases can be simply maintained, the expandability and the load balancing of the sub databases are realized, the sharding is flexible, and the cost is reduced.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a technology for managing data sets. Background technique [0002] In order to cope with the continuous and rapid growth of the user database, the original database needs to be fragmented to obtain higher throughput, better performance and larger storage capacity. [0003] At present, for the database sub-database, the method of sub-section modulo is mainly used. For example, since in most cases, the user ID (userid) is the foreign key of all user-related data, splitting the database according to the user ID can effectively avoid cross-database transactions and cross-database connection tables, making the same user Data is hashed on unique shards. When the userid is used as the segmentation standard, when the userid is less than an upper limit value M, take the modulus according to N, and hash the userid to 0, 1, ..., N-1, a total of N sub-databases; if the data volume con...

AI Technical Summary

Problems solved by technology

[0005] However, the current technical solution can well disperse the pressure from new users to the new database, but it cannot cope with the in-situ expansion of old user data

For userids smaller than the limit value M, the data growth has been pressed on the N machines from 0 to N-1, and cannot be dispersed

Images