Data processing method, data processing device, data processing equipment and computer-readable storage medium

Abstract

The invention provides a data processing method, a data processing device, data processing equipment and a computer-readable storage medium. The method includes: calculating the number of needed datatables, which are used for saving received customer label data, according to the amount of the customer label data, and creating the corresponding number of data tables; constructing a consistency hash ring, and uniformly mapping the data tables to the consistency hash ring; acquiring primary-key fields of all the the customer label data; utilizing a consistency hash algorithm to carry out hash processing on the primary-key fields to obtain hash values corresponding to the customer label data, wherein one hash value is equal to one space value on the consistency hash ring; determining a data table, which is used for storing corresponding customer label data, according to the hash values; and saving the customer label data in the data table. According to the method, the large amount of customer label data can be uniformly stored into all the data tables through a manner of hash table-dividing.

Description

technical field [0001] The present invention relates to the technical field of information processing, in particular to a data processing method, device, equipment and computer-readable storage medium. Background technique [0002] Projects often encounter situations where a large amount of data (ranging from hundreds of millions to billions) needs to be written into the database. The general method is to write stored procedures. [0003] At present, there is a method of storing large amounts of data in batches. The storage nodes are mapped to the consistent hash ring through the consistent hash algorithm, and the storage nodes are searched clockwise on the consistent hash ring according to the hash value of each piece of data. The data storage node finally writes the data to the corresponding storage node. [0004] The hash value after hashing of each piece of data has the characteristics of random distribution in the consistent hash ring, that is to say, the distribution ...

AI Technical Summary

Problems solved by technology

[0004] The hash value after hashing of each piece of data has the characteristics of random distribution in the consistent hash ring, that is to say, the distribution of the hash value after data hash on the consistent hash ring may appear sparse and dense , so that if you look for storage nodes in a clockwise direction to store data, it will easily lead to uneven data storage; in addition, when the number of storage nodes does not exceed 10, the storage nodes are mapped to the consistent hash ring according to the consistent hash algorithm In addition, the mapping of storage nodes on the consistent hash ring is not necessarily uniform. In this way, even if the hash value of the data is evenly distributed on the consistent hash ring, the data saved by the storage node that finally saves the data The amount is not necessarily uniform

Images