A real-time database data hot spot balancing method, device, equipment and medium

Abstract

The present application discloses a real-time database data hot spot equalization method, device, device and medium. The method includes: constructing a tag Huffman tree according to the frequency of occurrence of each letter in the tag name set; calculating the weighted path length of the target tag name corresponding to the real-time data according to the tag Huffman tree; Based on the weighted path length, the tag Huffman code corresponding to the target tag name, and the timestamp inversion value of the real-time data, the row key identifier of the real-time data is generated; the real-time data and the real-time data are combined with The row key identifier is written into the HBase database. The tag Huffman tree is constructed by the Huffman algorithm, the weight of the target tag name is determined according to the tag Huffman tree, and the row key is optimized by combining the Huffman code corresponding to the target tag name and the timestamp inversion value. , which can realize the balanced distribution of data to each node in the write operation, and improve the speed of data filtering in the read operation, thereby improving the performance of the real-time database system.

Description

technical field [0001] The invention relates to the field of computers, in particular to a method, device, equipment and medium for balancing data hotspots in a real-time database. Background technique [0002] At present, in the field of industrial control, the real-time database system needs to store large-scale real-time data as an important basis for production control, production analysis, production optimization and other processes. The real-time database is connected to all the collectors in the entire factory. These collectors are usually on-site. Uninterrupted collection, so the overall required storage is in units of TB or even PB. Through the HBase storage system, the storage problem of massive data can be effectively solved, but it also introduces how to distribute these real-time data evenly among storage nodes. And the technical problem of being able to read and write reliably and efficiently. Improper data storage will cause data hotspots, that is, the target ...

AI Technical Summary

Problems solved by technology

[0007] Among the above three methods, based on the way of adding salt to generate row keys, for read operations, since the client cannot reconstruct the complete row keys according to the query conditions to achieve fast filtering, it needs to scan the entire table for matching, which reduces the real-time database read performance

The way of generating row keys based on hash, because the hash algorithm has a certain probability of collision, resulting in the same hash value of different tag names, so the uniqueness of the row key cannot be guaranteed

The method of generating key values ​​through parameter inversion, because the rows in HBase are sorted according to the dictionary order of key values, that is, words based on alphabetical order are arranged in alphabetical order, parameter inversion sacrifices the order of row keys, reducing the Improve the efficiency of data scanning

Images