Key value storage-oriented near data computing cluster system

Abstract

The invention provides a Key value storage-oriented near-data computing cluster system. The near-data computing cluster system comprises a host end and a plurality of NDP devices. The host end is connected with each NDP device. The host end comprises a cluster equipment management module, a file distribution module and a file migration module; the file migration module is used for acquiring compression requirements of each compression storage layer of each NDP device; and the file migration module is used for migrating the to-be-compressed file to the NDP equipment of the corresponding storagekey value interval according to the key value range for storage and compression sorting. The computable storage array is composed of the multiple NDP devices, the storage capacity and the computing capacity of the whole system are guaranteed, meanwhile, the situation that the storage space of the host side is occupied is avoided, the host cpu bottleneck of database compression sorting operation is greatly relieved, and therefore the data processing efficiency of the whole system can be improved.

Description

technical field [0001] The invention relates to the technical field, in particular to a near-data computing cluster system oriented to key-value storage. Background technique [0002] The rapid development of computer technology and the Internet has led to the emergence of semi-structured and unstructured data, and the proportion of the total data is getting higher and higher, showing exponential growth. However, traditional relational databases cannot meet the high-efficiency storage, high concurrency, and high scalability requirements of massive data in an environment of increasing unstructured data scale. In contrast, key-value storage does not require a predefined data structure, and has been widely used in unstructured data storage and management to provide low-latency read and write speeds and support massive data traffic. At present, the key-value storage system widely uses the log structure merge tree (LSM-tree) to realize data storage and management, and convert ra...

AI Technical Summary

Problems solved by technology

However, the compression operation will occupy a large amount of I / O bandwidth on the host side and the storage device side, resulting in performance degradation. Updating the file table will also cause the problem of increasing the amount of written data.

Images