KV storage system optimization method for executing parallel tasks based on multi-NDP cooperative host end

Abstract

The invention provides a KV storage system optimization method for executing parallel tasks based on a multi-NDP cooperative host end. The method comprises the steps that firstly, a plurality of NDP devices are arranged in a storage system to form a multi-NDP computable storage array, each NDP device is provided with a corresponding key value threshold value, the NDP devices are sorted according to the key value threshold values, and key value intervals of the NDP devices are obtained; the host terminal distributes the obtained storage data to each NDP device for storage; when a compression task is triggered, a file participating in compression is acquired firstly, and the file participating in compression is allocated to corresponding NDP equipment to be compressed and stored in combination with the key value range and the key value threshold value of the file participating in compression. According to the method, through dynamic segmentation and allocation of the compression tasks, the effect of multi-CPU cooperation of the host end CPU and the NDP equipment array is achieved, the transverse expansion capacity of the NDP equipment and the key value storage system is achieved, andstorage resources and computing resources are expanded through the NDP equipment array.

Description

technical field [0001] The invention relates to the technical field of data compression, in particular to a method for optimizing a KV storage system based on multi-NDP cooperative hosts executing parallel tasks. Background technique [0002] The rapid development of computer technology and the Internet has led to the emergence of semi-structured and unstructured data, and the proportion of the total data is getting higher and higher, showing exponential growth. However, traditional relational databases cannot meet the high-efficiency storage, high concurrency, and high scalability requirements of massive data in an environment of increasing unstructured data scale. In contrast, key-value storage does not require a predefined data structure, and has been widely used in unstructured data storage and management to provide low-latency read and write speeds and support massive data traffic. At present, the key-value storage system widely uses the log structure merge tree (LSM-t...

AI Technical Summary

Problems solved by technology

However, these optimization solutions are only for a single NDP device, and in the face of today's massive data processing, a single NDP device can no longer meet the user's storage and processing needs

[0006] It can be seen that the existing research methods based on near-data processing do not take full advantage of the computing power of the host-side CPU and the requirements of storage device arrays.

Images