A distributed key-value storage system based on a computing offloading mechanism

Abstract

The application discloses a kind of distributed key-value storage systems based on computing offloading mechanism, including one master node and several data nodes;The computing offloading mechanism includes: when writing data, save key-value file and corresponding IP address to metadata, form key-value file-IP mapping table, update key-value file-IP mapping table when interlayer is combined, and part of the key-value file of data node is reorganized into LSM tree;For the computing query to be executed by master node, send the key to the corresponding data node according to the metadata information, and then perform the calculation in the corresponding data node, unload the computing task to the data node, and then return the data to the master node after the data node finishes processing the data, and the master node is summarized and screened. The application can reduce the transmission amount of data in the network, utilize the computing capacity of data node, and improve the read-write performance of distributed key-value storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer storage, and in particular to a distributed key-value storage system based on a computing offloading mechanism. BACKGROUND

[0002] With the advent of the big data era, data storage is increasing, and data characteristics have changed significantly, with unstructured data accounting for 80% of the total data. Under such data trends, key-value storage is widely used, which has no explicit restrictions on data structure and is highly scalable. Currently, there are some excellent key-value storage products that have emerged and are widely used, such as Leveldb used by Chrome browser, RocksDb improved by Facebook based on Leveldb, Redis and Memcached used by Twitter, etc.

[0003] Distributed architecture is a software system built on a network, which usually includes distributed file systems, distributed storage systems, distributed databases, etc. Due to the explosive growth of data in recent years, distributed architecture has been fully developed. There are many open-source and mature distributed system architectures in the industry today, such as Hadoop developed by the Apache Foundation, which includes the HDFS distributed file system and the HBase distributed key-value database, Amazon S3 (Amazon Simple Storage Service) distributed object storage developed by Amazon, which stores data in different nodes or data service centers, and the most widely used Ceph open-source distributed file system, which supports block, object, and file distributed storage.

[0004] With the increase in data volume, traditional single-machine key-value storage systems are insufficient to store such a large amount of data. Both LevelDB and RocksDB are single-machine versions of key-value storage systems. When storing data, the data is always stored on one machine. To cope with the growing data volume, many distributed key-value data systems based on LSM trees have appeared in the industry and academia, which use distributed storage to meet the growing demand for data volume, such as HBase and Cassandra. HBase is a distributed key-value storage system developed by the Apache Foundation, which provides persistent data storage and management tools using the Hadoop distributed framework and uses HDFS as the underlying storage system. Cassandra is an open-source distributed database developed by Meta, which has good scalability and has been adopted by well-known enterprises such as Digg and Twitter.

[0005] Although such distributed key-value storage systems have been widely applied, such key-value storage systems can only achieve relatively strong performance in the case of large network bandwidth. If the network bandwidth is low, the data transmission speed will occupy a long time, resulting in poor system performance. In a distributed environment, the transmission of data blocks needs to be transmitted through the network, but due to the structure of the LSM tree, the write amplification is serious, and in the case of low network bandwidth, the write amplification will cause more data transmission, resulting in poor performance. For example, when 10,000 queries are performed on a 1KB key-value pair, RocksDB needs to read 200M total data, while the actual data only needs 10M data. In the case of low bandwidth, the transmission of data will occupy a long time, thereby affecting the performance of the system. SUMMARY

[0006] In view of the problems of the prior art, the purpose of the present application is to provide a distributed key-value storage system based on a computing offloading mechanism, which can reduce the transmission amount of data in the network, utilize the computing power of data nodes, and improve the read-write performance of the distributed key-value storage system.

[0007] The present application adopts the following technical solutions:

[0008] A distributed key-value storage system based on a computing offloading mechanism includes one master node and a plurality of data nodes; the computing offloading mechanism includes:

[0009] When writing data, save the key-value file and the corresponding IP address in the metadata to form a key-value file-IP mapping table, update the key-value file-IP mapping table when merging between layers, and reorganize part of the key-value file of the data node into an LSM tree;

[0010] For a computing query to be performed by the master node, send the key to the corresponding data node according to the metadata information, and then perform computing on the corresponding data node, offload the computing task to the data node, and after the data node finishes processing the data, return the data to the master node for summarization and screening.

[0011] Preferably, the computing query is supported by a point-reading strategy, which specifically includes:

[0012] First, judge the key on the master node, record the name of all key-value files that may exist for this key, query the IP address of the key-value file through the key-value file-IP mapping table according to the name of the key-value file, encapsulate the key and the name of the key-value file, and send them to the data node at the corresponding IP address;

[0013] The data node corresponding to the IP address receives the encapsulated data, creates an iterator for each key-value file according to the key and the information of the key-value file, searches the target key-value file, encapsulates the key-value pair and the corresponding sequence number, and sends the key-value pair to the master node to complete the point reading operation.

[0014] If there is only one key-value file corresponding to the key, the key-value pair is directly sent to the master node to complete the point reading operation; if there are multiple key-value files corresponding to the key and distributed on the same data node, the multiple key-value files are filtered through the iterator query on the data node, the latest key-value pair is determined through the sequence number, and the key-value pair is sent to the master node to complete the point reading operation; if there are multiple key-value files corresponding to the key and distributed on different data nodes, each data node first performs the point reading operation, and then sends the key-value pair and the corresponding sequence number to the master node, the master node determines the new and old key-value pairs through the sequence number, selects the latest key-value pair, and completes the point reading operation.

[0015] Preferably, after the key-value storage system receives a user request, the data reading process includes:

[0016] Step 3.1, first search for the target data in the cache area, if found, go to step 3.2; if not found, go to step 3.3;

[0017] Step 3.2, the system completes the user's reading request and sends the target data read to the user;

[0018] Step 3.3, the input reading key is judged according to the key, and all possible key-value files are added to the key-value file list if the key is within the maximum key and minimum key range of a key-value file, and step 3.4 is performed; if not, go to step 3.5;

[0019] Step 3.4, the key is sent to the corresponding IP address according to the key-value file-IP address mapping table, and the corresponding data node is allowed to query the key-value, if the key-value exists, go to step 3.6; if not, go to step 3.7;

[0020] Step 3.5, skip the key-value file and continue to search for other key-value files;

[0021] Step 3.6, the data node returns the key-value pair and the corresponding sequence number to the master node;

[0022] Step 3.7, the target key-value pair is not found, and Null is returned;

[0023] Step 3.8, the master node filters according to the sequence number;

[0024] Step 3.9, the key-value storage system completes the user's reading request and sends the result to the user.

[0025] Preferably, after receiving a user write request, the data write process of the key-value storage system includes:

[0026] Step 4.1: First, write the data into the skip list structure. If the size limit is reached, proceed to step 4.2.

[0027] Step 4.2: Convert this skip list to a read-only skip list and create a new write skip list structure for data writing;

[0028] Step 4.3: Convert the read-only skip list from Step 4.2 into a key-value file and store it in the distributed file system;

[0029] Step 4.4: Record the name and IP address of the key-value file stored in Step 4.3 to form a key-value file-IP mapping table, which is then stored in the metadata information of the master node;

[0030] If no inter-layer merging occurs in step 4.5, proceed to step 4.7; otherwise, proceed to step 4.6.

[0031] Step 4.6: Read the data that has been merged between layers into the master node, discard the outdated old data, and recreate the key-value file and write it into the data node. At this time, the key-value file-IP mapping table is updated simultaneously, deleting the key-value file information that was merged between layers and writing the information into the new file.

[0032] Step 4.7: The key-value storage system completes the user's write request and sends a write success message to the user.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. This invention can reduce read data amplification in storage systems. Compared with the original distributed key-value storage system, the data to be read is changed from the original data blocks to key-value pairs, which reduces the amount of data read and thus reduces read data amplification.

[0035] 2. This invention can reduce the amount of data transmitted in the network. By using a computational offloading method, each data node reads at most the data it needs, and only that part of the data is transmitted in the network, replacing the original data block transmission, thereby reducing the amount of data transmitted in the network and improving system performance when the network bandwidth is low.

[0036] 3. This invention utilizes the computing power of data nodes, which are themselves machines with computing resources. By utilizing their computing power, the computing power of the entire cluster is enhanced, thereby improving the overall performance of the cluster. Attached Figure Description

[0037] Figure 1A structure schematic diagram of a distributed key-value storage system based on a computing offloading mechanism according to an embodiment of the present application;

[0038] Figure 2 A read flow chart of a distributed key-value storage system based on a computing offloading mechanism and a distributed point reading strategy according to an embodiment of the present application;

[0039] Figure 3 A read flow chart of a distributed key-value storage system based on a computing offloading mechanism and a distributed point reading strategy according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described below in connection with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not used to limit the scope of the present application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope of the appended claims of the present application.

[0041] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be described in detail below in connection with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0042] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. The described embodiments are merely some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0044] Reference is made to Figure 1 As shown in the drawings, the present application is a distributed key-value storage system based on a computing offloading mechanism, which comprises one master node and a plurality of data nodes; the computing offloading mechanism comprises:

[0045] When writing data, save the key-value file and the corresponding IP address in the metadata to form a key-value file-IP mapping table, update the key-value file-IP mapping table when merging between layers, and reorganize part of the key-value file of the data node into an LSM tree;

[0046] For the computing query to be executed by the master node, the key is sent to the corresponding data node according to the metadata information, and the computing is performed at the corresponding data node, the computing task is unloaded to the data node, and the data is returned to the master node after the data node finishes processing the data, and the master node performs summarization and screening.

[0047] Specifically, the computing query is supported by a point reading strategy, specifically including:

[0048] First, the key is judged on the master node, and the name of all key value files that may exist for the key is recorded, the IP address of the key value file is queried through the key value file-IP mapping table according to the name of the key value file, the key and the name of the key value file are encapsulated, and are sent to the data node at the corresponding IP address;

[0049] After the data encapsulated by the data node at the corresponding IP address is received, an iterator is created for each key value file according to the key and the information of the key value file, the target key value file is searched, and the key value pair and the corresponding sequence number are encapsulated;

[0050] If there is only one key value file corresponding to the key, the key value pair is directly sent to the master node to complete the point reading operation; if there are multiple key value files corresponding to the key and distributed on the same data node, the key value pairs searched by the iterators of the multiple key value files are screened on the data node, the latest key value pair is determined through the sequence number, and the key value pair is sent to the master node to complete the point reading operation; if there are multiple key value files corresponding to the key and distributed on different data nodes, each data node first performs the point reading operation described above, and then sends the key value pair and the corresponding sequence number to the master node, the master node determines the new and old of the key value pair through the sequence number, selects the latest key value pair, and completes the point reading operation.

[0051] As Figure 1 In the method, the distributed key value storage system based on the computing unloading mechanism includes a series of memory structures, such as a write skip list, a read-only skip list, a key value file-IP mapping table and a structure in a data node. Each node is a partial log-structured merge tree for the entire distributed key value storage system, and is a complete log-structured merge tree for the node itself.

[0052] Referring to Figure 2 Fig. 2 is a data reading flowchart of the distributed key value storage system based on the computing unloading mechanism, and specifically includes:

[0053] 101, the user sends a reading request to the system, which goes to 102;

[0054] 102, find the target data in the cache area, if found, get the target data, go to 103, otherwise, go to 104;

[0055] 103、return the query result to the user;

[0056] 104、read the data, judge all possible key-value files according to the key, if the key is within the range of the maximum key and the minimum key of a key-value file, add the key-value file to the list, and go to 105;

[0057] 105、send the key and corresponding information to the corresponding IP address according to the key-value file-IP address mapping table, and the corresponding data node queries the key to obtain the key-value pair and the sequence number, and goes to 106, if the query is empty, go to 107;

[0058] 106、return the key-value pair and the sequence number to the master node, and go to 108;

[0059] 107、the master node accepts empty data, and goes to 103;

[0060] 108、the master node filters according to the sequence number, and goes to 103.

[0061] Referring to Figure 3 the data write flowchart of the distributed key-value storage system based on the computing offloading mechanism, specifically comprising:

[0062] 201、the write request sent by the user to the system, and goes to 202;

[0063] 202、determine whether the write skip list is full, if not, go to 211, otherwise, go to 203;

[0064] 203、convert the write skip list to a read-only skip list, and go to 204;

[0065] 204、determine whether the size of the cache area is full, if not, go to 205, otherwise, go to 206;

[0066] 205、create a new write skip list, and go to 202;

[0067] 206、convert the read-only skip list to a key-value file, and transfer the key-value file to the distributed file system, and go to 207;

[0068] 207、save the key-value file and the node information stored therein to form a key-value file-IP mapping table, and save it in the metadata information of the master node, and go to 208;

[0069] 208、if inter-layer merging occurs, go to 209; otherwise, go to 211;

[0070] 209、read the data that occurs inter-layer merging to the master node, discard the obsolete old data and re-form the key-value file to transfer to the data node, and go to 210;

[0071] 210. updating the key-value file-IP mapping table, deleting the information of the key-value file which is merged between layers and writing the information of the new file, and going to 211;

[0072] 211. writing data into the write skip list, and going to 212;

[0073] 212. returning the write result to the user.

[0074] It is apparent for a person skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that it can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the present application being defined by the appended claims rather than the above description, and it is intended that all changes and modifications that come within the meaning and range of equivalency of the claims are to be embraced by the application. Any reference signs in the claims should not be construed as limiting the claims to the figures in which the reference signs are used. Further, it is explicitly stated that the word comprising does not exclude other elements not listed and the singular does not exclude the plural. Multiple units or systems recited in a system claim can also be implemented by one unit or system by means of software or hardware.

[0075] Finally, it should be noted that the above embodiments are merely used to describe the technical solutions of the present application, rather than limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit of the present application.

Claims

1. A distributed key-value storage system based on a compute offload mechanism, comprising one master node and a plurality of data nodes; characterized in that, The computing offloading mechanism comprises: Saving the key-value file and the corresponding IP address in the metadata when writing data, forming a key-value file-IP mapping table, updating the key-value file-IP mapping table when merging between layers, and reorganizing part of the key-value file of the data node into an LSM tree; For a computing query to be executed by the master node, the key is sent to the corresponding data node according to the metadata information, and then the computing task is offloaded to the data node for computing, and after the data node finishes processing the data, the data is returned to the master node for summarizing and screening; The computing query is supported through a point-reading strategy, specifically comprising: First, the key is judged on the master node, and the names of all key-value files possibly containing the key are recorded, the IP address of the key-value file is queried through the key-value file-IP mapping table according to the name of the key-value file, the key and the name of the key-value file are encapsulated, and are sent to the data node at the corresponding IP address; After the data node at the corresponding IP address receives the encapsulated data, an iterator is created for each key-value file according to the key and the information of the key-value file, the target key-value file is searched, and the key-value pair and the corresponding sequence number are encapsulated; If there is only one key-value file corresponding to the key, the key-value pair is directly sent to the master node to complete the point-reading operation; if there are multiple key-value files corresponding to the key and they are distributed in the same data node, the key-value pairs searched through the iterators of the multiple key-value files are screened on the data node, the latest key-value pair is determined through the sequence number, and the key-value pair is sent to the master node to complete the point-reading operation; if there are multiple key-value files corresponding to the key and they are distributed in different data nodes, each data node first performs the point-reading operation, and then sends the key-value pair and the corresponding sequence number to the master node, the master node determines the latest key-value pair through the sequence number, and completes the point-reading operation.

2. The distributed key-value storage system based on a compute offload mechanism according to claim 1, characterized in that, After the key-value storage system receives a user request, the data reading process comprises: Step 3.1, first searching for the target data in the cache area, if found, going to step 3.2; if not found, going to step 3.3; Step 3.2, the system completes the user's reading request, and sends the target data read to the user; Step 3.3, the key is inputted for reading, and all possible key-value files are determined according to the key, if the key is within the maximum key and the minimum key of a key-value file, the key-value file is added to the list, and goes to step 3.4; if not, goes to step 3.5; Step 3.4, the key is sent to the corresponding IP address according to the key-value file-IP address mapping table, and the data node corresponding to the IP address queries the key-value, if the key-value exists, goes to step 3.6; if not, goes to step 3.7; Step 3.5, skipping the key-value file and continuing to search for other key-value files; Step 3.6, the data node returns the key-value pair and the corresponding sequence number to the master node; Step 3.7, the target key-value pair is not found, and Null is returned; Step 3.8, the master node screens according to the sequence number; Step 3.9, the key-value storage system completes the user's reading request, and sends the result to the user.

3. The distributed key-value storage system based on a compute offload mechanism according to claim 1, characterized in that, The data writing procedure of the key-value storage system after receiving the user writing request comprises: Step 4.1, first writing in the skip list structure, if the size limit is reached, go to step 4.2; Step 4.2, converting the skip list into a read-only skip list, and creating a new writing skip list structure for data writing; Step 4.3, converting the read-only skip list of step 4.2 into a key-value file and storing in the distributed file system; Step 4.4, recording the name and IP address of the key-value file stored in step 4.3 to form a key-value file-IP mapping table, and saving in the metadata information of the master node; Step 4.5, if no inter-layer merging occurs, go to step 4.7; otherwise, go to step 4.6; Step 4.6, reading the data of which inter-layer merging occurs to the master node, discarding the obsolete old data and re-forming the key-value file and writing to the data node, at this time, the key-value file-IP mapping table is updated simultaneously, deleting the key-value file information of which inter-layer merging occurs and writing the information of the new file; Step 4.7, the key-value storage system completes the user writing request, and sends the writing success information to the user.

Images