A method for merging hash key-value data requests in key-value storage

Abstract

The present invention discloses a method for merging hash key-value data requests for key-value storage. The processing method is mainly divided into four parts: the definition of hash key-value data requests for key-value storage, the definition of merge status, the request merging and the update of merge status, and the conversion of merge status to copy read and write requests. The method merges and copies requests for the same key, thereby reducing the amount of copied data in "hot spot" scenarios, thereby reducing the network bandwidth occupancy, avoiding the waste of network bandwidth resources, and reducing the load of data copying, thereby improving the performance of data read and write services. In addition, the method can be applied to key-value storage that supports hash key-value data and has a certain degree of versatility.

Description

Technical Field

[0001] The present invention relates to fields requiring high-frequency real-time information processing and storage, such as real-time risk control, anti-fraud and other financial fields, and relates to a method for merging hash key-value data requests for key-value storage. Background Art

[0002] In areas requiring high-frequency, real-time information processing and storage, such as risk control and anti-fraud in the financial sector, large amounts of data, such as financial indicator data, need to be stored. This data is typically stored in key-value stores, such as Redis and Aerospike. Key-value stores like Redis and Aerospike support a hash key-value data structure, meaning that a key (hereinafter referred to as key) can be used to retrieve a value (hereinafter referred to as value). This value is a hash key-value table that stores several key-value pairs (hereinafter referred to as bin key and bin value). Financial indicator data can be stored in this hash key-value table structure.

[0003] Because large amounts of data are stored, a large number of data read and write requests are generated. Furthermore, in real-world scenarios, these large numbers of data read and write requests can create a "hotspot" phenomenon, where a large number of read and write requests act on only a small portion of the data. Furthermore, in real-world scenarios, to ensure high availability and fault tolerance of services, stored data needs to be replicated, so a large number of data read and write requests are also replicated. For example, Redis and Aerospike replicate data read and write requests from the master node to the slave nodes, thereby replicating the master node's data to the slave nodes.

[0004] However, when performing hash key-value data replication, key-value stores such as Redis and Aerospike simply copy all original data read and write requests from the primary node to the secondary node. When "hotspots" occur, the following two problems may arise:

[0005] (1) A large amount of request data will be sent from the master node to the slave node, which increases the load caused by data replication on the master and slave nodes and reduces the performance of data reading and writing services;

[0006] (2) The duplication of a large amount of request data will lead to the occupation of a large amount of network bandwidth, resulting in a waste of network bandwidth resources.

[0007] However, existing key-value storage systems rely on the simple read and write request replication method described above for hash key-value data replication, which fails to address the two aforementioned issues. The resulting degradation in read and write service performance and the significant waste of network bandwidth significantly impacts service quality in high-frequency, real-time information processing and storage applications (such as real-time risk control), making it difficult to meet business performance requirements. Summary of the Invention

[0008] In response to the problems and defects caused by the above-mentioned hash key-value data replication method, the present invention proposes a hash key-value data request merging method for key-value storage. When a "hotspot" phenomenon occurs, requests for the same key are merged, thereby reducing the amount of copied request data, reducing the load of data replication, and reducing the network bandwidth occupancy, thereby meeting the performance requirements in the field of high-frequency real-time information processing and storage.

[0009] The object of the present invention is achieved through the following technical solution: a method for merging hash key-value data requests for key-value storage, comprising the following steps:

[0010] (1) Definition of hash key-value data request for key-value storage: Since read requests do not modify data, this definition includes the following four types of requests, which only include write operations on data; the details are as follows:

[0011] a) REPLACE operation: delete the old value of the hash key-value table under key and replace it with the hash key-value table of the parameter binKeyValues ​​in the write request operation. If the key does not exist, a new one is created;

[0012] b) SET operation, updates the hash key value table under key;

[0013] c) DELETE operation, deletes the key-value pair in the hash key-value table under key, where the key exists in the binKeys list of the delete operation parameter; if the key does not exist, an error is returned; if the hash key-value table under key is empty after deletion, it is recorded as a REMOVE operation;

[0014] d) REMOVE operation, deletes all key-value pairs in the hash key-value table under key; if key does not exist, an error is returned;

[0015] (2) Definition of Merge Status: The merge status contains the following fields:

[0016] a) key: the key of key-value storage;

[0017] b) addKeys: the list of keys that need to be added to the hash key-value table under the key key;

[0018] c)delKeys: a list of keys that need to be deleted from the hash key-value table under key key, whose intersection with the addKeys list is empty;

[0019] d) data: a key-value table that records the key-value pairs that need to be added to the hash key-value table under the key;

[0020] e) state: Merge state identifier, used to process request merge logic and convert merged state into replicated read / write requests. The definition and meaning of the merge state identifier and the replicated read / write requests after conversion are as follows:

[0021] NONE, indicating an empty state, where no converted copy read or write requests are performed;

[0022] REPLACE: indicates that a request needs to be sent to replace the hash key table during replication. The converted replication read and write request is one REPLACE request.

[0023] SET, indicating that a request needs to be sent to update the hash key table during replication. The converted replication read and write request is a SET request.

[0024] DELETE: indicates that a request to delete part of the hash key table is required during replication. The converted replication read and write request is a DELETE request.

[0025] REMOVE: indicates that a request needs to be sent to delete the entire hash key-value table during replication. The converted replication read and write request is one REMOVE request.

[0026] COMPOSITE: indicates that a request needs to be sent during replication, which not only updates the key-value pairs in the hash key-value table but also deletes part of the hash key-value table. The converted replication read and write request is one SET request and one DELETE request.

[0027] (3) Request merging and merging status update: According to different types of requests and current merging status, corresponding to different operation request merging processing logic, update the merging status,

[0028] (4) Conversion from merge state to replicated read / write request: According to the definition of the merge state identifier, the merge state is converted into a replicated read / write request; for REPLACE and SET requests, the merged state data needs to be carried as the hash key value table for replacement and update; for DELETE requests, the delKeys list needs to be carried to indicate the key-value pairs of the hash key value table that need to be deleted from the data processing node.

[0029] Furthermore, in step (1), the SET operation is specifically as follows: if the key-value pair in the hash key-value table of the old value exists in the binKeyValues ​​hash key-value table in the parameter, it is replaced with the new value; if the key-value pair in the hash key-value table of the old value does not exist in the binKeyValues ​​hash key-value table, the old value is retained; if the key-value pair in the binKeyValues ​​hash key-value table does not exist in the old value, a new key-value pair is added; if the key does not exist, the same REPLACE operation is performed.

[0030] Furthermore, in step (3), the merge request and the merge status update are as follows:

[0031] a) REPLACE request: Set the merge flag to REPLACE, clear the fields addKeys, delKeys, and data in the merge state, set them to the default initial values, and then set new addKeys and data based on the hash key data carried in the request;

[0032] b) SET request: If the old merge flag is REMOVE, it is processed in the same way as a REPLACE request; otherwise, first add the key to be added to the hash key value data to addKeys, delete the key corresponding to delKeys, then update the data, and finally update the merge flag; if the old merge flag is NONE, it is updated to a SET operation; if the old merge flag is DELETE or COMPOSITE, assuming delKeys is empty, it is updated to SET, otherwise it is set to the COMPOSITE state flag to facilitate subsequent processing based on the flag;

[0033] c) DELETE request: First, remove the key to be deleted from addKeys and the key-value pair data to be deleted from data. Then, if the old merge status is not REPLACE, add the key to be deleted to delKeys. Finally, update the merge flag.

[0034] d) REMOVE request: Set the merge flag to REMOVE and clear addKeys, delKeys, and data.

[0035] Furthermore, in sub-step c) of step (3), the specific process of updating the merge identifier is: if the old merge identifier is NONE, it is updated to DELETE; if the old merge identifier is REPLACE and addKeys is empty, it is updated to REMOVE; if the old merge identifier is SET or COMPOSITE, if addKeys is empty, it is updated to DELETE, otherwise it is updated to COMPOSITE.

[0036] Beneficial effects of the present invention:

[0037] (1) This method merges and copies requests with the same key, thereby reducing the amount of copied data in "hotspot" scenarios, thereby reducing network bandwidth usage and avoiding waste of network bandwidth resources;

[0038] (2) This method reduces the amount of replicated data in the "hotspot" scenario, thereby reducing the load caused by replication and improving the performance of data reading and writing services.

[0039] (3) This method can be applied to key-value storage that supports hash key-value data, such as Redis, Aerospike, etc., and has good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 Schematic diagram of the SET request merge processing flow;

[0042] Figure 2 Schematic diagram of the merged processing flow for DELETE requests;

[0043] Figure 3 Schematic diagram of request merging of hash key-value data in an embodiment. DETAILED DESCRIPTION

[0044] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0045] In order to implement the hash key-value data request merging method for key-value storage and achieve the above-mentioned beneficial effects, the present invention needs to be implemented through the following steps:

[0046] (1) Definition of hash key-value data request for key-value storage: Since read requests do not modify data, this definition includes the following four types of requests, which only include write operations on data.

[0047] a) REPLACE(key,binKeyValues): This is called a REPLACE operation. It deletes the old value in the hash key-value table under key and replaces it with the binKeyValues ​​hash key-value table in the parameter. If key does not exist, it is created.

[0048] b) SET(key,binKeyValues): This is recorded as a SET operation. It updates the hash key-value table under key. If the key-value pair in the hash key-value table of the old value exists in the binKeyValues ​​hash key-value table in the parameter, it is replaced with the new value. If the key-value pair in the hash key-value table of the old value does not exist in the binKeyValues ​​hash key-value table, the old value is retained. If the key-value pair in the binKeyValues ​​hash key-value table does not exist in the old value, a new key-value pair is added. If key does not exist, the same REPLACE operation is performed.

[0049] c) DELETE(key,binKeys): This is recorded as a DELETE operation, which deletes the key-value pair in the hash table under key that exists in the binKeys list. If the key does not exist, an error is returned; if the hash table under key is empty after deletion, it is recorded as a REMOVE operation.

[0050] d) REMOVE(key): This is a REMOVE operation that deletes all key-value pairs in the hash table under key. If key does not exist, an error is returned.

[0051] (2) Definition of merge status: For hash key-value data request merge method, it is necessary to define the merge status and maintain the merge status for each key. The merge status contains the following fields:

[0052] a) key: the key of key-value storage;

[0053] b) addKeys: the list of keys that need to be added to the hash key-value table under the key key;

[0054] c)delKeys: a list of keys that need to be deleted from the hash key-value table under key key, whose intersection with the addKeys list is empty;

[0055] d) data: a key-value table that records the key-value pairs that need to be added to the hash key-value table under the key;

[0056] e) state: Merge state identifier, used to process the request merge logic and to convert the merge state into a replicated read / write request. The definition and meaning of the merge state identifier and the replicated read / write request after conversion are shown in Table 1.

[0057] Table 1 Definition, meaning and replication read and write requests of merge status identifier

[0058]

[0059] (3) Request merging and merging status update: Based on different types of requests and the current merging status, the following request merging processing logic is used to update the merging status, as follows:

[0060] a) REPLACE request: Set the merge flag to REPLACE, clear addKeys, delKeys, and data, and then set new addKeys and data based on the hash key data carried in the request.

[0061] b) SET request: If the old merge flag is REMOVE, it is processed the same as the REPLACE request; otherwise, first add the key to be added to the hash key data to addKeys, delete the key corresponding to delKeys, then update the data, and finally update the merge flag. If the old merge flag is NONE, it is updated to SET; if the old merge flag is DELETE or COMPOSITE, assuming delKeys is empty, it is updated to SET, otherwise it is set to COMPOSITE. See for details. Figure 1 Flowchart of the process.

[0062] c) DELETE request: First, remove the key to be deleted from addKeys and remove the key-value pair data to be deleted from data; then, if the old merge status is not REPLACE, add the key to be deleted to delKeys; finally, update the merge flag. If the old merge flag is NONE, update to DELETE; if the old merge flag is REPLACE and addKeys is empty, update to REMOVE; if the old merge flag is SET or COMPOSITE, if addKeys is empty, update to DELETE, otherwise update to COMPOSITE. See for details. Figure 2 flow chart.

[0063] d) REMOVE request: Set the merge flag to REMOVE and clear the addKeys, delKeys and data data.

[0064] (4) Conversion from merge state to replicated read / write request: According to Table 1, the merge state is converted into replicated read / write request. For REPLACE and SET requests, the merged state data must be carried as the hash key table to be replaced and updated; for DELETE requests, the delKeys list must be carried to indicate the key-value pairs in the hash key table that need to be deleted from the slave node.

[0065] The method for merging hash key-value data requests in the key-value storage described above is further described by way of an embodiment.

[0066] This example has the following assumptions:

[0067] (1) In the initial state, the key-value store stores one key-value pair, whose key is "key" and whose value is a hash key-value table containing one key-value pair, whose key is "bin1" and whose value is "1";

[0068] (2) In the initial state, the merge flag in the merge state is NONE, and other fields are empty;

[0069] This example combines three requests for key-value pairs with the key "key", such as Figure 3 shown.

[0070] The first request is a SET request, carrying a key-value pair (bin2, 2). The hash key-value data with the key "key" contains two key-value pairs: (bin1, 1) and (bin2, 2). In the merge state, data contains the key-value pair (bin2, 2), and addKeys contains the key "bin1." The merge flag is SET.

[0071] The second request is a DELETE request with the key "bin1." At this point, the hash value data with the key "key" contains only one key-value pair, (bin2,2). In the merge state, data and addKeys remain unchanged, while delKeys adds "bin1," and the merge flag is COMPOSITE.

[0072] The third request is a REMOVE request. In this case, the value with the key "key" is cleared. In the merge state, data, addKeys, and delKeys are also cleared, and the merge flag is REMOVE.

[0073] After the three requests are merged, they are converted to a replicated request, which is a single REMOVE request. Without this method, data replication requires sending the three requests mentioned above. With this method, only one request is sent, saving the replication overhead of two requests.

[0074] Implementation Effect

[0075] To verify the effectiveness of this method, we connected this method to Redis and compared it with Redis without this method. The test plan is shown in the following table:

[0076] The server node hardware and software configuration is as follows:

[0077] CPU: Intel(R)Xeon(R)CPU E5-2620 v3@2.20GHz(12-cores 24-threads)

[0078] Memory: 250GB

[0079] Operating system: CentOS 7.8

[0080] The number of key-value pairs is 10 million, and the size of each data is about 500 bytes. In order to generate a "hotspot" scenario, the data conforms to the zipf distribution, with coefficients of 0.99 and 1.22 respectively. The higher the coefficient, the more serious the "hotspot" phenomenon.

[0081] Redis is built as a cluster of 4 master nodes and 4 slave nodes.

[0082] The loads requested in the experiment are 25% REPLACE, 25% SET, 25% DELETE, and 25% REMOVE. The first two loads are simulated by the HMSET command, and the last two loads are simulated by the HDEL command.

[0083] First, we conducted a stress load test with no flow restriction to compare the throughput before and after the implementation of this method. The experimental data is shown in Table 2.

[0084] Table 2 Data access operation throughput before and after accessing this method

[0085]

[0086] As shown in Table 2, in the "hotspot" scenario, after implementing this method, the data access throughput has been significantly improved compared to without implementing this method. Moreover, the more severe the "hotspot" phenomenon is, the greater the throughput improvement ratio after implementing this method.

[0087] Next, we conducted a throughput-limited network bandwidth test, limiting the throughput to 50,000 operations per second, and compared the network bandwidth usage of data replication before and after the implementation of this method. The experimental data is shown in Table 3.

[0088] Table 3 Network bandwidth occupied by data replication before and after accessing this method

[0089]

[0090] As shown in Table 3, in the "hotspot" scenario, after implementing this method, the network bandwidth occupied by data replication is significantly reduced. Moreover, the more severe the "hotspot" phenomenon is, the greater the reduction in the network bandwidth occupied by data replication after implementing this method.

[0091] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for merging hash key-value data requests in a key-value storage, characterized in that: The following steps are involved: (1) Definition of hash key-value data request for key-value storage: Since read requests do not modify data, this definition includes the following four types of requests, which only include write operations on data; the details are as follows: a) REPLACE operation: delete the old value of the hash key-value table under key and replace it with the hash key-value table of the parameter binKeyValues ​​in the write request operation. If the key does not exist, a new one is created; b) SET operation, updates the hash key value table under key; c) DELETE operation, deletes the key-value pair in the hash key-value table under key, where the key exists in the binKeys list of the delete operation parameter; if the key does not exist, an error is returned; if the hash key-value table under key is empty after deletion, it is recorded as a REMOVE operation; d) REMOVE operation, deletes all key-value pairs in the hash key-value table under key; if key does not exist, an error is returned; (2) Definition of Merge Status: The merge status contains the following fields: a) key: the key of key-value storage; b) addKeys: the list of keys that need to be added to the hash key-value table under the key key; c)delKeys: a list of keys that need to be deleted from the hash key-value table under key key, whose intersection with the addKeys list is empty; d) data: a key-value table that records the key-value pairs that need to be added to the hash key-value table under the key; e) state: Merge state identifier, used to process request merge logic and convert merged state into replicated read / write requests. The definition and meaning of the merge state identifier and the replicated read / write requests after conversion are as follows: NONE, indicating an empty state, where no converted copy read or write requests are performed; REPLACE: indicates that a request needs to be sent to replace the hash key table during replication. The converted replication read and write request is one REPLACE request. SET, indicating that a request needs to be sent to update the hash key table during replication. The converted replication read and write request is a SET request. DELETE: indicates that a request to delete part of the hash key table is required during replication. The converted replication read and write request is a DELETE request. REMOVE: indicates that a request needs to be sent to delete the entire hash key-value table during replication. The converted replication read and write request is one REMOVE request. COMPOSITE: indicates that a request needs to be sent during replication, which not only updates the key-value pairs in the hash key-value table but also deletes part of the hash key-value table. The converted replication read and write request is one SET request and one DELETE request. (3) Request merging and merging status update: According to different types of requests and current merging status, corresponding to different operation request merging processing logic, update the merging status, (4) Conversion from merge state to replicated read / write request: According to the definition of the merge state identifier, the merge state is converted into a replicated read / write request; for REPLACE and SET requests, the merged state data needs to be carried as the hash key value table for replacement and update; for DELETE requests, the delKeys list needs to be carried to indicate the key-value pairs of the hash key value table that need to be deleted from the data processing node.

2. A method for merging hash key-value data requests for key-value storage according to claim 1, characterized in that: In step (1), the SET operation is specifically as follows: if the key-value pair in the hash key-value table of the old value exists in the binKeyValues ​​hash key-value table in the parameter, it is replaced with the new value; if the key-value pair in the hash key-value table of the old value does not exist in the binKeyValues ​​hash key-value table, the old value is retained; if the key-value pair in the binKeyValues ​​hash key-value table does not exist in the old value, a new key-value pair is added; if the key does not exist, the same REPLACE operation is performed.

3. The method for merging hash key-value data requests for key-value storage according to claim 1, characterized in that: In step (3), the merge request and merge status update are as follows: a) REPLACE request: Set the merge flag to REPLACE, clear the fields addKeys, delKeys, and data in the merge state, set them to the default initial values, and then set new addKeys and data based on the hash key data carried in the request; b) SET request: If the old merge flag is REMOVE, it is processed in the same way as a REPLACE request; otherwise, first add the key to be added to the hash key value data to addKeys, delete the key corresponding to delKeys, then update the data, and finally update the merge flag; if the old merge flag is NONE, it is updated to a SET operation; if the old merge flag is DELETE or COMPOSITE, assuming delKeys is empty, it is updated to SET, otherwise it is set to the COMPOSITE state flag to facilitate subsequent processing based on the flag; c) DELETE request: First, remove the key to be deleted from addKeys and the key-value pair data to be deleted from data. Then, if the old merge status is not REPLACE, add the key to be deleted to delKeys. Finally, update the merge flag. d) REMOVE request: Set the merge flag to REMOVE and clear addKeys, delKeys, and data.

4. The method for merging hash key-value data requests for key-value storage according to claim 3, characterized in that: In sub-step c) of step (3), the specific process of updating the merge identifier is: if the old merge identifier is NONE, it is updated to DELETE; if the old merge identifier is REPLACE and addKeys is empty, it is updated to REMOVE; if the old merge identifier is SET or COMPOSITE, if addKeys is empty, it is updated to DELETE, otherwise it is updated to COMPOSITE.

Images