Key-Value Distributed Balanced Storage Method Based on Programmable Data Plane

Abstract

The present invention proposes a key-value distributed balanced storage method based on a programmable data plane. This method is implemented on a P4 programmable switch. By statistically analyzing the client key-value request traffic on the data plane, it discriminates hot keys with high request frequencies, and reports the hot keys and their access frequencies to the control plane. The control plane calculates the number of replica nodes required for the hot keys based on this information, and then converts them into matching items and their action matching parameters in the data plane action matching table, and then distributes them to the hot key matching action table in the data plane. During the initial client request process, the data replication of the hot keys is completed on the replica nodes, and then these storage nodes can jointly respond to requests for the hot keys. The present invention has no impact on the original functions of the existing P4 programmable switch, can operate together with other functions of the switch, is generally applicable to the existing data centers, and has a good effect on distributed key-value storage load balancing.

Description

Technical Field

[0001] The present invention relates to the technical field of computer networks, and particularly to a key-value distributed balanced storage method based on a programmable data plane. Background Art

[0002] Similar to web services, search engines, etc., which have to process tens of millions of requests every day and have high requirements for latency and throughput. These applications are usually supported by distributed key-value storage. The distributed key-value storage divides key-value objects into multiple storage servers to achieve scalability and load distribution. In an ideal case, a cluster consisting of N nodes with a throughput of up to T can meet the throughput requirement of N*T.

[0003] However, in the real-world key-value storage workload, the popularity distribution follows a power-law distribution, resulting in unbalanced access frequencies. For those nodes containing popular objects, their loads are naturally high. Therefore, these storage systems usually exhibit highly asymmetric access patterns. The system is bottlenecked due to the performance overload of some nodes, while some other nodes are not fully utilized and fail to achieve the expected throughput. The overloaded nodes cannot process client requests in a timely manner, so the queue of uncompleted requests becomes very long. The system will have a high tail latency, which violates the quality-of-service requirements. The traditional consistent hashing algorithm cannot handle such situations well. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a key-value distributed balanced storage method based on a programmable data plane, which can balance the loads among storage nodes under workload skew.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: A key-value distributed balanced storage method based on a programmable data plane, specifically including the following steps:

[0006] After the key-value client submits a request to the distributed key-value storage, the P4 switch parses the key information in the data packet on the data plane and inputs the key information into the key access frequency statistics module;

[0007] The P4 data plane, according to the key access frequency information and based on the configured key access frequency threshold T, reports the keys with access frequencies higher than T and their frequency information to the control end;

[0008] The control end detects the hot key reporting information from the data plane, calculates the number of required replica storage nodes based on the counted frequency numbers, and issues the hot key and the replication factor to the hot key matching action table on the data plane;

[0009] When the key field k in the key request data packet on the client hits the hot key matching action table in the data plane, the operation type of the key is determined. If it is a read operation, the data plane calculates the destination node A through the node calculation function according to the replication factor, and then the destination node A judges the existence of the key k. If it is true, it is forwarded to the A node for processing; otherwise, it is forwarded to the master node of the key. Then, when the master node replies to the data packet of the request to the data plane, the data plane clones a data packet to the A node. The A node stores the key value, and the A node with this hot key is marked in the data plane; if it is a write, it is forwarded to the master node;

[0010] For key requests that are not in the hot key matching action table, they are all forwarded to the master node of the key.

[0011] In a preferred embodiment, the key access frequency statistics are specifically as follows: The key access frequency represents the number of times a key is accessed within a time period; considering the storage space limitation of the programmable data plane and the requirement for statistical accuracy, the data plane uses the count - minsketch algorithm to count the key requests passing through the programmable switch. Its structure consists of a register array with k rows and each bucket size of w. Each row maps the key to the corresponding register array by a pair - wise independent hash function; then the frequency statistics of a key on this structure are:

[0012] frequency(key)=min(k i [h i (key)])#(1)

[0013] where k i represents each row of the register array, and h i represents the hash function corresponding to each row of the register array.

[0014] In a preferred embodiment, the control plane periodically clears the counting information of the count - min sketch module. When calculating the hot key frequency, consider the hot key frequencies of two periods. One is the frequency statistical information of the latest period, denoted as frequency(key) cur , and the other is the frequency statistical information of the previous period, denoted as frequency(key) pre , then the statistical frequency frequency(key) corresponding to a key is expressed as:

[0015] frequency(key)=frequency(key) cur +α*frequency(key) pre #(2)

[0016] Among them, α represents the weight of the previous stage frequency statistics, α ∈ [0, 1], which is expressed as

[0017]

[0018] Among them, t now represents the current time, t curend represents the time when the current latest stage ends, and t represents the duration of each stage.

[0019] In a preferred embodiment, the hot key replication number, i.e., Number Of Replication, NOR represents the number of replica nodes required for each hot key; for the number of replicas of a hot key key, it is considered related to its access frequency, and the higher the access frequency of the hot key, the more replica nodes are allocated; the NOR of a hot key key can be expressed as:

[0020]

[0021] Among them, T is the access frequency threshold constant determined as a hot key.

[0022] In a preferred embodiment, multi-value hash means that the same key is mapped to multiple distinct values in a finite set through the hash operation; the method of salting is introduced in the hash function operation; then the multi-value hash operation for a hot key is expressed as:

[0023] h(key, i) = hash(key, salt[i]) % w#(5)

[0024] Among them, i is the i-th hash calculation value of the key key, and salt[i] is the i-th salting value participating in the hash calculation.

[0025] In a preferred embodiment, hot key routing means that the programmable switch forwards the key-value request of the client to the corresponding storage node according to the key; the hot key table attributes include key, NOR, Regiter index, key stores the known hot keys, NOR stores the number of replica nodes corresponding to the hot key, and Register index stores the indexes of the polling register and the valid register corresponding to the hot key;

[0026] The polling register stores the salting index value used by each known hot key After each hot key obtains the corresponding value, the corresponding

[0027] The multi-value hash logic module is internally implemented by the multi-value hash algorithm, and calculates the id of the destination node according to the key and ​

[0028] The valid register is used to determine whether the calculated node stores the key-value object of the query key. It is implemented by a bitmap internally. When bitmap[i] is 1, it means that the node with node_id = i stores the key-value object of the key, and when it is 0, it means there is no such object.

[0029] The composition attributes of the node address table include node_id and node_address. node_id is the index value of each key-value storage node on the data plane, and node_address is the IP address of the key-value storage node corresponding to each node_id.

[0030] In a preferred embodiment, when the data plane detects a request q from the client for the key key, since the key hits the hot key table, the multi-value hash module is executed to calculate that the destination node is node1. By querying the bitmap of the key, it is found that node1 is invalid, that is, node1 does not store the key-value pair of the hot key key. Then the data plane adds the node1 information to the copy field in the request q and forwards it to the master node of the key. After the master node processes it, a reply packet r is generated and forwarded to the data plane. When the data plane detects that the reply packet r contains the copy field, it executes the clone method inside the data plane to clone a packet c from r. The original reply packet is forwarded to the requesting client. Then the data plane modifies the destination address of the packet c to the address of node1 and then forwards it to node1. Node1 completes the storage of the key-value pair of the hot key key, and the data plane marks node1 as valid on the bitmap of the key. After that, node1 can serve requests for the key, thus sharing the load for the master node of the key.

[0031] During the execution process, there is a transmission delay. This part of the delay comes from the delay T of the data plane forwarding to the master node switch→homeNode , the delay T of the master node replying to the data plane homeNode→switch , and the delay T of the data plane forwarding the cloned packet to the replica node switch→replicateNode , and these three parts are equal. Therefore, the total delay delay = 3T. As a result, some packets will be marked with the copy field, causing the data plane to execute repeated cloning multiple times, wasting bandwidth and the processing overhead of the storage node. Temporarily store the key and the clone field containing the replica node information, and set the expiration duration to delay. When the master node receives a key request containing the clone field, if it has been cloned before, it clears the clone field when replying.

[0032] Compared with the prior art, the present invention has the following beneficial effects: BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of key access frequency statistics for a preferred embodiment of the present invention;

[0034] Figure 2 Schematic diagram of key routing for a preferred embodiment of the present invention;

[0035] Figure 3 Schematic diagram of hot key object replication for a preferred embodiment of the present invention. Detailed implementation manners

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further descriptions of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0038] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] A key-value distributed balanced storage method based on a programmable data plane, refer to Figures 1 to 3 , and specifically includes the following steps:

[0040] After the key-value client submits a request to the distributed key-value storage, the P4 switch parses the key information in the data packet on the data plane and inputs the key information into the key access frequency statistics module;

[0041] Based on the key access frequency information, the P4 data plane reports the keys with an access frequency higher than the configured key access frequency threshold T and their frequency information to the control end according to the configured key access frequency threshold T;

[0042] The control end detects the hot key reporting information from the data plane, calculates the number of required replica storage nodes according to the counted frequency number, and issues the hot key and the replication factor to the hot key matching action table on the data plane;

[0043] When the key field k in the key request data packet of the client hits the hot key matching action table in the data plane, the operation type of the key is discriminated. If it is a read operation, the data plane calculates the destination node A through the node calculation function according to the replication factor, and then the destination node A judges the existence of the key k. If it is true, it is forwarded to the A node for processing; otherwise, it is forwarded to the master node of the key. Then, when the master node replies to the data packet of the request to the data plane, the data plane clones a data packet to the A node. The A node stores the key value, and the data plane marks that the A node has this hot key; if it is a write, it is forwarded to the master node;

[0044] For key requests that are not in the hot key matching action table, they are all forwarded to the master node of the key.

[0045] Specifically, key access frequency statistics: The key access frequency represents the number of times a key is accessed within a time period. Considering the storage space limitation of the programmable data plane and the requirement for statistical accuracy, the data plane uses the count-min sketch algorithm to statistically analyze the key requests passing through the programmable switch. As Figure 1 shown, its structure consists of a register array with k rows and each bucket size of w. Each row maps the key to the register array of the corresponding row by a pair-wise independent hash function. Then, the frequency statistics of a key on this structure are:

[0046] frequency(key) = min(k i [h i (key)])#(1)

[0047] where k i represents each row of the register array, and h i represents the hash function corresponding to each row of the register array.

[0048] Specifically, ensuring the timeliness of key access frequency statistics: A hot key has a generation, maintenance, and attenuation process, and has a certain life cycle. When performing key access frequency statistics, it is necessary to consider the timeliness of the internal data of the statistics. Outdated statistical data will interfere with the statistics and discrimination of new hot keys. To ensure the timeliness of statistical data information, the control plane periodically clears the counting information of the count-min sketch module. When calculating the hot key frequency, consider the hot key frequencies in two periods. One is the frequency statistical information of the latest period, denoted as frequency(key) cur , and the other is the frequency statistical information of the previous period, denoted as frequency(key) pre . Then, the statistical frequency frequency(key) of a key can be expressed as:

[0049] frequency(key)=frequency(key) cur +α*frequency(key) pre #(2)

[0050] Among them, α represents the weight of the frequency statistics in the previous stage, α∈[0,1], which can be expressed as

[0051]

[0052] Among them, t now Indicates the current time. Indicates the time when the latest stage ends, and t indicates the duration of each stage.

[0053] Specifically, the number of hot key replications: The number of hot key replications (NOR) indicates the number of replica nodes required for each hot key. The load imbalance between storage nodes comes from the fact that nodes with hot keys are more frequently accessed by clients, while nodes without hot keys are relatively idle. Therefore, the load balancing method between nodes is to distribute hot key requests to as many nodes as possible. However, this process requires copying hot objects to these replica nodes, which requires additional bandwidth and storage overhead. Therefore, for the number of replicas of a hot key, consider its access frequency. Hot keys with higher access frequencies are assigned more replica nodes. The NOR of a hot key can be expressed as:

[0054]

[0055] Wherein, T is the access frequency threshold constant for determining a hot key.

[0056] Specifically, multi-value hash: multi-value hash means that the same key can be mapped to multiple different values ​​in a finite set through hash operations. Originally, the same key can only get one value under the hash function calculation, so the present invention introduces the salting method in the hash function calculation. Then the multi-value hash operation for a hot key can be expressed as:

[0057] h(key,i)=hash(key,salt[i])%w#(5)

[0058] Among them, i is the i-th hash calculation value of the key key, and salt[i] is the i-th salt value involved in the hash calculation.

[0059] Specifically, key routing: hot key routing means that the programmable switch forwards the client's key value request to the corresponding storage node according to the key. Figure 2As shown in (a) and (b), the hot key routing is implemented on the data plane and consists of a hot key table, a polling register, a multi-value hash logic module, a valid register, and a node address table.

[0060] The hot key table attributes include key, NOR, Register index. The key stores the known hot keys, NOR stores the number of replica nodes corresponding to the hot keys, and Register index stores the indexes of the polling register and the valid register corresponding to the hot keys.

[0061] The polling register stores the salt index values used by each known hot key. After each hot key obtains the corresponding value, the corresponding

[0062] The multi-value hash logic module is implemented by a multi-value hash algorithm inside. According to the key and it can calculate the id of the destination node.

[0063] The valid register is used to determine whether the calculated node stores the key-value object of the query key. It is implemented by a bitmap inside. When bitmap[i] is 1, it means that the node with node_id = i stores the key-value object of key, and when it is 0, it means there is none.

[0064] The node address table attributes include node_id and node_address. node_id is the index value of each key-value storage node on the data plane, and node_address is the ip address of the key-value storage node of each node_id.

[0065] Specifically, for the replication of hot key objects: when the control plane sends a new hot key key and its configuration to the data plane, the data plane can calculate the destination node through the multi-value hash module. This destination node can be the primary node of the key or a replica node of the key. At the beginning, these replica nodes do not have the key-value pair of the key, and the key-value pair of the key needs to be copied to these replica nodes. The present invention proposes a method of replication after acquisition. As Figure 3 (a) shows, the data plane detects the request q of the client for the key key. Since the key hits the hot key table, the multi-value hash module is executed to calculate the destination node as node1. By querying the bitmap of the key, it is found that node1 is invalid, that is, node1 does not store the key-value pair of the hot key key. Then the data plane adds the node1 information to the replica field in the request q and forwards it to the primary node of the key. After the primary node processes it, it generates a reply packet r and forwards it to the data plane, as Figure 3As shown in (b), when the data plane detects that the reply packet r contains a copy field, it executes the clone method inside the data plane to clone the packet r into a new packet c. The original reply packet is forwarded to the requesting client. Then the data plane modifies the destination address of packet c to the address of node1 and forwards it to node1. Node1 completes the storage of the key-value pair of the hot key key. The data plane marks node1 as valid on the bitmap of key. After that, node1 can serve requests for key, thus sharing the load for the primary node of key and achieving load balancing among storage nodes.

[0066] During the execution process, there is a transmission delay. This part of the delay comes from the delay T of the data plane forwarding to the primary node switch→homeNode , the delay T of the primary node replying to the data plane homeNode→switch , and the delay T of the data plane forwarding the cloned packet to the replica node switch→replicateNode . These three parts are roughly equal. Therefore, the total delay delay = 3T. As a result, some packets will be marked with the copy field, causing the data plane to execute repeated cloning multiple times, wasting bandwidth and the processing overhead of storage nodes. Therefore, it is designed that when the primary node detects a packet containing the clone field, it temporarily stores the key and the clone field containing the replica node information, and sets the expiration duration to delay. When the primary node receives a key request containing the clone field, if it has been cloned before, it clears the clone field when replying.

Claims

1. A key-value distributed balanced storage method based on a programmable data plane, characterized in that, Specifically, it includes the following steps: After the key-value client submits a request to the distributed key-value store, the P4 switch parses the key information in the data packet on the data plane and inputs the key information into the key access frequency statistics module; Based on the key access frequency information, the P4 data plane reports the keys with access frequencies higher than the configured key access frequency threshold T and their frequency information to the control plane according to the key access frequency threshold T; The control plane detects the hot key reporting information from the data plane, calculates the number of required replica storage nodes based on the counted frequency, and sends the hot key and the replication factor to the hot key matching action table on the data plane; When the key field k in the key request data packet of the client hits the hot key matching action table on the data plane, the operation type of the key is determined. If it is a read operation, the data plane calculates the destination node A through the node calculation function according to the replication factor, and then judges the existence of the key field k by node A. If it is true, it is forwarded to node A for processing; Otherwise, it is forwarded to the primary node of the key. Then, when the primary node replies to the data packet of the request to the data plane, the data plane clones a data packet to node A. Node A stores the key value and marks the existence of this hot key on the data plane; If it is a write, it is forwarded to the primary node; For key requests not in the hot key matching action table, they are all forwarded to the primary node of the key; The specific key access frequency statistics are as follows: The key access frequency represents the number of times a key is accessed within a time period. Considering the storage space limitation of the programmable data plane and the requirement for statistical accuracy, the data plane uses the count-minsketch algorithm to statistically analyze the key requests passing through the programmable switch. Its structure consists of a register array with k rows and each bucket size of w. Each row maps the key to the register array of the corresponding row by a pairwise independent hash function. Then, the frequency statistics of a key on this structure are as follows: frequency(key) = min(k i [h i (key)]) #(1) Among them, k i represents each row of the register array, and h i represents the hash function corresponding to each row of the register array.

2. The key-value distributed balanced storage method based on a programmable data plane according to claim 1, characterized in that, The control plane periodically clears the counting information of the count-min sketch module. When calculating the hot key frequency, the hot key frequencies of two periods are considered. One is the frequency statistical information of the latest period, denoted as frequency(key) cur , and the other is the frequency statistical information of the previous period, denoted as frequency(key) pre , then the statistical frequency frequency(key) corresponding to a key is expressed as: frequency(key) = frequency(key) cur + α * frequency(key) pre #(2) Among them, α represents the weight of the previous stage frequency statistics, α ∈ [0, 1], which is expressed as Among them, t now represents the current time, represents the time when the current latest stage ends, and t represents the duration of each stage.

3. The key-value distributed balanced storage method based on a programmable data plane according to claim 1, characterized in that, The number of hot key replications, that is, the Number Of Replication (NOR), represents the number of replica nodes required for each hot key. For the number of replicas of a hot key key, it is considered to be associated with its access frequency. The higher the access frequency of a hot key, the more replica nodes are allocated to it. The NOR of a hot key key can be expressed as: Among them, T is the access frequency threshold constant for determining a hot key.

4. The key-value distributed balanced storage method based on a programmable data plane according to claim 2, characterized in that, Multi-value hash means that the same key is mapped to multiple distinct values in a finite set through hash operations. The method of salting is introduced in the hash function operation. Then, the multi-value hash operation for a hot key is expressed as: h(key, i) = hash(key, salt[i]) % w #(5) Among them, i is the i-th hash calculation value of the key key, and salt[i] is the i-th salting value participating in the hash calculation.

5. The key-value distributed balanced storage method based on a programmable data plane according to claim 4, characterized in that, Hot-key routing means that a programmable switch forwards the key-value request of a client to the corresponding storage node according to the key; the hot-key table attribute contains key, NOR, Register index, where key stores the known hot keys, NOR stores the number of replica nodes corresponding to the hot keys, Register index stores the indexes of the polling register and the valid register corresponding to the hot key; The polling register stores the salt index value used by each known hotkey Each hotkey obtains the corresponding After obtaining the value, the corresponding The multi-value hash logic module is internally implemented by a multi-value hash algorithm. Based on the key and calculate the ID of the destination node; The valid register is used to determine whether the calculated node stores the key-value object of the query key. It is implemented using a bitmap internally. When bitmap[i] is 1, it means that the node with node_id = i stores the key-value object of key, and when it is 0, it means there is none; The composition attribute of the node address table contains node_id and node_address. node_id is the index value of each key-value storage node on the data plane, and node_address is the IP address of the key-value storage node of each node_id.

6. The key-value distributed balanced storage method based on a programmable data plane according to claim 5, characterized in that, When the data plane detects a request q from the client for the key key, since the key hits the hot-key table, the multi-value hash module is executed to calculate that the destination node is node1. By querying the bitmap of the key, it is found that node1 is invalid, that is, node1 does not store the key-value pair of the hot key key. Then the data plane adds the node1 information to the replica field in the request q and forwards it to the primary node of the key. After the primary node processes it, it generates a reply packet r and forwards it to the data plane. When the data plane detects that the reply packet r contains a replica field, it executes the clone method inside the data plane to clone a packet c from r. The original reply packet is forwarded to the requesting client. Then the data plane modifies the destination address of the packet c to the address of node1 and then forwards it to node1. Node1 completes the storage of the key-value pair of the hot key key, and the data plane marks node1 as valid on the bitmap of the key. After that, node1 can serve the requests for the key, thus sharing the load for the primary node of the key; During the execution process, there is a transmission delay, and this part of the delay comes from the delay T of the data plane forwarding to the master node switch→homeNode , the delay T of the master node replying to the data plane homeNode→switch , the delay T of the data plane forwarding the cloned packet to the replica node switch→replicateNode , these three parts are equal, so the total delay delay = 3T. Therefore, some packets will be marked with the replica field, causing the data plane to perform repeated cloning multiple times, wasting bandwidth and the processing overhead of storage nodes; temporarily store the key and the cloning field containing the replica node information, and set the expiration duration to delay; when the master node receives a key request containing the cloning field, if it has been cloned before, clear the cloning field when replying.

Images