A method for evaluating hardware resource usage of data sharding in key-value storage

Abstract

The present invention discloses a method for evaluating the use of hardware resources of data slices for key-value storage, and belongs to the field of databases. The method comprises: by collecting data slice load and node hardware resource use information, establishing a node hardware resource use evaluation model based on the slice load, and by iteratively optimizing the evaluation parameters in the model, finally achieving an accurate evaluation of the actual hardware resource use of the data slice, thereby providing an efficient hardware resource use management solution for the distributed key-value storage system. Compared with the existing solutions, the implementation method of the present invention is relatively simple, and only the information collection code of the non-core part of the storage cluster needs to be appropriately modified, and the core logic is configured at the central node; at the same time, the present invention can effectively support the dynamic scheduling and optimization of the system, thereby improving the overall performance and resource utilization efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the field of databases, and in particular to a method for evaluating the use of hardware resources for data sharding in key-value storage. Background Art

[0002] In distributed key-value storage systems, data fault tolerance, load balancing, and resource optimization are core research issues. In existing architectures, data sharding management based on range partitioning has become the mainstream. These systems divide data into multiple shards based on key value ranges, thereby distributing data to different nodes in the cluster to achieve efficient data storage and access.

[0003] However, due to the uneven distribution of client requests, these range-divided systems often lead to uneven use of hardware resources between nodes. When the resource imbalance problem is serious, some nodes may become system bottlenecks, affecting the throughput and response speed of the entire cluster. The current common solution is to achieve load balancing through data shard migration, that is, setting load indicators, identifying hot shards with high load, and using a greedy algorithm to migrate these shards to nodes with lower hardware resource utilization.

[0004] Existing shard migration mechanisms mostly use data shard load as a migration indicator, because it is the basic migration unit in distributed storage systems, and there is a certain correlation between node hardware resource usage and shard load. However, affected by factors such as the size of key-value pairs in the shard, the status of the master copy, and the storage performance of the node, the shard load cannot fully reflect its actual usage of node hardware resources. Therefore, it is difficult to accurately measure the actual resource usage of the node using shard load as an indicator, which may lead to unsatisfactory results after migration, and even cause secondary hot spots in extreme cases, limiting the overall performance optimization of the system. Summary of the invention

[0005] The present invention proposes a method for evaluating the use of hardware resources in data sharding for key-value storage, which aims to build an accurate hardware resource use evaluation model, realize effective evaluation of the use of hardware resources in data sharding through multiple rounds of iterative optimization, and provide an accurate hardware resource use management solution for distributed key-value storage systems. Compared with other designs, the implementation is simple, and there is no need to modify the storage cluster code, and it can be implemented only in the central node.

[0006] The data sharding hardware resource usage evaluation method for key-value storage of the present invention comprises:

[0007] S110: Collect data shard load information and node hardware resource usage information;

[0008] S120: using the collected data sharding load information and node hardware resource usage information, establish a node hardware resource usage model, where the node hardware resource usage model is used to evaluate the impact of the data sharding load on the hardware resource usage in each node, thereby predicting the actual node hardware resource usage information;

[0009] S130: Implementing iteration of data sharding hardware resource usage evaluation parameters through the node hardware resource usage model. In each round of iteration, the node hardware resource usage information is estimated using the current data sharding hardware resource usage evaluation parameters and the periodically collected data sharding load information, and the data sharding hardware resource usage evaluation parameters are adjusted according to the error between the estimated value and the periodically collected actual value;

[0010] S140: Perform data sharding hardware resource usage evaluation using the data sharding hardware resource usage evaluation parameters obtained after iteration.

[0011] Beneficial effects:

[0012] The present invention can realize accurate evaluation of hardware resource usage of data shards in a distributed key-value storage system, provide a reliable basis for dynamic scheduling and optimization of resources, help solve the problem of unbalanced resource usage in existing systems, and improve the overall performance and efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of an operation flow of a method for evaluating hardware resource usage for data sharding of a key-value storage.

[0014] Figure 2 Gather diagrams for data shard load information and node hardware resource usage information. DETAILED DESCRIPTION

[0015] The following is a further detailed description of a method for evaluating hardware resource usage of key-value storage data sharding according to a specific embodiment of the present invention in conjunction with the accompanying drawings.

[0016] Embodiment 1:

[0017] This embodiment relates to a method for evaluating hardware resource usage of data shards for key-value storage, and is described by taking CPU usage evaluation of shards as an example. Figure 1 It is a schematic diagram of the operation flow of a method for evaluating the use of hardware resources for data sharding in a key-value storage system according to the present embodiment.

[0018] like Figure 1 As shown, the method specifically comprises the following steps:

[0019] S110: Collect data shard load information and node hardware resource usage information.

[0020] This step may include: establishing an information collection framework, periodically collecting two types of key information and reporting them to the management layer center node: collecting each data shard load information from the storage layer data nodes, and collecting node hardware resource usage information from the hardware layer.

[0021] Figure 2 A schematic diagram of collecting data shard load information and node hardware resource usage information is given. First, data shard load information is collected from the data nodes of the storage layer. For example, the data shard load information can be the number of requests per second (QPS), and the data shard load information is reported to the central node. Then, node hardware resource usage information is collected from the hardware layer infrastructure. For example, the node hardware resource usage information can be the CPU usage rate, and the node hardware resource usage information is reported to the central node.

[0022] S120: Using the collected data sharding load information and node hardware resource usage information, a node hardware resource usage model is established. The node hardware resource usage model evaluates the impact of the data sharding load on the hardware resource usage in each node, thereby predicting the actual hardware resource usage information of the node.

[0023] Assuming that the impact of each shard load on hardware resource usage is quantifiable, the hardware resource usage model of the node is constructed as follows (assuming that there are n data shards on the i-th node):

[0024] ;

[0025] The meanings of the parameters are as follows:

[0026] : Represents the estimated hardware resource usage of node i.

[0027] : The actual load information of shard j.

[0028] : Hardware resource usage evaluation parameter for shard j.

[0029] : Storage layer performance parameters of node i.

[0030] : Average performance parameter of all nodes in the cluster.

[0031] : Model error term, used to correct the difference between actual hardware resource consumption and theoretical estimation.

[0032] S130: Implementing data sharding hardware resource usage evaluation parameter iteration through the node hardware resource usage model.

[0033] In order to achieve accurate evaluation of hardware resource usage, the present invention uses multiple rounds of iterations to adjust the parameters of the evaluation model to improve the model accuracy. In each round of iteration, the node hardware resource usage information is estimated using the current evaluation parameters and the periodically collected data slice load information, and the parameters are adjusted according to the error between the estimated value and the periodically collected actual value. This process will continue until the preset convergence condition is met.

[0034] In this embodiment, the specific steps are as follows:

[0035] S01. Initial evaluation parameter estimation: Set the initial hardware resource usage evaluation parameters to be uniformly distributed:

[0036] ;

[0037] S02, node hardware resource usage estimation: Use the current evaluation parameters to estimate the node's hardware resource usage (assuming that the current iteration is the kth round):

[0038] ;

[0039] S03. Calculate the error term: The error term is calculated using the following formula to measure the accuracy of the current evaluation parameters:

[0040] ;

[0041] S04, update parameters: according to the calculated error term, adjust the evaluation parameters by gradient descent method ( is the learning rate):

[0042] ;

[0043] S05, iteration: repeat steps S02-S04 until the error term converges, that is, the following conditions are met ( is the preset error threshold):

[0044] ;

[0045] S140: Perform data sharding hardware resource usage evaluation using the data sharding hardware resource usage evaluation parameters obtained after iteration.

[0046] Through multiple rounds of iterative optimization of parameters, the model can gradually approach the actual resource usage of the node. The evaluation parameters after iteration to convergence are substituted into the hardware resource usage evaluation model of the data shard to obtain the hardware resource usage evaluation of the data shard. The hardware resource usage evaluation model of the data shard is:

[0047] ;

[0048] in, is the hardware resource utilization evaluation result of shard j.

[0049] The above describes the specific implementation process of the present invention. The present invention provides an accurate hardware resource usage management solution for a distributed key-value storage system by constructing a hardware resource usage evaluation model and continuously optimizing parameters.

Claims

1. A method for evaluating the use of hardware resources for data sharding in key-value storage, characterized in that: include: S110: Collect data shard load information and node hardware resource usage information; S120: using the collected data sharding load information and node hardware resource usage information, establish a node hardware resource usage model, where the node hardware resource usage model is used to evaluate the impact of the data sharding load on the hardware resource usage in each node, thereby predicting the actual node hardware resource usage information; S130: implementing iteration of data sharding hardware resource usage evaluation parameters through the node hardware resource usage model. In each round of iteration, the node hardware resource usage information is estimated using the current data sharding hardware resource usage evaluation parameters and the periodically collected data sharding load information, and the data sharding hardware resource usage evaluation parameters are adjusted according to the error between the estimated value and the periodically collected actual value; wherein the data sharding hardware resource usage evaluation parameters are parameters used for evaluation by the established node hardware resource usage model; S140: Perform data sharding hardware resource usage evaluation using the data sharding hardware resource usage evaluation parameters obtained after iteration.

2. The method for evaluating the use of hardware resources for data sharding of key-value storage according to claim 1, characterized in that: S110 includes: establishing an information collection framework, periodically collecting two types of key information and reporting them to the management layer center node: collecting the load information of each data slice from the storage layer data node, and collecting the node hardware resource usage information from the hardware layer.

3. The method for evaluating the use of hardware resources for data sharding of key-value storage according to claim 2, characterized in that: S120 includes: assuming that there are n data shards on the i-th node, a node hardware resource usage model is established as follows: ； in, : indicates a node Estimated hardware resource usage, : Data Sharding The actual load information, : Data Sharding Hardware resource usage evaluation parameters, :node Storage layer performance parameters, : The average performance parameter of all nodes in the cluster, : Model error term, used to correct the difference between actual hardware resource consumption and theoretical estimation.

4. The method for evaluating the use of hardware resources for data sharding of key-value storage according to claim 3, characterized in that: S130 includes: S01. Initial evaluation parameter estimation: Set the initial hardware resource usage evaluation parameters to be uniformly distributed: ； S02. Node hardware resource usage estimation: Use the current initial hardware resource usage evaluation parameters to estimate the node's hardware resource usage information: ； k means that the current iteration is the kth round; S03. Calculate the error term: The error term is calculated using the following formula to measure the accuracy of the current initial hardware resource usage evaluation parameters: ； S04. Update parameters: According to the calculated error term, adjust the evaluation parameters by gradient descent method: ； is the learning rate; S05, iteration: Repeat steps S02-S04 until the error term converges, that is, the following conditions are met: ； is the preset error threshold.

5. The method for evaluating the use of hardware resources for data sharding of key-value storage according to claim 4, characterized in that: S140 includes: Substitute the data sharding hardware resource usage evaluation parameters after iteration to convergence into the data sharding hardware resource usage evaluation model to obtain the data sharding hardware resource usage evaluation. The data sharding hardware resource usage evaluation model is: ； in, For Sharding The hardware resource usage assessment results of the .

Images