Sharding state allocation method considering cross-shard transaction quantity and system load balancing

Abstract

The application discloses a sharding state allocation method considering cross-sharding transaction quantity and system load balancing, characterized by adopting a multi-level cost model and a workload balancing method to realize efficient state allocation of a sharding system, wherein the method optimizes a system cost model according to different overhead levels of cross-sharding transactions, constructs a graph based on historical transaction data in an initialization stage, and generates an initial data state mapping to sharding by using a graph partitioning algorithm; in an adjustment stage, the data state is transferred from an overloaded sharding to an underutilized sharding to optimize the utilization of overall resources. Compared with the prior art, the application has efficient state allocation, effectively reduces the overhead caused by cross-sharding transactions, maintains load balancing, has a low time complexity, greatly improves system performance and resource utilization, meets the application needs of various scenes, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a shard state allocation method on a consortium chain that takes into account both the number of cross-shard transactions and system load balancing. Background Art

[0002] Blockchain is a distributed system that integrates multiple computer technologies, including distributed data storage, peer-to-peer protocols, consensus mechanisms, and asymmetric encryption. It boasts characteristics such as immutability, transparency, and traceability. In recent years, consortium blockchains have been widely adopted in large-scale collaborative scenarios such as finance and supply chains. However, the rapid growth of the user base and the urgent need for transaction processing performance have conflicted with the inherent performance bottlenecks of blockchain. Sharding technology, a promising solution, enables concurrent transaction processing by dividing the entire blockchain system into numerous independent shards. However, in a sharded system, cross-shard transactions may occur. The excessive number of such transactions and the inefficient processing of cross-shard transactions can significantly impact the overall system performance.

[0003] In recent years, numerous works related to sharding have employed hash-based methods to partition blockchain account data into different shards. However, this often results in nearly 90% of transactions being executed across shards. To reduce the number of cross-shard transactions, some work constructs historical transactions into a graph and uses classic graph partitioning algorithms to cap the number of cross-transactions between accounts on different shards. However, this approach only considers the balance of graph vertex weights and fails to account for the balance of transaction workload across shards. While some work has utilized community detection methods to detect community structure in systems and optimize system throughput, the results are unpredictable. Furthermore, all of these methods assume that cross-shard transaction costs are similar across shards. However, in large-scale scenarios, shards are often distributed across different locations, and cross-shard transactions across distant partitions typically incur higher costs than transactions between adjacent shards. Furthermore, while it makes sense to intentionally assign the set of state accessed to a specific shard when processing a transaction, the state assigned to a shard may involve access patterns from both local and remote shards. In other words, blindly assigning states to the same shard may significantly increase the probability of users around other shards remotely accessing this shard.

[0004] In summary, existing technologies for reducing cross-shard transaction processing ignore load balancing between different shards, resulting in underutilized system resources. In scenarios with large-scale participants, the varying costs of cross-shard transactions further complicate this issue. This is because the state assigned to a shard may involve access patterns from both local and remote shards, and the remote access events caused by state transfers themselves also increase the network access overhead of transactions. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of the existing technology and provide a shard state allocation method that takes into account the number of cross-shard transactions and system load balancing. It uses a greedy algorithm to optimize load balancing and construct a multi-level cost model. It uses historical transaction data to construct a graph, and uses a graph partitioning algorithm to generate a mapping of the initial data state to the shards, so that the data state is transferred from the overloaded shard to the underutilized shard. During the load adjustment phase, the data state of the overloaded shard is migrated to the underutilized shard to minimize the occurrence of cross-shard transactions, ensure efficient state allocation of the shard system, and improve system performance. This method optimizes the system cost model according to the different overhead levels of cross-shard transactions, uses a greedy algorithm to optimize load balancing, and optimizes overall resource utilization to meet application needs. It has good application prospects.

[0006] The specific technical solution for achieving the purpose of the present invention is: a shard state distribution method that takes into account the number of cross-shard transactions and system load balancing, characterized in that the method includes: construction of a multi-level cost model suitable for large-scale cross-shard transactions and workload balancing based on a greedy algorithm. The multi-level cost model construction suitable for large-scale cross-shard transactions is based on the assumption that each shard has a specific workload carrying capacity upper limit. By allocating different levels of costs for cross-shard transactions accessing states in different physical areas, it provides heuristic guidance for the distribution of states in shards in different regions, ensuring that the execution cost of cross-shard transactions is as low as possible while making full use of the overall performance resources of the system; the workload balancing based on the greedy algorithm is to sort the edge weights in the transaction graph composed of overloaded shards in descending order before execution, and retain the states with higher edge weights in the original shards, thereby increasing the probability of edges with smaller weights becoming cut edges, aiming to reduce the processing cost of the state access part that becomes a cross-shard transaction.

[0007] The construction of the multi-level cost model suitable for large-scale cross-shard transactions specifically includes:

[0008] Step A1: Modeling initialization phase

[0009] Based on regional characteristics (such as physical distance), the state access cost of cross-regional transactions is divided into different levels. The access heuristic cost of cross-shard transactions in the same cost level area is the same. This is consistent with the fact that in reality, the execution overhead of transaction access across countries and continents is greater than the execution overhead of access within the local area network.

[0010] Step A2: During the transaction load balancing phase

[0011] When the load of a transaction is transferred between specific regions, the state access cost of cross-shard transactions in all other regions whose access states intersect with the load will also change. Therefore, consider whether to transfer the K-hop state of the shard as well.

[0012] The workload balancing based on the greedy algorithm specifically includes:

[0013] Step B1: Sorting the edge weights of the shard transaction graph

[0014] Overloaded shards cannot handle all transaction loads, while non-overloaded shards still have available idle computing resources. Based on the heuristic guidance of the transaction graph by the above-constructed cost model, a weight is assigned to each edge in the transaction graph composed of all shards to be unloaded, and the edges are sorted according to the weight.

[0015] Step B2: State migration scheduling algorithm

[0016] Using a greedy approach, a non-overloaded shard is selected as the destination for a specific state migration. Since the multi-level cost model is constructed, the impact of migrating the state required for a cross-shard transaction to another specific shard on overall system throughput is known. Therefore, the algorithm can always find a locally optimal destination shard, reducing the load on overloaded shards and fully utilizing the computing resources of idle shards while keeping the cost of global cross-shard transactions as low as possible.

[0017] The step A1 specifically includes the following steps:

[0018] Step A11: The system initialization cost is based on recent historical transactions, where the distribution structure of transactions in the shards is also consistent with the historical distribution records.

[0019] Step A12: The number of states accessed by each transaction in the system's shards varies. To facilitate cost calculation, we normalize the number of states accessed by a transaction into multiple sub-transactions, each of which accesses only two states. Therefore, the presence of an edge between two state nodes in the transaction graph indicates that a (sub-)transaction accessed both states, and the cost of access is the weight of this edge.

[0020] The step A2 specifically includes the following steps:

[0021] Step A21: Maintain metadata related to load cost calculation in a single shard, including the number of intra-shard transactions and the number of cross-shard transactions at each cost level. The total load of the shard is the sum of the number of intra-shard transactions multiplied by the cost of a single intra-shard transaction, plus the number of cross-shard transactions at each cost level multiplied by the actual cost of that level.

[0022] Step A22: During the load migration process, the change in intra-shard and cross-shard transactions in each shard is calculated. The initial throughput of the shard obtained from historical data plus the total transfer change is the theoretical upper limit of the final shard's throughput.

[0023] Step A23: The total system actual throughput is the sum of the actual throughputs of each shard. The metadata of the load cost maintenance is used to determine whether a specific shard is overloaded. The actual throughput of a single shard is the throughput that its proportional load capacity can actually support.

[0024] Step A24: Based on the contribution of a single transaction state transition to the total system throughput obtained in the above steps, and the fact that a single transaction state transition often causes changes in the access costs of other (sub)transactions with intersecting access states, when this state transition occurs, the change in total system throughput is used to determine whether the states of its K-hop neighbors need to be transferred to the destination shard.

[0025] The step B2 specifically includes the following steps:

[0026] Step B21: When a specific shard S is overloaded, select a non-overloaded shard S with the lowest cost for cross-shard state access.

[0027] Step B22: Traverse the weight edge of the previously sorted shard S in descending order, and add the two states connected by this edge to S until shard S is no longer overloaded or shard S is fully loaded.

[0028] Step B23: For high-overhead cross-shard transactions that still exist in the entire system, consider transferring their load within K hops to more appropriate shards with the goal of optimizing the total system throughput.

[0029] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0030] 1) This invention aims to optimize the efficiency of system resource utilization in large-scale cross-shard transaction scenarios. Through a two-stage state distribution scheme, it effectively reduces the overhead brought by cross-shard transactions while maintaining load balance and having low time complexity.

[0031] 2) This paper constructs a multi-level cost model and uses a graph partitioning algorithm in conjunction with historical transaction data to generate an initial shard mapping. During the adjustment process, a greedy algorithm is used to transfer the data state of overloaded shards to underutilized shards, minimizing the number and cost of cross-shard transactions.

[0032] 3) This invention fully considers the different cost differences of cross-shard transactions, enabling the system to achieve efficient state allocation in large-scale application scenarios, thereby improving the overall system performance and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the sharding scenario in Example 1;

[0034] Figure 2 A flowchart of the workload. DETAILED DESCRIPTION

[0035] This invention comprises a two-stage shard state allocation scheme: first, it constructs a multi-level cost model and combines it with a greedy algorithm to generate an initial load-balanced shard map to effectively allocate the initial data state. Second, it dynamically adjusts the shard state based on the cost differences of cross-shard transactions, migrating data from high-load shards to less-utilized shards, reducing the number of cross-shard transactions and improving system performance. While maintaining low time complexity, this approach optimizes resource utilization, ensures efficient load balancing in large-scale scenarios, and achieves low-overhead allocation of cross-shard transactions and optimized system performance.

[0036] In order to reduce cross-shard transaction overhead by optimizing throughput, we first model the necessary performance metrics, including the workload and throughput of the i-th shard, and the total system throughput σ. We use η and 1 to represent the overhead of a single cross-shard transaction and an intra-shard transaction for accessing two states with a single overhead level of l, respectively. Where η>1, this is because processing cross-shard transactions is more complex than intra-shard transactions. Let and are the number of intra-shard transactions and cross-shard transactions with level l overhead for the i-th shard, respectively. The throughput of each shard is modeled based on the recent historical transaction data G and the state distribution result. The workload of the i-th shard is the total workload τ to be processed, which is calculated by the following formula (a):

[0037]

[0038] The workload change of a shard consists of four parts: the increase and decrease of transaction workload within the shard, and the increase and decrease of transaction workload across shards. Therefore, the workload change Δ of the i-th shard after the state transfer is calculated by the following formula (b):

[0039]

[0040] in, They are the changes in intra-shard transaction load and cross-shard transaction load involved in the i-th shard, respectively.

[0041] The workload τ of the i-th shard after the state transition is calculated by the following formula (c):

[0042] τ=τ+Δ(c).

[0043] The actual throughput of a shard can be further divided into two cases, depending on whether the workload exceeds the processing capacity λ of the shard. Specifically, if the workload of the shard is lower than λ, it means that all transactions can be successfully processed. Therefore, the throughput σ of the i-th shard is expressed as follows (d):

[0044]

[0045] The total system throughput is calculated as the sum of the throughput of each shard. Suppose if we transfer state x (originally assigned to shard p) and its K-hop neighbors to shard q, the throughput after the transfer is It is expressed by the following formula (e):

[0046]

[0047] in, σ (,X,+,,) are the total system throughput and the throughput of the i-th shard after transferring from shard p via state X and K-hop neighbors, respectively.

[0048] Therefore, the total throughput gain Δ when transferring state X (initially assigned to shard p) and its K-hop neighbors to shard q is (X,+,,) Calculated by the following formula (f):

[0049]

[0050] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0051] Example 1

[0052] See Figure 1 , this sharding partitioning scenario example, its consortium chain includes four shards located in: Location 1, Location 2, Location 3, and Location 4. Each shard stores a part of the state that does not overlap with each other, and there is a smaller physical distance between Location 1 and Location 2, and between Location 3 and Location 4. Each vertex in the graph represents a state, each edge represents two states that are commonly accessed in the same transaction, and the weight reflects the state access cost of such transactions. The access cost of each cross-shard transaction is divided into two levels (L1 and L2) here. Specifically, the state access cost of cross-shard transactions across different continents is the highest. In the figure, the cross-shard transaction cost between Location 1 and Location 4 is assigned to level L1. Since the state transfer between shards will increase additional overhead, it is obviously higher than the L2 level sharding cost between Location 1 and Location 2.

[0053] During the load balancing phase, to reduce the cross-shard overhead caused by states A and C in the diagram, state C might be moved from Somewhere 4 to Somewhere 1. However, at the same time, states B and C will also transition from coexisting on a specific shard to being dispersed across different shards, causing transactions involving states B and C to become more expensive cross-shard transactions. Therefore, the neighbor search space for state C is further expanded to include the shards at Somewhere 3 and Somewhere 4. In contrast, when attempting to reduce cross-shard transactions involving states I and H, the search space for their K-hop neighbors only requires the shards Somewhere 3 and Somewhere 4, respectively, which have lower levels of cross-region overhead.

[0054] The workload balancing based on the greedy algorithm is used. The overall process is detailed in the algorithm shown in Table 1 below:

[0055] Table 1 Load balancing process

[0056]

[0057] Table 1 above describes the overall load balancing process. This greedy algorithm is used to achieve efficient state partitioning in a sharded system to address the overhead associated with cross-shard transactions. The input includes the transaction graph G = (V, E, W) and the transaction load of each shard. The goal is to partition the state set V into several subsets V, V4, …, V to ensure load balancing across each shard and reduce the frequency of cross-shard transactions.

[0058] The greedy algorithm uses the METIS graph partitioning algorithm to generate an initial state shard map, so that the initial state set is evenly distributed across the shards. For each remaining overloaded shard G, the algorithm enters the load adjustment phase. At this point, the edge set E′ within the shard is sorted according to edge weights to prioritize high-weight edges, thereby reducing the number of cross-shard transactions during the adjustment process. In the load adjustment loop, the greedy algorithm checks whether the target shard is an underloaded shard and has the lowest cross-shard transaction overhead with the current shard G. If the conditions are met, the greedy algorithm traverses the edges within the shard, obtains the set of visited vertices, and adds them to the state set to be migrated, uresV. If the load of the current shard exceeds the threshold λ, and the load of the target shard is less than λ after the merger, the greedy algorithm will update the state sets of both shards, transferring part of the overloaded shard's state to the underutilized shard. The shard allocation results are updated in real time during this process.

[0059] After the load balancing step is completed, the greedy algorithm further optimizes the cost of cross-shard transactions. For each remaining high-cost cross-shard transaction, the algorithm identifies the set of K-hop neighbors of its state and calculates the allocation result that maximizes the throughput gain using the following formula (f):

[0060]

[0061] This step can further update the workload and throughput of the shards, thereby achieving a more optimal resource allocation effect. Ultimately, the algorithm returns the final state partition result, allowing the system to achieve load balancing and effectively reduce the additional overhead of cross-shard transactions.

[0062] See Figure 2 , the workload of the overloaded shard is composed of historical transactions, where the shard is assigned fourteen states, namely states G, H, I, J, L, M, N, O, P, Q, R, S, T, U. Assume that it is attached Figure 1 For the shard at location 1, a greedy algorithm is first applied to reduce the workload by transferring the state to the nearest shard at location 2, while ensuring that the workload of the shard at location 1 does not fall below its processing capacity and that the workload of the shard at location 2 does not exceed its processing capacity. The algorithm divides the graph into two parts and moves the state of the part marked in green to the shard at location 2. However, this introduces four new cross-shard transaction accesses, namely cross-shard transactions between M and L, R and S, U and S, and T and S. At this point, if the shard at location 1 is still overloaded, the algorithm will transfer the remaining part to the shard at location 3 or location 4.

[0063] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A shard state allocation method that takes into account the number of cross-shard transactions and system load balancing, characterized by: The method includes: constructing a multi-level cost model suitable for large-scale cross-shard transactions and workload balancing based on a greedy algorithm. The multi-level cost model suitable for large-scale cross-shard transactions specifically includes: Step A1: Modeling initialization phase Classify the state access costs of cross-region transactions and build a multi-level cost model; Step A2: Transaction load balancing adjustment phase Dynamically adjust the shard status based on the cost difference of cross-shard transactions, that is, consider whether to transfer the K-hop status of the shard; The workload balancing based on the greedy algorithm specifically includes: Step B1: Sorting the edge weights of the shard transaction graph Assign weights to each edge in the transaction graph based on the constructed multi-level cost model, and sort the edges in the shard transaction graph based on their weights; Step B2: State migration scheduling algorithm Use a greedy algorithm to select a non-overloaded shard as the destination for a specific state migration. Utilize the computing resources of idle shards to reduce the load on overloaded shards, making the cost of global cross-shard transactions as low as possible. The step A1 specifically includes: Step A11: The system initialization cost is based on recent historical transactions, where the distribution structure of transactions in the shards is consistent with the historical distribution records; Step A12: Decompose a transaction accessing multiple states into multiple sub-transactions. Each sub-transaction only accesses the transferred state node and the retained state node. The cost of its access is the weight of the two state node edges, where the node edge is the line connecting the two state nodes. The step A2 specifically includes: Step A21: Calculation of total shard load Maintain metadata related to the load cost calculation for a single shard, including the number of intra-shard transactions and the number of cross-shard transactions at each cost level. The total load of the shard is the sum of the number of intra-shard transactions multiplied by the cost of a single intra-shard transaction, plus the number of cross-shard transactions at each cost level multiplied by the actual cost of that level. Step A22: Calculation of the theoretical throughput limit The theoretical throughput cap for the final shard is calculated by adding the initial throughput of the shard to the total transfer change. The initial throughput is obtained from historical data. The total transfer change is the sum of the changes in intra-shard transactions and cross-shard transactions in each shard during the load migration process. Step A23: Specific Shard Overload judgment The actual throughput of the total system is the sum of the actual throughput of each shard. Based on the metadata maintained by the load cost, it is determined whether a specific shard is overloaded. The actual throughput of the shard is the throughput number actually supported by its proportional load capacity. For shards that are not overloaded; Step A24: Neighborhood state transition Based on the contribution of a single transaction state transfer to the actual throughput of the total system, and the changes in access costs of other sub-transactions with access state intersections caused by the transfer of a single transaction state, it is determined whether the states of its K-hop neighbors need to be transferred to the destination shard when the state transfer occurs.

2. The shard state allocation method that takes into account the number of cross-shard transactions and system load balancing according to claim 1 is characterized in that: The step B2 specifically includes: Step B21: When a specific shard When overloaded, select a non-overloaded shard with the lowest cost for cross-shard state access. ; Step B22: Traverse the shards in descending order The weight edge of the edge adds the two states connected by this edge to the non-overloaded shard , until the sharding No more overload or fragmentation Fully loaded; Step B23: For high-overhead cross-shard transactions in the entire system, consider transferring their load within K hops to non-overloaded shards with the goal of optimizing the total system throughput.

Images