Fragmentation state distribution method giving consideration to cross-fragment transaction number and system load balance

Abstract

The invention discloses a fragmentation state distribution method considering the number of cross-fragmentation transactions and system load balance, which is characterized in that efficient state distribution of a fragmentation system is realized by adopting a multi-level cost model and a workload balance method, and the method optimizes a system cost model according to different overhead levels of the cross-fragmentation transactions, so that the system load balance is realized. In the initialization stage, a graph is constructed based on historical transaction data, mapping from an initial data state to fragments is generated by using a graph division algorithm, and in the adjustment stage, the data state is transferred to the fragments which are not fully utilized from the overload fragments, so that the utilization of overall resources is optimized. Compared with the prior art, the method has the advantages of high-efficiency state allocation, effective reduction of overhead caused by cross-fragment transaction, load balance keeping, low time complexity, great improvement of system performance and resource utilization rate, meeting of application requirements of various scenes, and good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a shard state allocation method on a consortium chain that takes into account both the number of cross-shard transactions and system load balancing. Background Art

[0002] Blockchain is a distributed system that integrates distributed data storage, P2P protocol, consensus mechanism, asymmetric encryption and other computer technologies. It has the characteristics of being tamper-proof, open, transparent and traceable. In recent years, consortium chains have been widely used in large-scale collaborative scenarios such as finance and supply chain, but the rapid growth of the user base and the urgent demand for transaction processing performance conflict with the execution performance bottleneck of the blockchain itself. Sharding technology, as a highly promising solution, achieves concurrent transaction processing by dividing the entire blockchain system into many separate shards. However, in the sharding scenario, the system will have cross-shard transactions. The excessive number of such transactions and the inefficiency of processing cross-shard transactions will greatly affect the overall performance of the system.

[0003] In recent years, many works related to sharding technology have adopted hash-based methods to divide blockchain account data into different shards, but this often results in nearly 90% of transactions being executed in a cross-shard form. In order to reduce the number of cross-shard transactions, some works construct historical transactions into graphs and use classic graph partitioning algorithms to control the upper limit of interactive transactions between accounts in different shards. However, this method only considers the balance of graph vertex weights, but does not consider the balance of transaction workloads between different shards. Although there are also works that use community detection methods to detect community structures in the system and optimize system throughput, the results obtained by this method are unpredictable. In addition, all these methods assume that the cross-shard transaction overhead between different shards is similar. However, in large-scale scenarios, shards are usually distributed in different locations, and cross-shard transactions with far-separated partitions usually have higher overhead than those between adjacent shards. In addition, although it makes sense to intentionally attribute the set of states that need to be accessed to a specific shard when processing transactions, the state assigned to a shard may involve access patterns of both local and remote shards. In other words, blindly assigning states to the same shard may significantly increase the probability that users around other shards will remotely access this shard.

[0004] In summary, the existing technology of reducing cross-shard transaction processing ignores the load balancing between different shards, resulting in insufficient utilization of system resources. In the scenario of large-scale participants, the change of cross-shard transaction overhead makes the problem more complicated, because the state assigned to a shard may involve the access mode of both local and remote shards, and the remote access event caused by the state transfer itself will also increase the network access overhead of the transaction. Summary of the invention

[0005] The purpose of the present invention is to provide a shard state allocation method that takes into account the number of cross-shard transactions and system load balancing in response to the deficiencies of the prior art. A greedy algorithm is used to optimize load balancing and construct a multi-level cost model, a graph is constructed using historical transaction data, and a graph partitioning algorithm is used to generate a mapping of the initial data state to the shard, so that the data state is transferred from the overloaded shard to the underutilized shard. In the load adjustment stage, the data state of the overloaded shard is migrated to the underutilized shard to minimize the occurrence of cross-shard transactions, ensure efficient state allocation of the shard system, and improve system performance. The method optimizes the system cost model according to the different overhead levels of cross-shard transactions, optimizes load balancing using a greedy algorithm, and optimizes overall resource utilization to meet application needs, and has a good application prospect.

[0006] The specific technical scheme for achieving the purpose of the present invention is: a shard state allocation method that takes into account the number of cross-shard transactions and system load balancing, characterized in that the method includes: construction of a multi-level cost model suitable for large-scale cross-shard transactions and workload balancing based on a greedy algorithm, wherein the construction of the multi-level cost model suitable for large-scale cross-shard transactions is to allocate different levels of costs for cross-shard transactions to access states in different physical areas under the assumption that each shard has a specific workload carrying capacity upper limit, thereby providing heuristic guidance for the distribution of states in shards in different regions, ensuring that the execution cost of cross-shard transactions is as low as possible while making full use of the overall performance resources of the system; the workload balancing based on the greedy algorithm is to, before execution, sort the edge weights in the transaction graph composed of the overloaded shards in descending order, retain the states with higher edge weights in the original shards, thereby increasing the probability of edges with smaller weights becoming cut edges, aiming to reduce the state access part that becomes a cross-shard transaction with a lower processing cost.

[0007] The construction of the multi-level cost model suitable for large-scale cross-shard transactions specifically includes:

[0008] Step A1: Modeling initialization phase

[0009] Based on regional characteristics (such as physical distance), the state access cost of cross-regional transactions is divided into levels. The access heuristic cost of cross-shard transactions in the same cost level area is the same. This is consistent with the fact that in reality, the execution overhead of transaction access across countries and continents is greater than the execution overhead of access within the local area network.

[0010] Step A2: During the transaction load balancing adjustment phase

[0011] When the load of a transaction is transferred between specific areas, the state access cost of cross-shard transactions in all other areas that have access states that intersect with the load will also change accordingly. Therefore, consider whether to transfer the K-hop state of the shard as well.

[0012] The workload balancing based on the greedy algorithm specifically includes:

[0013] Step B1: Sorting edge weights of shard transaction graph

[0014] Overloaded shards cannot handle all transaction loads, while non-overloaded shards still have available idle computing resources. Based on the heuristic guidance of the transaction graph by the cost model constructed above, a weight is assigned to each edge in the transaction graph consisting of all shards to be unloaded, and the edges are sorted according to the weights.

[0015] Step B2: State Migration Scheduling Algorithm

[0016] Through the greedy idea, a non-overloaded shard is selected as the destination for the migration of a specific state. After the multi-level cost model is constructed, the impact of transferring the state that needs to be accessed by a cross-shard transaction to another specific shard on the overall system throughput is known. Therefore, the algorithm can always find a local optimal destination shard, so that the cost of global cross-shard transactions is as low as possible while the load of overloaded shards is reduced and the computing resources of idle shards are fully utilized.

[0017] The step A1 specifically includes the following steps:

[0018] Step A11: The system initialization cost is based on recent historical transactions, where the distribution structure of transactions in the shards is also consistent with the historical distribution records.

[0019] Step A12: The number of states accessed by each transaction in the shards within the system is different. To facilitate the calculation of the cost, normalization is performed and a transaction accessing multiple states is decomposed into multiple sub-transactions, where each sub-transaction only accesses two states. Therefore, the existence of an edge between two state nodes in the transaction graph means that a (sub) transaction accesses these two states, and the cost of the access is the weight of this edge.

[0020] The step A2 specifically includes the following steps:

[0021] Step A21: Maintain meta-information related to load cost calculation in a single shard, including the number of intra-shard transactions and the number of cross-shard transactions at each cost level. The total load of the shard is the sum of the number of intra-shard transactions multiplied by the cost of a single intra-shard transaction, plus the number of cross-shard transactions at each cost level multiplied by the actual cost of that level.

[0022] Step A22: Count the changes in intra-shard transactions and cross-shard transactions in each shard during the load migration process. The initial throughput of the shard obtained from historical data plus the total transfer change is the theoretical upper limit of the final shard's throughput.

[0023] Step A23: The total system actual throughput is the sum of the actual throughputs of each shard. The metadata of the load cost maintenance is used to determine whether a specific shard is overloaded. The actual throughput of a single shard is the throughput that its proportional load capacity can actually support.

[0024] Step A24: Based on the contribution of the single transaction state transfer to the actual throughput of the total system obtained in the above steps, and the transfer of the single transaction state often causes the change of the access cost of other (sub) transactions with access state intersections. Therefore, when the state is transferred, the change of the total system throughput is used to determine whether its K-hop neighbor state needs to be transferred to the destination shard.

[0025] The step B2 specifically includes the following steps:

[0026] Step B21: When a specific shard S is overloaded, select a non-overloaded shard S with the lowest cost for cross-shard state access.

[0027] Step B22: traverse the weight edge of the shard S that has been previously sorted in descending order, and add the two states connected by this edge to S until shard S is no longer overloaded or shard S is fully loaded.

[0028] Step B23: For the high-overhead cross-shard transactions that still exist in the entire system, consider transferring the load within K hops to more appropriate shards with the goal of optimizing the total system throughput.

[0029] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0030] 1) The present invention aims to optimize the efficiency of system resource utilization in large-scale cross-shard transaction scenarios. Through a two-stage state allocation scheme, the overhead caused by cross-shard transactions can be effectively reduced while maintaining load balance and having low time complexity.

[0031] 2) The present invention constructs a multi-level cost model and generates an initial shard mapping using a graph partitioning algorithm in combination with historical transaction data. During the adjustment process, the data state of the overloaded shard is transferred to the underutilized shard based on a greedy algorithm, minimizing the number and cost of cross-shard transactions.

[0032] 3) The present invention fully considers the different cost differences of cross-shard transactions, enabling the system to achieve efficient state allocation in large-scale application scenarios, thereby improving the overall system performance and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a slicing partitioning scenario in Example 1;

[0034] Figure 2 A flowchart of the workload. DETAILED DESCRIPTION

[0035] The present invention includes a two-stage shard state allocation scheme: first, through the construction of a multi-level cost model, combined with a greedy algorithm to generate an initial load-balanced shard mapping to effectively allocate the initial data state; second, dynamically adjust the shard state according to the cost difference of cross-shard transactions, migrate the data of high-load shards to shards with lower utilization, reduce the number of cross-shard transactions and improve system performance. Under the premise of low time complexity, this method optimizes resource utilization, ensures efficient load balancing in large-scale scenarios, and realizes low-overhead allocation of cross-shard transactions and optimization of system performance.

[0036] In order to reduce the cross-shard transaction overhead by optimizing throughput, we first model the necessary performance metrics, including the workload and throughput of the i-th shard, and the total system throughput σ. We use η and 1 to represent the overhead of a single cross-shard transaction with two states with l overhead levels and the overhead of an intra-shard transaction, respectively. Where η>1, this is because processing cross-shard transactions is more complex than intra-shard transactions. Let and are the number of intra-shard transactions and cross-shard transactions with level l overhead for the i-th shard, respectively. The throughput of each shard is modeled based on the recent historical transaction data G and the state distribution result. The workload of the i-th shard is the total workload τ to be processed, which is calculated by the following formula (a):

[0037]

[0038] The workload change of a shard includes four parts: the increase and decrease of transaction workload within the shard, and the increase and decrease of transaction workload across shards. Therefore, the workload change Δ of the i-th shard after state transfer is calculated by the following formula (b):

[0039]

[0040] in, They are respectively the intra-shard transaction load change and the cross-shard transaction load change involved in the i-th shard.

[0041] The workload τ of the i-th shard after the state transfer is calculated by the following formula (c):

[0042] τ=τ+Δ(c).

[0043] Depending on whether the workload exceeds the processing capacity λ of the shard, the actual throughput of the shard can be further divided into two cases. Specifically, if the workload of the shard is lower than λ, it means that all transactions can be processed successfully. Therefore, the throughput σ of the i-th shard is expressed by the following formula (d):

[0044]

[0045] The total system throughput is calculated as the sum of the throughput of each shard. Suppose if we transfer state x (originally assigned to shard p) and its K-hop neighbors to shard q, then the throughput after the transfer is It is expressed by the following formula (e):

[0046]

[0047] in, σ (,X,+,,) are the total system throughput and the throughput of the i-th shard after transferring from shard p via state X and K-hop neighbors, respectively.

[0048] Therefore, the total throughput gain Δ when transferring state X (initially assigned to shard p) and its K-hop neighbors to shard q is (X,+,,) Calculated by the following formula (f):

[0049]

[0050] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0051] Example 1

[0052] See also Figure 1 , this sharding partitioning scenario example, its alliance chain includes four shards located in: Somewhere 1, Somewhere 2, Somewhere 3, and Somewhere 4, each shard stores a part of the state that does not intersect with each other, and there is a smaller physical distance between Somewhere 1 and Somewhere 2, and Somewhere 3 and Somewhere 4. Each vertex in the graph represents a state, each edge represents two states that are commonly accessed in the same transaction, and the weight reflects the state access overhead of such transactions. The access overhead of each cross-shard transaction is divided into two levels (L1 and L2) here. Specifically, the state access overhead of cross-shard transactions across different continents is the highest. In the figure, the cross-shard transaction cost between Somewhere 1 and Somewhere 4 shards is assigned to level L1. Since the state transfer between shards will increase additional overhead, it is obviously higher than the L2 level sharding overhead between Somewhere 1 and Somewhere 2.

[0053] In the load balancing phase, in order to reduce the cross-shard overhead caused by states A and C in the figure, state C may be transferred from Somewhere 4 to Somewhere 1. But at the same time, states B and C will also change from coexisting in a specific shard to being dispersed in different shards, which will cause transactions involving states B and C to become cross-shard transactions with higher overhead. Therefore, the neighbor search space of state C is further expanded to include shards in Somewhere 3 and Somewhere 4. In contrast, when trying to reduce cross-shard transactions involving states I and H, the search space of K-hop neighbors of these two states only requires two shards, Somewhere 3 and Somewhere 4, which have lower levels of cross-region overhead.

[0054] The workload balancing based on the greedy algorithm is used. The overall process is detailed in the algorithm shown in Table 1 below:

[0055] Table 1 Load balancing process

[0056]

[0057] Table 1 above describes the overall process of load balancing. This greedy algorithm is used to achieve efficient state partitioning in the sharding system to deal with the overhead problem caused by cross-shard transactions. The input includes the transaction graph G = (V, E, W) and the transaction load information of each shard. The purpose is to divide the state set V into several subsets V, V4, ..., V to ensure load balancing of each shard and reduce the frequency of cross-shard transactions.

[0058] The greedy algorithm uses the METIS graph partitioning algorithm to generate the initial state shard mapping so that the initial state set is evenly distributed in each shard. For each remaining overloaded shard G, the algorithm enters the load adjustment phase. At this time, the edge set E′ in the shard is sorted according to the edge weight so that high-weight edges are processed first, thereby reducing the number of cross-shard transactions during the adjustment process. In the load adjustment loop, the greedy algorithm checks whether the target shard is an underloaded shard and has the smallest cross-shard transaction overhead with the current shard G. If the conditions are met, the greedy algorithm traverses the edges in the shard, obtains the set of visited vertices, and adds them to the state set uresV to be migrated. If the load of the current shard exceeds the threshold λ, and the load of the target shard is less than λ after the merger, the greedy algorithm will update the state sets of the two shards and transfer part of the state of the overloaded shard to the underutilized shard. The shard allocation results will be updated in real time during this process.

[0059] After the load balancing step is completed, the greedy algorithm further optimizes the cost of cross-shard transactions. For each remaining high-cost cross-shard transaction, the algorithm identifies the set of K-hop neighbors of its state and calculates the allocation result that maximizes the throughput gain using the following formula (f):

[0060]

[0061] This step can further update the workload and throughput of the shard, thereby achieving a better resource allocation effect. Finally, the algorithm returns the final state partition result, allowing the system to achieve load balancing and effectively reduce the additional overhead of cross-shard transactions.

[0062] See also Figure 2 , the workload of the overloaded shard is composed of historical transactions, where the shard is assigned fourteen states, namely states G, H, I, J, L, M, N, O, P, Q, R, S, T, U. Assume that it is attached Figure 1 For the shard at location 1, the greedy algorithm is first applied to reduce the workload by transferring the state to the nearest shard at location 2, while ensuring that the workload of the shard at location 1 does not fall below its processing capacity, and the workload of the shard at location 2 does not exceed its processing capacity. The algorithm divides the graph into two parts and moves the state of the part marked in green to the shard at location 2. However, this will introduce four new cross-shard transaction accesses, namely cross-shard transactions between M and L, R and S, U and S, and T and S. At this time, if the shard at location 1 is still overloaded, the algorithm will transfer the remaining part to the shard at location 3 or location 4.

[0063] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A shard state allocation method that takes into account the number of cross-shard transactions and system load balancing, characterized in that: The method includes: constructing a multi-level cost model suitable for large-scale cross-shard transactions and workload balancing based on a greedy algorithm. The multi-level cost model construction suitable for large-scale cross-shard transactions specifically includes: Step A1: Modeling initialization phase Classify the state access costs of cross-region transactions and build a multi-level cost model; Step A2: Transaction load balancing adjustment phase Dynamically adjust the shard status based on the cost difference of cross-shard transactions, that is, consider whether to transfer the K-hop status of the shard; The workload balancing based on the greedy algorithm specifically includes: Step B1: Sorting edge weights of shard transaction graph Assign weights to each edge in the transaction graph based on the constructed multi-level cost model, and sort the edges in the shard transaction graph based on their weights; Step B2: State Migration Scheduling Algorithm Use a greedy algorithm to select a non-overloaded shard as the destination for a specific state migration, use the computing resources of idle shards to reduce the load of overloaded shards, and make the cost of global cross-shard transactions as low as possible.

2. The shard state allocation method that takes into account the number of cross-shard transactions and system load balancing according to claim 1 is characterized in that: The step A1 specifically includes: Step A11: The system initialization cost is based on recent historical transactions, where the transaction distribution structure in the shards is consistent with the historical distribution records; Step A12: Decompose a transaction accessing multiple states into multiple sub-transactions, each sub-transaction only accesses the transferred state node and the retained state node, and the access cost is the weight of the two state node edges, where the node edge is the connection between the two state nodes.

3. The shard state allocation method that takes into account the number of cross-shard transactions and system load balancing according to claim 1 is characterized in that: The step A2 specifically includes: Step A21: Calculation of total shard load Maintain meta-information related to the calculation of load cost in a single shard, including the number of intra-shard transactions and the number of cross-shard transactions at each cost level. The total load of the shard is the sum of the number of intra-shard transactions multiplied by the cost of a single intra-shard transaction, plus the number of cross-shard transactions at each cost level multiplied by the actual cost of that level; Step A22: Calculation of the theoretical throughput limit The theoretical throughput upper limit of the final shard is obtained by adding the initial throughput of the shard to the total transfer change. The initial throughput is obtained from historical data. The total transfer change is the sum of the intra-shard transaction changes and cross-shard transaction changes in each shard during the load migration process. Step A23: Specific Shard S i Overload judgment The actual throughput of the total system is the sum of the actual throughputs of each shard. Based on the metadata maintained by the load cost, it is determined whether a specific shard is overloaded. The actual throughput of the shard is the throughput number actually supported by its proportional load capacity. i For shards that are not overloaded; Step A24: Neighborhood state transition Based on the contribution of a single transaction state transfer to the actual throughput of the total system and the change in access cost of other sub-transactions with access state intersection caused by the transfer of a single transaction state, it is determined whether the K-hop neighbor states need to be transferred to the destination shard when the state is transferred.

4. The shard state allocation method that takes into account the number of cross-shard transactions and system load balancing according to claim 1 is characterized in that: The step B2 specifically includes: Step B21: When a specific shard S i When overloaded, select a non-overloaded shard S with the lowest cost for cross-shard state access j ; Step B22: Traverse the shards S in descending order i The weight edge of this edge adds the two states connected by this edge to the non-overloaded shard S j , until the fragment S i No more overload or fragmentation j Fully loaded; Step B23: For high-overhead cross-shard transactions in the entire system, with the goal of optimizing the total system throughput, consider transferring the load within K hops to the shards that are not overloaded.

Images