A dynamic blockchain sharding method and system based on a composite clustering algorithm

Abstract

The application discloses a kind of dynamic blockchain sharding method and system based on composite clustering algorithm, the method includes: based on blockchain sharding system, determine transaction frequency matrix and sharding allocation matrix, construct the target optimization transaction model of blockchain sharding system;Based on the target optimization transaction model of blockchain sharding system, preliminary global division processing is carried out through agglomerative hierarchical clustering algorithm;The preliminary blockchain sharding result is divided and processed by DBSCAN clustering algorithm, and the blockchain sharding result after secondary division is obtained;Based on the blockchain sharding result after secondary division, carry out noise account division processing, obtain the final blockchain sharding result.The application can preferentially aggregate high-frequency associated accounts and classify low-frequency accounts, minimizing the proportion of cross-shard transactions and achieving load balancing.The application is a kind of dynamic blockchain sharding method and system based on composite clustering algorithm, which can be widely applied in the field of blockchain transaction technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of blockchain transaction, in particular to a dynamic blockchain sharding method and system based on a composite clustering algorithm. BACKGROUND

[0002] Sharding technology, as one of the mainstream means to improve the scalability of blockchain, has attracted much attention in recent years. The core idea is to effectively improve the throughput and performance of the blockchain by dividing the computing and storage burden of the network. However, although sharding technology has shown great potential in the theoretical level, it still faces many key challenges in practical application, especially in the aspects of sharding load balancing and excessive proportion of cross-sharding transactions. Sharding load imbalance can lead to resource shortage in some shards, while other shards are idle, thus affecting the overall efficiency of the system. Load imbalance not only wastes system resources, but also causes performance bottlenecks. Especially when the network load is high, some shards may be congested due to handling too many requests, thereby affecting the response time and throughput of the entire blockchain system. At the same time, an excessively high proportion of cross-sharding transactions will cause additional communication and computing overhead. First, the communication overhead increases, as cross-sharding transactions require frequent information exchange between shards, increasing the communication cost. Second, the processing process is complicated, as the state update of cross-sharding transactions involves ledger modification and consistency guarantee of multiple shards, which usually requires complex protocols (such as two-phase commit or lock-based protocols) to ensure transaction atomicity. This not only increases the transaction processing time, but also may cause performance degradation due to operation conflicts, which has a significant impact on network performance and latency. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide a dynamic blockchain sharding method and system based on a composite clustering algorithm, which can prioritize the aggregation of high-frequency associated accounts and classify the processing of low-frequency accounts, thereby minimizing the proportion of cross-sharding transactions and achieving load balancing.

[0004] The first technical solution adopted by the present application is: a dynamic blockchain sharding method based on a composite clustering algorithm, comprising the following steps:

[0005] Based on the blockchain sharding system, determine the transaction frequency matrix and the shard allocation matrix, and construct the target optimization transaction model of the blockchain sharding system;

[0006] Based on the target optimization transaction model of the blockchain sharding system, perform preliminary global division processing through the agglomerative hierarchical clustering algorithm to obtain the preliminary blockchain sharding result;

[0007] Perform division processing on the preliminary blockchain sharding result through the DBSCAN clustering algorithm to obtain the secondary divided blockchain sharding result;

[0008] Based on the secondary division result of the blockchain shard, noise account division processing is performed to obtain the final blockchain shard result.

[0009] Further, the step of determining the transaction frequency matrix and the shard allocation matrix based on the blockchain sharding system to construct the target optimization transaction model of the blockchain sharding system specifically includes:

[0010] Based on the blockchain sharding system, historical transaction data between all accounts is obtained to construct a transaction frequency matrix;

[0011] Based on the blockchain sharding system, a cross-shard transaction total frequency and a system load balancing objective function are defined;

[0012] A preset weight coefficient is introduced, and the cross-shard transaction total frequency and the system load balancing objective function are combined and represented in matrix form to obtain a shard allocation matrix;

[0013] The transaction frequency matrix and the shard allocation matrix are combined and used to represent the calculation process of cross-shard transactions to construct the target optimization transaction model of the blockchain sharding system.

[0014] Further, the expression of the target optimization transaction model of the blockchain sharding system is specifically as follows:

[0015]

[0016] In the above formula, λ represents the weight coefficient, M represents the transaction frequency matrix, Z represents the shard allocation matrix, Z T represents the transpose matrix of the shard allocation matrix, i represents the i-th account, j represents the j-th account, Z ip represents whether the account i is allocated to the shard p, Z jq represents whether the account j is allocated to the shard q, and sum(·) represents the total transaction frequency of all cross-shard transactions.

[0017] Further, the step of obtaining a preliminary blockchain shard result by performing preliminary global division processing on the target optimization transaction model based on the blockchain sharding system specifically includes:

[0018] Based on the transaction frequency matrix in the target optimization transaction model based on the blockchain sharding system, a distance expression based on transaction frequency is defined to obtain the similarity degree between accounts;

[0019] An account is regarded as a cluster to construct an initial cluster set;

[0020] Based on the similarity degree between accounts, the inter-cluster distance of the initial cluster set is obtained by single-linkage method, and the minimum value of the inter-cluster distance and the cluster number of the initial cluster set are determined.

[0021] By clustering the initial cluster set through the condensed hierarchical clustering algorithm, if the minimum value of the cluster distance is greater than the preset cluster distance threshold or the number of clusters does not meet the target value, the two clusters corresponding to the minimum value of the cluster distance are iteratively merged, and the minimum value of the cluster distance and the number of clusters are updated;

[0022] Until the updated minimum value of the cluster distance is less than the preset cluster distance threshold or the updated number of clusters meets the target value, a preliminary blockchain sharding result is obtained.

[0023] Further, the expression of the cluster distance is specifically as follows:

[0024]

[0025] In the above formula, C i and C j represent two clusters, a p and a q represent any accounts belonging to C i and C j , d(a p , a q ) represents the similarity between a p account and a q account, and D(C i , C j ) represents the distance between C i cluster and C j cluster.

[0026] Further, the step of dividing the preliminary blockchain sharding result through the DBSCAN clustering algorithm to obtain a secondary divided blockchain sharding result specifically includes:

[0027] Constructing a subset data matrix according to the transaction frequency between the accounts in the clusters in the preliminary blockchain sharding result, and determining the account distance in the subset data matrix;

[0028] Based on the DBSCAN clustering algorithm, defining a neighborhood radius and a minimum number of neighborhood points;

[0029] Iterating through each account in the preliminary blockchain sharding result through the DBSCAN clustering algorithm, and constructing a neighborhood point set of each account according to the account distance in the subset data matrix and the neighborhood radius;

[0030] Determining the number of neighborhood points of the account according to the neighborhood point set of the account and judging;

[0031] If the neighborhood point number of the account is greater than or equal to the minimum neighborhood point number, the account is marked as a core account, and all accounts in the core account neighborhood are added to the current cluster until no new account can be added, and a density cluster is obtained;

[0032] If the neighborhood point number of the account is less than the minimum neighborhood point number and belongs to the density cluster, the account is marked as a boundary account;

[0033] If the neighborhood point number of the account is less than the minimum neighborhood point number and does not belong to the density cluster, the account is marked as a noise account;

[0034] The density cluster, the boundary account, and the noise account are integrated to obtain a blockchain sharding result after secondary division.

[0035] Further, the expression of the account distance in the subset data matrix is specifically as follows:

[0036]

[0037] f(a p ,a q )=T i [p][q]

[0038] In the above formula, d(a p ,a q ) represents the account distance in the subset data matrix, T i [p][q] represents the transaction frequency between accounts a p and a q , f(·) represents a transaction frequency function, and a r and a s represent accounts for normalization reference.

[0039] Further, based on the blockchain sharding result after secondary division, a noise account division process is performed to obtain a final blockchain sharding result, which specifically includes:

[0040] According to the noise account set, the noise account is measured and divided to determine the transaction frequency of the noise account;

[0041] A transaction frequency threshold is defined, noise accounts with a transaction frequency greater than the transaction frequency threshold are marked as high-frequency noise accounts, and noise accounts with a transaction frequency less than the transaction frequency threshold are marked as low-frequency noise accounts;

[0042] The transaction association score of the high-frequency noise account and all shards in the blockchain sharding result after secondary division is obtained, and the high-frequency noise account is assigned to the shard corresponding to the highest transaction association score;

[0043] The low-frequency noise accounts are evenly distributed to all shards in the final blockchain sharding result by a random hash algorithm or are concentrated to a cold account shard.

[0044] Further, the expression of the transaction correlation score is specifically as follows:

[0045]

[0046] In the above formula, R(a i ,C q ) represents the transaction correlation score, a i represents a high-frequency noise account, C q represents an account set of the qth shard, M ij represents the transaction frequency of account a i with a j in the last epoch time window.

[0047] The second technical solution adopted by the application is a dynamic blockchain sharding system based on a composite clustering algorithm, comprising:

[0048] A first module is configured to determine a transaction frequency matrix and a shard allocation matrix based on the blockchain sharding system, and construct a target optimization transaction model of the blockchain sharding system;

[0049] A second module is configured to perform preliminary global division processing on the target optimization transaction model of the blockchain sharding system by using a hierarchical clustering algorithm, and obtain a preliminary blockchain sharding result;

[0050] A third module is configured to perform division processing on the preliminary blockchain sharding result by using a DBSCAN clustering algorithm, and obtain a secondary divided blockchain sharding result;

[0051] A fourth module is configured to perform noise account division processing based on the secondary divided blockchain sharding result, and obtain a final blockchain sharding result.

[0052] The method and system have the advantages that the application determines a transaction frequency matrix and a sharding allocation matrix based on a blockchain sharding system, constructs a target optimization transaction model of the blockchain sharding system, can minimize the proportion of cross-sharding transactions and realize load balancing, further based on the target optimization transaction model of the blockchain sharding system, performs preliminary global division processing through a condensed hierarchical clustering algorithm to obtain a preliminary blockchain sharding result, and performs division processing on the preliminary blockchain sharding result through a DBSCAN clustering algorithm to obtain a blockchain sharding result after secondary division, through a composite clustering algorithm, preferentially aggregates high-frequency associated accounts, maximizes the transaction density within the shard, significantly reduces the occurrence of cross-sharding transactions, reduces the cross-sharding communication cost, improves the system throughput, and finally based on the blockchain sharding result after secondary division, performs noise account division processing, through the classification processing of low-frequency accounts, designs a flexible allocation strategy. The low-frequency accounts are randomly and uniformly allocated or concentrated to a dedicated 'cold account shard', thereby realizing dynamic balancing of the load between shards, fully utilizing the computing power resources of the system, and further realizing dynamic balancing of the shard load and optimization of cross-sharding transactions. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a step flow chart of a dynamic blockchain sharding method based on a composite clustering algorithm according to the application;

[0054] Figure 2 is a structural block diagram of a dynamic blockchain sharding system based on a composite clustering algorithm according to the application;

[0055] Figure 3 is a framework schematic diagram of dynamic sharding division of accounts by a composite clustering algorithm according to the application;

[0056] Figure 4 is a schematic diagram of the proportion of repeated transaction account pairs and random account pairs according to the application. DETAILED DESCRIPTION

[0057] The application will be further described in detail below in combination with the drawings and specific embodiments. For the step numbers in the following embodiments, only the setting is for the convenience of description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] First of all, it needs to be pointed out that sharding load balancing is a key issue in blockchain sharding technology. The main challenge lies in how to reasonably allocate computing and storage resources in the network to ensure that the load of each shard can be effectively allocated, so as to avoid some shards being over-utilized while others are idle. Load imbalance not only wastes system resources, but also causes performance bottlenecks, especially when the network load is high, some shards may be congested due to handling too many requests, thereby affecting the response time and throughput of the entire blockchain system.

[0059] Secondly, while sharding technology improves the scalability of the blockchain system, it inevitably introduces an important challenge, namely the cross-shard transaction problem. Since sharding divides accounts or data into different shards, when data interaction involving multiple shards occurs, cross-shard transactions are generated. Compared with intra-shard transactions, the processing complexity of cross-shard transactions is significantly increased. First, the communication overhead increases, as cross-shard transactions require frequent information exchange between shards, increasing communication costs. Second, the processing flow is complicated, as the state update of cross-shard transactions involves multiple shard ledger modifications and consistency guarantees, usually requiring complex protocols (such as two-phase commit or lock-based protocols) to ensure transaction atomicity. This not only increases transaction processing time, but also may cause performance degradation due to operation conflicts.

[0060] To address the two technical challenges of high proportion of cross-shard transactions and load imbalance between shards, the present embodiment discovers a key feature by observing real-life transaction behavior: accounts with higher activity often have stronger dependencies, in other words, accounts that have transacted before are more likely to transact again in the future. This feature provides a valuable entry point for the present invention, based on which the historical transaction data between accounts can be used to infer the probability of future transactions, thereby optimizing the shard allocation strategy for accounts.

[0061] Based on this, referring to Figure 1 , the present invention provides a dynamic blockchain sharding method based on a composite clustering algorithm, which includes the following steps:

[0062] S100, based on a blockchain sharding system, determine a transaction frequency matrix and a shard allocation matrix, and construct a target optimization transaction model of the blockchain sharding system;

[0063] S110, based on the blockchain sharding system, obtain historical transaction data between all accounts, and construct a transaction frequency matrix;

[0064] Specifically, assume that the blockchain sharding system contains N accounts, denoted as A={a1,a2,…,a N}. The historical transactions between accounts can be represented by a transaction frequency matrix M∈RN×N denotes, where M ij denotes account a i with a j transaction frequency within the last epoch time window. For ease of processing, the transaction matrix needs to satisfy the following properties:

[0065] 1) Symmetry:

[0066]

[0067] The transaction frequency matrix is symmetric, indicating that the transaction frequency is bidirectional.

[0068] Non-negativity:

[0069]

[0070] The transaction frequency between accounts cannot be negative.

[0071] Diagonal is zero:

[0072]

[0073] S120, based on a blockchain sharding system, defines a cross-shard transaction total frequency and a system load balancing objective function;

[0074] Specifically, the system modeling research goal is to divide the account set A into K clusters (shards) S = {S1, S2, … S K}, so that the following two goals are met at the same time, one is cross-shard transaction minimization: accounts with high-frequency transactions are allocated to the same shard as much as possible; and load balancing, to ensure that the number of account transactions in each shard is roughly uniform, to balance computing resources.

[0075] Further, the objective function is modeled under the two goals.

[0076] First, cross-shard transaction minimization, the performance overhead brought by cross-shard transactions is the main performance bottleneck of the sharding system. Define δ ij as an indicator variable for whether accounts a i and a j belong to different shards, and its expression is:

[0077]

[0078] Further define the total cross-shard transaction frequency, and its expression is:

[0079]

[0080] The goal of the embodiment of the application is to minimize the total cross-shard transaction frequency, and its expression is:

[0081]

[0082] Secondly, load balancing, the shard load balancing requires the number of account transactions of each shard to be as uniform as possible. Define T p The total number of account transactions for shard p is |T p |. The goal of load balancing can be expressed as minimizing the maximum deviation of the number of account transactions of each shard, which is expressed as:

[0083]

[0084] Therefore, the load balancing goal is:

[0085]

[0086] In the above formula, B min represents the load balancing goal to be minimized.

[0087] S130, introduce a preset weight coefficient, combine the total frequency of cross-shard transactions with the system load balancing objective function and perform matrix representation to obtain a shard allocation matrix;

[0088] Specifically, to optimize the cross-shard transaction proportion and load balancing at the same time, a weight parameter λ∈[0,1] is introduced, and the comprehensive objective function is:

[0089] F(S)=λ·f c +(1-λ)·B

[0090] In the above formula, λ represents the trade-off between cross-shard transactions and load balancing.

[0091] S140, combine the transaction frequency matrix and the shard allocation matrix to represent the calculation process of cross-shard transactions, and construct a target optimization transaction model of the blockchain sharding system.

[0092] Specifically, to simplify the calculation, the objective function can be expressed in matrix form. Define the shard allocation matrix Z∈{0,1} N×K , where:

[0093]

[0094] The relationship between the transaction frequency matrix M and the shard allocation matrix Z is used to describe the calculation process of cross-shard transactions as:

[0095] f c =sum(M⊙(ZZ T =0))

[0096] Where ⊙ represents element-wise multiplication.

[0097] The load balancing target can be calculated by the row and column of Z, and its expression is:

[0098]

[0099] Then by combining the two, the target can be optimized to model:

[0100]

[0101] In the above formula, λ represents a weight coefficient, M represents a transaction frequency matrix, Z represents a shard allocation matrix, Z T represents the transposed matrix of the shard allocation matrix, i represents the i-th account, j represents the j-th account, Z ip = 1 indicates that the account i is allocated to the shard p, otherwise Z ip = 0, and sum(·) represents the total transaction frequency of all cross-shard transactions.

[0102] S200, based on the target optimization transaction model of the blockchain sharding system, a preliminary global division process is performed through the agglomerative hierarchical clustering algorithm to obtain a preliminary blockchain sharding result;

[0103] Specifically, in the target optimization transaction model based on the blockchain sharding system, a transaction frequency matrix is defined to obtain the similarity between accounts based on a transaction frequency distance expression; an account is regarded as a cluster, and an initial cluster set is constructed; based on the similarity between accounts, the inter-cluster distance of the initial cluster set is obtained through single linkage, and the minimum value of the inter-cluster distance and the number of clusters of the initial cluster set are determined; the initial cluster set is clustered through the agglomerative hierarchical clustering algorithm, and if the minimum value of the inter-cluster distance is greater than a preset inter-cluster distance threshold or the number of clusters does not meet a target value, the two clusters corresponding to the minimum value of the inter-cluster distance are iteratively merged, and the minimum value of the inter-cluster distance and the number of clusters are updated; until the updated minimum value of the inter-cluster distance is less than the preset inter-cluster distance threshold or the updated number of clusters meets the target value, a preliminary blockchain sharding result is obtained.

[0104] In this embodiment, the target optimization transaction model based on the blockchain sharding system assumes that the blockchain sharding system contains N accounts, denoted as A = {a1, a2, …, a N}. The historical transactions between accounts can be represented by a transaction frequency matrix M ∈ R N×N , where M ij represents the transaction frequency between account a i and a j in the last epoch time window. In order to measure the similarity between accounts, the invention defines a transaction frequency-based distance formula, whose expression is:

[0105]

[0106] where max(M) denotes the maximum value in the transaction frequency matrix, used for normalization to ensure that all distance values are within the interval [0, 1]. The smaller the distance value, the closer the transaction relationship between the two accounts.

[0107] Agglomerative Hierarchical Clustering is adopted, which starts with N single-account clusters and gradually merges the closest clusters to generate a hierarchical clustering structure. Initially, each account is an independent cluster, and the initial cluster set is C = { {a1}, {a2}, …, {aN}}. During the clustering process, the minimum distance between all clusters needs to be calculated each time. Single-Linkage is used to define the distance between clusters, i.e.: N

[0108]

[0109] where C i and C j represent two clusters, a p and a q are any accounts belonging to C i and C j respectively. After finding the two closest clusters C i and C j , they are merged into a new cluster C k = C i ∪C j , and the cluster set is updated as:

[0110] C←(C\{C i ,C j})∪{C k}

[0111] This process is iterated until the preset stopping condition is met. The stopping condition can be that the number of clusters reaches the target value k, or the minimum distance D(C i ,C j ) between clusters exceeds the preset threshold ρ. After clustering is completed, the system generates k clusters, each containing a group of accounts with high transaction frequency and close association with each other.

[0112] The result of hierarchical clustering provides a good initial grouping structure for subsequent refinement. Although this stage can effectively identify account groups with strong transaction relationships, it does not adequately consider the distribution of cluster internal density, which may miss some special noise accounts. Therefore, in the next stage, density clustering is combined to further refine, to identify core accounts in high-density clusters and noise accounts in low-density areas, thereby optimizing the quality of account division. ​

[0113] S300. The initial blockchain sharding results are divided using the DBSCAN clustering algorithm to obtain the secondary blockchain sharding results.

[0114] Specifically, a subset data matrix is ​​constructed based on the transaction frequency between accounts in the clusters in the preliminary blockchain sharding results, and the account distance in the subset data matrix is ​​determined. Based on the DBSCAN clustering algorithm, the neighborhood radius and minimum number of neighborhood points are defined. Each account in the preliminary blockchain sharding results is traversed using the DBSCAN clustering algorithm, and a neighborhood point set is constructed for each account based on the account distance and neighborhood radius in the subset data matrix. The number of neighborhood points for each account is determined based on its neighborhood point set and is then evaluated. If the number of neighborhood points for an account is greater than or equal to the minimum number of neighborhood points, the account is marked as a core account, and all accounts in the core account's neighborhood are added to the current cluster until no more accounts can be added, resulting in a density cluster. If the number of neighborhood points for an account is less than the minimum number of neighborhood points but belongs to a density cluster, the account is marked as a boundary account. If the number of neighborhood points for an account is less than the minimum number of neighborhood points and does not belong to a density cluster, the account is marked as a noise account. The density clusters, boundary accounts, and noise accounts are integrated to obtain the blockchain sharding results after the second partitioning.

[0115] In this embodiment, the present invention employs the DBSCAN clustering algorithm to further refine the results of the first-stage hierarchical clustering, in order to identify core accounts, boundary accounts, and noisy accounts that may exist in each cluster. DBSCAN is characterized by its density-based clustering method, which does not require pre-specifying the number of clusters. It can effectively identify clusters in high-density regions through local density information, while simultaneously detecting low-density noise points. This aligns perfectly with the dynamic account characteristics that require processing in blockchain sharding systems.

[0116] In this stage, each cluster generated by hierarchical clustering is first used as the input set for DBSCAN. Assume the hierarchical clustering result is divided into k initial clusters, denoted as C = {{C1}, {C2}, ..., {C...}. k For each cluster C i A subset data matrix M is constructed using the transaction frequency between accounts in the cluster as a feature. i In the DBSCAN processing, two key parameters are defined: the neighborhood radius ε and the minimum number of neighboring points MinPts. ε determines the distance threshold at which points are considered "neighbors," while MinPts defines the minimum number of points required to form a high-density region. Account a in the transaction frequency matrix... p With a q The distance can be measured by the following formula, which is expressed as:

[0117]

[0118] f(a p ,a q )=T i [p][q]

[0119] where T i [p][q] represents the transaction frequency between accounts a p and a q , the normalized distance measure d(a p ,a q ) ensures that the relative transaction closeness between accounts can effectively reflect their neighborhood relationship, and a r and a s represent the accounts used for normalization reference, which are respectively associated with a p and a q .

[0120] The first step of DBSCAN processing is to traverse all accounts in each cluster C i , and calculate the neighborhood point set N p (a ε ) of each account a p , which is defined as a set of accounts satisfying the following conditions, and the specific expression is:

[0121] N ε (a p )={a q ∈C i |d(a p ,a q )≤ε}

[0122] If the number of neighborhood points of a certain account |N ε (a p )|≥MinPts, then the account is marked as a core account (Core Account), indicating that it is at the center of a high-density area. With the core account as the seed, DBSCAN begins to expand the cluster: all accounts in its neighborhood are added to the current cluster, and the neighborhood of these newly added accounts is recalculated to find potential other core accounts for further expansion. This process is recursively repeated until no new accounts can be added, forming a complete dense cluster.

[0123] For those accounts with less than MinPts neighbors, if they exist in the neighborhood of a core account, they are marked as border accounts. Border accounts, although not the center of high-density areas, are somehow associated with high-density areas, so they can be included in the corresponding cluster. Those accounts that neither meet the core account condition nor belong to any core account neighborhood are marked as noise accounts. The identification process of noise accounts is an important feature of DBSCAN, and the goal is to find those accounts that are distributed in isolation, which may have an important impact on subsequent shard load balancing and cross-shard transaction optimization.

[0124] As DBSCAN is executed step by step, the initial cluster C generated by hierarchical clustering is further refined into higher resolution density clusters, and the accounts within each cluster are explicitly divided into core accounts, border accounts, and noise accounts, and finally a fine cluster division structure and noise account set of each cluster are output. The core of this stage is to make full use of density information to conduct in-depth analysis on account distribution, thereby improving the robustness of the system and the flexibility of shard division.

[0125] S400, based on the secondary division result of the blockchain shard, performing noise account division processing to obtain a final blockchain shard result.

[0126] Specifically, according to the noise account set, the noise accounts are measured and divided to determine the transaction frequency of the noise accounts; a transaction frequency threshold is defined, noise accounts with a transaction frequency greater than the transaction frequency threshold are marked as high-frequency noise accounts, and noise accounts with a transaction frequency less than the transaction frequency threshold are marked as low-frequency noise accounts; the transaction association score of all shards in the secondary division result of the blockchain shard and the high-frequency noise accounts is obtained, and the high-frequency noise accounts are assigned to the shard corresponding to the highest transaction association score; the low-frequency noise accounts are uniformly distributed to all shards in the secondary division result of the blockchain shard by using a random hash algorithm, and a final blockchain shard result is obtained.

[0127] In this embodiment, after hierarchical clustering and DBSCAN are completed, the accounts marked as noise are abnormal accounts that do not belong to any cluster. These accounts often cannot form a close transaction relationship cluster with other accounts due to the isolation or abnormality of their transaction behavior. Noise accounts are not completely equivalent to low-transaction-frequency accounts, which may include high-transaction-frequency but transaction-distribution-dispersed accounts, or low-transaction-frequency and transaction-object-sparse accounts. In order to more accurately process these accounts, we need to further classify the noise accounts to determine the nature of their transaction frequency, and then take targeted optimization strategies.

[0128] First, the noise accounts are measured and divided according to the transaction frequency. The transaction frequency of each account can be defined as the sum of all transaction times within a specified time window, denoted as S(a i ), and its expression is as follows:

[0129]

[0130] In order to distinguish high-frequency noise accounts and low-frequency noise accounts, the threshold value θ F of transaction frequency can be set as needed. When S(a i ) ≥ θ F , the account a i is considered a high-frequency noise account; otherwise, when S(a i ) ≤ θ F , it is considered a low-frequency noise account. The setting of threshold value θ F can be dynamically adjusted by the statistical distribution of transaction frequency, for example, using the upper quartile of transaction frequency as the basis for division.

[0131] The characteristics of high-frequency noise accounts are that their transaction frequency is high, but the transaction distribution is relatively dispersed, and there are transactions with accounts in multiple shards, which cannot be effectively attributed to a specific cluster. Low-frequency noise accounts, on the other hand, have a low total transaction frequency and usually have only sparse transactions with a small number of accounts. After such division, we can design appropriate processing strategies based on the above characteristics.

[0132] For high-frequency noise accounts, the transaction association score R(a i , C q ) between the account and the shard needs to be considered first to reduce the overhead of cross-shard transactions. The expression of the transaction association score is as follows:

[0133]

[0134] where C q represents the account set of the qth shard. The high-frequency noise account a i will be assigned to the shard C k with the highest association score, i.e.:

[0135]

[0136] This allocation strategy effectively reduces the frequency of cross-shard transactions by prioritizing the main transaction needs of high-frequency noise accounts, thereby significantly improving the transaction efficiency and throughput capacity of the system.

[0137] For low-frequency noise accounts, the allocation strategy can be flexibly selected according to actual needs. One method is to use a random hash algorithm to evenly distribute low-frequency noise accounts to each shard, thereby further achieving load balancing of account transactions between shards. Another method is to concentrate the allocation of these accounts to a dedicated cold shard. Designing such a shard to specifically handle low-frequency account transactions is highly consistent with the situation of uneven node computing power in the actual shard system. By having a shard composed of low-power nodes handle low-frequency noise accounts, the computing power resources in the system can be fully utilized, thereby achieving more efficient load balancing.

[0138] Through the classification and fine processing of noise accounts, the algorithm realizes efficient management of abnormal accounts, avoiding the decline of system performance caused by noise accounts. This strategy can effectively reduce the proportion of cross-shard transactions, balance shard load, and improve the throughput and response capability of the system.

[0139] In summary, the pseudo code flow of the clustering composite algorithm of the embodiment of the present application is shown in Table 1. First, the reference committee needs to convert all transaction matrices into a cluster C = {{C1}, {C2}, …, {C k}}, and use the agglomerative hierarchical clustering to select the two closest clusters from it each time to merge, generating a specified number of clusters (i.e. Line 1-7); then, further apply DBSCAN in each preliminary cluster using the preset parameters ε and MinPts for fine division to capture the local characteristics of transaction frequency within the cluster and identify noise accounts (i.e. Line 8-17); for noise accounts, they are divided into high-frequency and low-frequency categories according to their transaction frequency, high-frequency accounts are allocated to the optimal shard according to relevance, and low-frequency accounts are randomly allocated or concentrated (i.e. Line 18-22). Finally, the reconstructed account network can be broadcast to all shards, and each shard updates the local information to the latest status after receiving the broadcast.

[0140] Table 1 Pseudo code flow table of clustering composite algorithm

[0141]

[0142]

[0143] Finally, according to the present application, the embodiment of the present application uses a composite clustering algorithm to dynamically divide the accounts into shards, to realize the reconstruction of the shard of the blockchain system. The protocol execution is divided into five stages: user transaction request, node screening and preliminary clustering, cluster refinement and noise account processing, account partitioning and shard reconstruction, and shard result synchronization. Figure 3

[0144] ​1) User transaction request: Users submit transaction requests to the sharded blockchain system through the network, and transactions are allocated to the transaction pool for processing according to account relationships and shard mapping. Since the propagation time of transactions in the network is much smaller than the time required for shard consensus, it is assumed that transactions are synchronized to all shards. Each honest node manages a unified view of transactions according to the global transaction pool maintained locally. The transaction pool design includes a resource allocation mechanism to prevent network resource congestion and flooding attacks on a single shard, while providing stable data sources for subsequent shard division.

[0145] 2) Node screening and preliminary clustering: Nodes extract transaction data from the transaction pool and construct a transaction frequency matrix between accounts. Then, a new round of account shard division begins. First, hierarchical clustering is used for rough division. Referring to the transaction frequency matrix, nodes perform hierarchical clustering on accounts to preliminarily divide larger clusters. In this stage, the number of clusters does not need to be specified in advance. If a specific number of shards is desired, a global preliminary shard scheme can be obtained by truncating the clustering tree.

[0146] 3) In-cluster refinement and noise accounts: DBSCAN clustering method is run separately for each hierarchical clustering cluster to further refine the structure of the cluster. DBSCAN clustering method not only detects high-density areas in the cluster, but also identifies low-density isolated accounts (i.e., noise accounts). For noise accounts, they can be divided into high-frequency noise accounts and low-frequency noise accounts according to transaction frequency, and then different noise processing methods can be used according to the type of noise accounts to reduce their impact on shard performance and security.

[0147] 4) Account partitioning and shard reconstruction: The shard division result is executed and reconstructed by the reference committee at each epoch. The committee first verifies node identity, updates the active node list, and adjusts the shard ownership of some nodes according to the established rules to enhance system dynamics and security. Based on the results of hierarchical clustering and DBSCAN, the reference committee balances shard load according to transaction frequency and account association, optimizes inter-shard transaction flow, and completes the account shard scheme. After shard division, the current transaction pool state, account partition information, and shard state are written into the state block to provide data support for the next epoch.

[0148] 5) Shard result synchronization: In this stage, peer nodes update their states according to the information in the state block. Nodes determine whether to migrate shards based on the account partition results in the state block. For nodes that change shards, they send a join request to the new shard and update the ledger information and intra-shard neighbor relationships. After updating, the system starts a new round of consensus, and nodes process new transaction requests based on the latest shard information to ensure stable operation of the shard system.

[0149] Specifically, the embodiment of the present application fundamentally reduces the occurrence of cross-fragment transactions by analyzing the historical transaction relationship between accounts and dividing accounts with high correlation into the same fragment. At the same time, in order to avoid the problem of resource imbalance between fragments, the accounts with strong correlation are further distributed to each fragment through a reasonable uniform division strategy, thereby reducing cross-fragment transactions while achieving balanced allocation of fragment load. The research motivation provides a theoretical basis and practical direction for solving the key challenges in the practical application of fragmentation technology, and has important practical significance.

[0150] In order to verify whether the historical transaction relationship between accounts can reflect the probability of future transaction occurrence, the embodiment of the present application carries out experimental analysis based on 1000000 real historical transaction data of Ethereum. The experiment first extracts and analyzes the account pair relationship in the actual transaction data, counts the transaction frequency of each account pair, and calculates the proportion of repeated transaction pairs in the actual transaction pairs to verify the existence and degree of correlation between accounts in the real transaction network. Subsequently, by randomly generating a set of account pairs of the same size, it is checked whether these random account pairs appear in the historical transaction pairs, and the proportion of repeated transactions is calculated as a comparison benchmark.

[0151] The experimental results are shown in Figure 4 The repeated transaction proportion of the actual transaction pair is as high as 29.32%, while the repeated transaction proportion of the randomly generated account pair is only 0.92%. This significant difference shows that the correlation between accounts in the historical transaction pair is significantly higher than that of the random account pair, supporting the hypothesis that high-activity accounts have stronger dependence. The experiment further verifies the feasibility of using historical transaction data to infer the probability of future transaction occurrence, providing empirical support for the subsequent optimization of the fragmentation division scheme based on historical account transactions, and achieving dynamic balance of fragment load and optimization of cross-fragment transactions.

[0152] In summary, the embodiment of the present application has the following improvements compared to the prior art:

[0153] 1) By using a composite clustering algorithm, high-frequency correlated accounts are preferentially aggregated to maximize intra-fragment transaction density and significantly reduce the occurrence of cross-fragment transactions. This optimization reduces cross-fragment communication costs and improves system throughput.

[0154] 2) Through the classification processing of low-frequency accounts, a flexible allocation strategy is designed. Low-frequency accounts are randomly and uniformly allocated or concentrated to a dedicated "cold account fragment", thereby achieving dynamic balance of fragment load and fully utilizing the computing power resources of the system.

[0155] 3) Can accurately identify transaction anomalies of noise accounts, high-frequency noise accounts are isolated to the most relevant shards to prevent abnormal transactions from affecting the global; Low-frequency noise accounts are processed centrally, reducing their interference with core shards. This mechanism effectively improves the system's attack resistance.

[0156] Referring to Figure 2 A dynamic blockchain sharding system based on a composite clustering algorithm, comprising:

[0157] The first module 201 is configured to determine a transaction frequency matrix and a shard allocation matrix based on the blockchain sharding system, and construct a target optimization transaction model of the blockchain sharding system.

[0158] The second module 202 is configured to perform preliminary global division processing on the target optimization transaction model of the blockchain sharding system by using a hierarchical clustering algorithm, and obtain a preliminary blockchain sharding result.

[0159] The third module 203 is configured to perform division processing on the preliminary blockchain sharding result by using a DBSCAN clustering algorithm, and obtain a secondary divided blockchain sharding result.

[0160] The fourth module 204 is configured to perform noise account division processing based on the secondary divided blockchain sharding result, and obtain a final blockchain sharding result.

[0161] The content in the above method embodiments is applicable to the system embodiments, the system embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0162] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A dynamic blockchain sharding method based on a composite clustering algorithm, characterized in that, The method comprises the following steps: Based on the blockchain sharding system, the historical transaction data between all accounts is obtained, and a transaction frequency matrix is constructed; Based on the blockchain sharding system, a cross-shard transaction total frequency and a system load balancing target function are defined; A preset weight coefficient is introduced, the cross-shard transaction total frequency and the system load balancing target function are combined, and matrix representation is performed to obtain a shard allocation matrix; The transaction frequency matrix and the shard allocation matrix are combined to represent the calculation process of the cross-shard transaction, and a target optimization transaction model of the blockchain sharding system is constructed; The expression of the target optimization transaction model of the blockchain sharding system is specifically as follows: In the above formula, λ represents a weight coefficient, M represents a transaction frequency matrix, Z represents a shard allocation matrix, Z T represents a transposed matrix of the shard allocation matrix, i represents an i-th account, j represents a j-th account, Z ip represents whether the account i is allocated to a shard p, Z jq represents whether the account j is allocated to a shard q, and sum(·) represents a total transaction frequency of all cross-shard transactions. Based on the target optimization transaction model of the blockchain sharding system, a preliminary global division is performed through a condensed hierarchical clustering algorithm to obtain a preliminary blockchain sharding result; The preliminary blockchain sharding result is divided through a DBSCAN clustering algorithm to obtain a secondary divided blockchain sharding result; Based on the secondary divided blockchain sharding result, noise account division is performed to obtain a final blockchain sharding result.

2. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 1, characterized in that, The step of performing preliminary global division on the target optimization transaction model of the blockchain sharding system through the condensed hierarchical clustering algorithm to obtain the preliminary blockchain sharding result specifically comprises the following steps: Based on the transaction frequency matrix in the target optimization transaction model of the blockchain sharding system, a distance expression based on the transaction frequency is defined to obtain the similarity degree between accounts; An account is regarded as a cluster to construct an initial cluster set; Based on the similarity degree between accounts, the inter-cluster distance of the initial cluster set is obtained through a single-link method, and the minimum value of the inter-cluster distance and the cluster number of the initial cluster set are determined; The initial cluster set is clustered through the condensed hierarchical clustering algorithm, if the minimum value of the inter-cluster distance is greater than a preset inter-cluster distance threshold or the cluster number does not meet a target value, two clusters corresponding to the minimum value of the inter-cluster distance are iteratively merged, and the minimum value of the inter-cluster distance and the cluster number are updated; Until the minimum value of the updated inter-cluster distance is less than the preset inter-cluster distance threshold or the updated cluster number meets the target value, the preliminary blockchain sharding result is obtained.

3. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 2, characterized in that, The expression of the inter-cluster distance is specifically as follows: In the above formula, C i C j Indicate two clusters, a p With a q Indicates belonging to C i and C j Any account, d(a p ,a q ) represents a p Account and a q Account similarity, D(C) i C j ) represents C i Clusters and C j The distance between clusters.

4. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 3, characterized in that, The step of dividing the preliminary blockchain sharding result through the DBSCAN clustering algorithm to obtain the secondary divided blockchain sharding result specifically comprises the following steps: A subset data matrix is constructed according to the transaction frequency between accounts in the cluster in the preliminary blockchain sharding result, and the account distance in the subset data matrix is determined; Based on the DBSCAN clustering algorithm, a neighborhood radius and a minimum number of neighborhood points are defined; Each account in the preliminary blockchain sharding result is traversed through the DBSCAN clustering algorithm, and a neighborhood point set of each account is constructed according to the account distance in the subset data matrix and the neighborhood radius; The number of neighborhood points of the account is determined according to the neighborhood point set of the account and is judged; If the number of neighborhood points of the account is greater than or equal to the minimum number of neighborhood points, the account is marked as a core account, and all accounts in the neighborhood of the core account are added to the current cluster until no new account can be added, and a density cluster is obtained; If the neighborhood point number of the account is less than the minimum neighborhood point number and belongs to the density cluster, mark the account as a boundary account; If the neighborhood point number of the account is less than the minimum neighborhood point number and does not belong to the density cluster, mark the account as a noise account; Integrate the density cluster, the boundary account and the noise account to obtain the blockchain sharding result after secondary division.

5. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 4, characterized in that, The expression of the account distance in the subset data matrix is specifically as follows: f(a p ,a q ) = T i [p][q] In the above formula, d(a p ,a q ) represents the account distance in the subset data matrix, T i [p][q] represents the transaction frequency between accounts a p and a q , f(·) represents the transaction frequency function, and a r and a s represent accounts used for normalization reference.

6. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 5, characterized in that, The step of performing noise account division processing based on the blockchain sharding result after secondary division to obtain the final blockchain sharding result specifically includes: According to the noise account set, measuring and dividing the noise accounts to determine the transaction frequency of the noise accounts; Defining a transaction frequency threshold, marking the noise accounts with a transaction frequency greater than the transaction frequency threshold as high-frequency noise accounts, and marking the noise accounts with a transaction frequency less than the transaction frequency threshold as low-frequency noise accounts; Obtaining the transaction correlation score of the high-frequency noise accounts and all shards in the blockchain sharding result after secondary division, and distributing the high-frequency noise accounts to the shard corresponding to the highest transaction correlation score; Distributing the low-frequency noise accounts to all shards in the blockchain sharding result after secondary division or concentrating the low-frequency noise accounts to a cold account shard by using a random hash algorithm to obtain the final blockchain sharding result.

7. The dynamic blockchain sharding method based on the composite clustering algorithm according to claim 6, characterized in that, The expression of the transaction correlation score is specifically as follows: In the above formula, R(a i , a q ) represents the transaction correlation score, a i represents a high-frequency noise account, C q represents an account set of the qth slice, M ij represents the transaction frequency of account a i with account a j in the previous epoch time window. 8.A dynamic blockchain sharding system based on a composite clustering algorithm, characterized in that, The following modules are included: The first module is configured to obtain historical transaction data between all accounts based on a blockchain sharding system, and construct a transaction frequency matrix; Defining a cross-shard transaction total frequency and a system load balancing objective function based on the blockchain sharding system; Introducing a preset weight coefficient, combining the cross-shard transaction total frequency and the system load balancing objective function, and performing matrix representation to obtain a shard allocation matrix; Combining the transaction frequency matrix and the shard allocation matrix to represent the calculation process of cross-shard transactions, and constructing a target optimization transaction model of the blockchain sharding system; The expression of the target optimization transaction model of the blockchain sharding system is specifically as follows: In the above formula, λ represents a weight coefficient, M represents a transaction frequency matrix, Z represents a shard allocation matrix, Z T represents a transposed matrix of the shard allocation matrix, i represents an i-th account, j represents a j-th account, Z ip represents whether the account i is allocated to a shard p, Z jq represents whether the account j is allocated to a shard q, sum(·) represents a total transaction frequency of all cross-shard transactions; The second module is configured to perform preliminary global division processing on the target optimization transaction model of the blockchain sharding system by using a condensed hierarchical clustering algorithm to obtain a preliminary blockchain sharding result; The third module is configured to perform division processing on the preliminary blockchain sharding result by using a DBSCAN clustering algorithm to obtain a blockchain sharding result after secondary division; The fourth module is configured to perform noise account division processing based on the blockchain sharding result after secondary division to obtain a final blockchain sharding result.

Images