Distributed sharding and load balancing method and system based on dynamic hypergraph partitioning

Abstract

The application discloses a distributed sharding and load balancing method and system based on dynamic hypergraph partitioning, and the method comprises the following steps: acquiring historical transaction records and constructing a unified hypergraph model; counting the load weight of each vertex, extracting candidate hub nodes and classifying them into global credential hubs or strong state application hubs, and respectively adopting an asynchronous lock-free state management strategy and an anchor master shard strategy; after removing the global credential hubs, extracting a local sub-hypergraph containing only strong state application hubs, and determining the master shard as a guide anchor by using an offline partitioning algorithm; for the remaining movable ordinary nodes, a streaming greedy algorithm is adopted, a comprehensive score is calculated based on a positive attraction score and a negative load penalty term, and the nodes are distributed to the shard with the highest score. The application reduces the cross-shard transaction rate, realizes real computing power load balancing, and improves the algorithm execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of distributed systems and large-scale data processing technology, and more specifically, to a distributed sharding and load balancing method and system based on dynamic hypergraph partitioning. Background Technology

[0002] With the development of applications such as distributed collaborative computing, public distributed ledger systems face severe performance bottlenecks. State sharding is widely recognized as one of the best solutions for improving the scalability of distributed systems. State sharding distributes the node states and computational tasks of the entire network across multiple parallel shards, which are then processed in parallel by different computing clusters. However, if a distributed transaction (business request) involves nodes distributed across different shards, cross-shard transactions (CTX) occur. Cross-shard transactions require a complex two-phase commit (2PC) protocol to guarantee atomicity, which leads to extremely high communication latency. Therefore, how to rationally partition node data to minimize the cross-shard transaction ratio (CSR) is the core challenge of sharding technology.

[0003] Existing technologies mainly use the following methods for partitioning, which have obvious drawbacks: 1. Hash / Random Partitioning: Randomly assigned based on the hash value of the node address.

[0004] Disadvantages: Completely ignoring the business call relationships between nodes leads to a cross-shard transaction rate of over 90%, resulting in a decrease in system performance instead of an increase.

[0005] 2. Partitioning based on ordinary graphs: Abstracting historical call records into edges between two points (ordinary graph), and using graph partitioning algorithms.

[0006] Disadvantages: It disrupts the "atomicity" of distributed business requests. Complex distributed applications often involve cascading calls to multiple logical components in a single request (e.g., node A calls component B, and B in turn calls component C). Ordinary graphs forcibly break down multi-party calls into binary edges, losing complete call chain information.

[0007] 3. Hypergraph partitioning based on request count: Use hyperedges to wrap all nodes participating in the same transaction to protect atomicity.

[0008] Disadvantage 1 (Ignoring Real Computational Costs): Using only the number of requests as the basis for load assessment ignores the complex execution costs (such as gas consumption) of different business logics in a programmable network. This leads to computationally intensive business nodes being treated the same as simple state transition nodes, causing some shards to crash due to exhaustion of computing power.

[0009] Disadvantage 2 (Ignoring Business Semantics and Long Tail Effect): Real distributed ledger networks are scale-free networks, with a few "super hubs" (such as basic digital credentials / common components) participating in massive transactions. Existing algorithms treat basic resource flows and complex collaborative computing components equally, causing partitioning algorithms to experience memory overflows when processing these super hubs, or generating massive amounts of pseudo-cross-shard traffic in order to accommodate super hubs.

[0010] Chinese invention application No. 202211424549.4 discloses "A Blockchain Dynamic Sharding Expansion Method Based on Graph Flow Segmentation," which includes: Step 1: A node calculates the target Nonce value that meets the current Target by running a hash function, and performs a modulo operation based on the current number of shards. The result is the shard number to which the node belongs. The node needs to initiate an admission transaction. After the transaction is successfully packaged into a block, the node can initiate a formal transaction in the next epoch; Step 2: During the transaction process, each shard packages the transaction data in its own transaction pool and produces a block. The first transaction block of each epoch is preceded by the state block of the previous epoch. The block structure of the entire system is a topological graph structure, and each epoch converges to a state. Step 3: Execute the graph flow segmentation algorithm. Each node continuously reads transaction information from the newly generated blocks and updates the state graph of all current nodes. When the current epoch ends, the state graph is also determined. The transaction information serves as the input to the graph flow segmentation algorithm, including the state graph and the transaction account. Step 4: After the graph state and account are segmented in Step 3, the blockchain system reaches a consensus on the segmentation result. This consensus stage uses the PBFT algorithm and generates a state block. This state block serves as the convergence block for all current shards, indicating the end of the current epoch. Step 5: After the state block of the current epoch reaches a consensus, the associated shards change their states according to the block content to start a new round of transactions. This method uses a normal graph (with transaction input and output sides as edges) for graph flow segmentation, losing the complete atomicity of multi-node cascading calls. Summary of the Invention

[0011] To address the technical problems existing in the background art, this invention provides a distributed sharding and load balancing method and system based on dynamic hypergraph partitioning. The technical solution adopted by this invention is as follows: The first aspect of this invention provides a distributed sharding and load balancing method based on dynamic hypergraph partitioning, the method comprising: Obtain historical transaction records and construct a unified hypergraph model H = (V, E, , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; The total load weight of each vertex in the vertex set V is calculated, and the top-ranked vertices with a preset proportion are extracted as candidate hub nodes. Based on the business logic characteristics of each candidate hub node, they are classified as global credential hubs or strong state application hubs. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. After removing the global credential hub from the unified hypergraph model, a local induced sub-hypergraph containing only the strong state application hub is extracted. The local induced sub-hypergraph is pre-assigned using an offline multi-level hypergraph partitioning algorithm to determine the primary partition to which each strong state application hub belongs, and each primary partition is used as a guiding anchor point for the subsequent streaming partitioning stage. For the movable ordinary nodes in the vertex set V other than the global credential hub and the strong state application hub, a streaming greedy algorithm is used to allocate them sequentially. For each node to be allocated, the comprehensive score of its allocation to each candidate shard is calculated, and it is allocated to the shard with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the shard's real-time load.

[0012] As a preferred embodiment, the vertex set V includes external request nodes and internal logical component nodes, each hyperedge e in the hyperedge set E corresponds to a distributed transaction and contains all nodes that interact within that transaction, and the hyperedge weight set... The weights of each hyperedge in ( The actual computing power consumption of this distributed transaction is represented by the vertex load weight set. Vertex load weights ( It is obtained by accumulating the weights of each hyperedge containing the vertex after amortization.

[0013] As a preferred embodiment, the vertex load weight ( The calculation method for ) is as follows:

[0014] in, Let E(v) be the set of all superedges containing vertex v.

[0015] As a preferred approach, methods for classifying candidate hub nodes into global credential hubs or strong-state application hubs based on their business logic characteristics include: If a candidate hub node contains an operation in its historical transactions that matches the preset voucher operation feature code rule base, and the cumulative computing power consumption value of the operation accounts for a proportion of the node's total consumption value that is greater than a preset threshold τ, then it is determined to be a global voucher hub. Otherwise, it is determined to be a strong state application hub.

[0016] As a preferred embodiment, the asynchronous lock-free state management strategy is as follows: The voucher quota status of the global voucher hub is stored in the respective shards according to the request initiator. When cross-shard transfer occurs, the source shard directly deducts the voucher quota and generates a status certificate. The target shard verifies the certificate and increases the corresponding quota. There is no need to execute the cross-shard two-phase commit protocol. The anchor point master sharding strategy is as follows: Each strong-state application hub is assigned a unique primary shard, and only this primary shard is granted exclusive write permissions to its core state. Cross-shard call requests from other shards are forwarded to this primary shard for unified execution via asynchronous message routing.

[0017] As a preferred embodiment, the offline multi-level hypergraph partitioning algorithm aims to minimize cross-shard communication costs while satisfying load balancing constraints; wherein, the objective function for minimizing cross-shard communication costs is:

[0018] Wherein, the range of summation This indicates that all hyperedges in the local induced hypergraph have been traversed. For super-edge In mapping relationship The number of independent shards spanned below; The load balancing constraints are:

[0019] Among them, the summation term This represents the sum of the total load weights of all strong-state application hub nodes participating in the offline partitioning.

[0020] As a preferred embodiment, the formula for calculating the comprehensive score is as follows: =

[0021] in: The positive gravity score is calculated using the following formula: =

[0022] Wherein, the range of summation This indicates traversing the entire superatlas. All nodes containing pending allocation Historical super-border, For containing nodes to be assigned Historical super-border, This represents the computing power weight of the hyperedge. This superedge has already been allocated to a partition. The number of neighboring nodes, This represents the total number of nodes contained in the superedge. For the guiding coefficient, if the superedge Includes fragments that have been pre-allocated offline. Strong state application hub anchors will grant multiplied rewards. Let it be a constant. ,otherwise Set to 1; The negative load penalty is calculated using the following formula: =

[0023] in, As a penalty regulator, This represents the accumulated computing load of shard k under the current allocation progress. The system's average expected load is the total load of all nodes in the network. Divide by the number of fragments K.

[0024] A second aspect of the present invention provides a distributed sharding and load balancing system based on dynamic hypergraph partitioning, the system comprising: The hypergraph building module is used to obtain historical transaction records and construct a unified hypergraph model H = (V, E, ... , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; The node classification module is used to calculate the total load weight of each vertex in the vertex set V, extract the top-ranked vertices as candidate hub nodes, and classify each candidate hub node into a global credential hub or a strong state application hub according to the business logic characteristics of each candidate hub node. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. The offline pre-allocation module is used to extract a local induced sub-hypergraph containing only the strong state application hub after removing the global credential hub from the unified hypergraph model, and to pre-allocate the local induced sub-hypergraph using an offline multi-level hypergraph partitioning algorithm to determine the main partition to which each strong state application hub belongs, and to use each main partition as a guiding anchor point for the subsequent streaming partitioning stage. The streaming dynamic partitioning module is used to sequentially allocate movable ordinary nodes in the vertex set V, excluding the global credential hub and the strong state application hub, using a streaming greedy algorithm. For each node to be allocated, the module calculates its comprehensive score for allocation to each candidate partition and allocates it to the partition with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the real-time load of the partition.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned distributed sharding and load balancing method based on dynamic hypergraph partitioning.

[0026] A fourth aspect of the present invention provides a computer device, including a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the aforementioned distributed sharding and load balancing method based on dynamic hypergraph partitioning.

[0027] Compared with the prior art, the beneficial effects of this invention are: This invention employs hyperedge weight modeling based on actual computing power consumption (Gas), ensuring that shard load assessment matches actual computing resource requirements. This avoids computational hotspots caused by partitioning based solely on request counts, achieving true computing power load balancing across shards. By preserving complete transaction cascading call relationships through hyperedges, the atomicity of distributed transactions is protected. Compared to ordinary graph partitioning methods that decompose multi-party calls into binary edges, this significantly reduces the proportion of cross-shard transactions. Function signatures are used to identify and separate global credential hubs from strong-state application hubs. Asynchronous lock-free state sharding is applied to credential hubs that satisfy the commutative law, eliminating pseudo-cross-shard communication overhead and avoiding unnecessary cross-shard costs caused by high-frequency access to common components. An anchor-based master sharding strategy is adopted for strong-state application hubs, using them as gravitational anchors for streaming partitioning to guide high-frequency interactive ordinary nodes to follow the allocation, further reducing the cross-shard transaction rate, achieving an improvement of approximately 45% compared to the current best hypergraph algorithm. The two-stage architecture of "offline pre-allocation + online streaming follow" reduces the allocation complexity of massive ordinary nodes to linear O(N), completely solving the memory overflow problem caused by super hubs in traditional multi-level graph partitioning. The resharding runtime efficiency is improved by 5-7 times, meeting the second-level scheduling requirements. Attached Figure Description

[0028] Figure 1 The flowchart of the distributed sharding and load balancing method based on dynamic hypergraph partitioning provided in this embodiment is shown. Detailed Implementation

[0029] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0030] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0032] Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. The invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] Example 1 Please refer to Figure 1 This embodiment provides a distributed sharding and load balancing method based on dynamic hypergraph partitioning, the method including: S1: Obtain historical transaction records and construct a unified hypergraph model H = (V, E, ... , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; In one specific embodiment, the vertex set V includes external request nodes and internal logical component nodes, each hyperedge e in the hyperedge set E corresponds to a distributed transaction and contains all nodes that interact within that transaction, and the hyperedge weight set... The weights of each hyperedge in ( The actual computing power consumption of this distributed transaction is represented by the vertex load weight set. Vertex load weights ( It is obtained by accumulating the weights of each hyperedge containing the vertex after amortization.

[0035] In one specific embodiment, the vertex load weight ( The calculation method for ) is as follows:

[0036] in, Let E(v) be the set of all superedges containing vertex v.

[0037] Specifically, step S1 includes: Obtain all historical transaction records from the previous scheduling period and construct a hypergraph H = (V, E, , The definitions of each item are as follows: Record Acquisition and Storage Format: Historical transaction records are stored as a list of distributed transaction execution trace (TransactionTrace) logs or in JSON format. The specific fields extracted from each record include: transaction hash, initiator address, a list of addresses of all cascaded internal logic components, total transaction computational power consumption (Gas Used), and the first 4 bytes of the call input data (Function Selector, function signature identifier).

[0038] Vertex set V: includes all external request nodes (such as end users) and internal logical component nodes (such as programmable code nodes).

[0039] Hyperedge set E: Maps each distributed transaction to a corresponding hyperedge e. This hyperedge e encompasses all nodes that interacted during the execution of this transaction (including the request initiating node and all cascading internal logic component nodes). Since a transaction involves at least an initiator and a callee, the total number of vertices contained in each hyperedge (i.e., the hyperedge cardinality) is... satisfy ≥2.

[0040] Hyperedge weight set Extract the actual computing power consumption (e.g., Gas value) during the execution of this distributed transaction as the superedge weight. ( ).

[0041] Vertex load weight set The computational cost of a given hyperedge is distributed evenly among all vertices participating in that hyperedge using a simple arithmetic average. A given vertex... Total computational load weight ( The calculation method is as follows: accumulate the average value of all hyperedges that the vertex participates in, that is... ,in Let E(v) be the set of all superedges containing vertex v.

[0042] S2: Calculate the total load weight of each vertex in the vertex set V, extract the top-ranked vertices as candidate hub nodes, and classify each candidate hub node into a global credential hub or a strong state application hub according to the business logic characteristics of each candidate hub node. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. In one specific embodiment, the method for classifying candidate hub nodes into global credential hubs or strong state application hubs based on their business logic characteristics includes: If a candidate hub node contains an operation in its historical transactions that matches the preset voucher operation feature code rule base, and the cumulative computing power consumption value of the operation accounts for a proportion of the node's total consumption value that is greater than a preset threshold τ, then it is determined to be a global voucher hub. Otherwise, it is determined to be a strong state application hub.

[0043] In a specific embodiment, the asynchronous lock-free state management strategy is as follows: The voucher quota status of the global voucher hub is stored in the respective shards according to the request initiator. When cross-shard transfer occurs, the source shard directly deducts the voucher quota and generates a status certificate. The target shard verifies the certificate and increases the corresponding quota. There is no need to execute the cross-shard two-phase commit protocol. The anchor point master sharding strategy is as follows: Each strong-state application hub is assigned a unique primary shard, and only this primary shard is granted exclusive write permissions to its core state. Cross-shard call requests from other shards are forwarded to this primary shard for unified execution via asynchronous message routing.

[0044] Specifically, step S2 includes: Calculate the total load weight of each vertex within the period. ( Sort the nodes in descending order and extract the top 0.05% of high-load nodes as a candidate hub node set. For each candidate hub node, business logic characteristics are determined based on the following algorithm and rule base: The first category (Global Credential Hub): The system has a built-in credential operation feature code rule base (CredentialSig), which includes function feature codes that satisfy the commutative law for operations such as transfer (e.g., 0xa9059cbb), approval (e.g., 0x095ea7b3), or issuance. The algorithm traverses all historical transactions of the candidate node. If it contains a feature code from the rule base, the computational cost of the corresponding transaction is accumulated. If the accumulated cost of the operation accounts for a proportion of the node's total cost greater than a preset threshold τ, it is determined to be a "Global Credential Hub".

[0045] The implementation mechanism of the global asynchronous state sharding strategy is as follows: A "QuotaSharding" model is adopted for this type of hub. Logically, it is globally visible, but physically, a lock-free "Cancel-and-Issue" asynchronous model is used. Specifically, the user's specific credential quota status is retained in the local shard where the request originated. When a cross-shard transfer occurs, the source shard directly deducts (cancels) the credential quota and generates a state proof. The target shard verifies this proof and then increases (issues) the corresponding quota. Since this process does not involve strong consistency two-phase commit (2PC) lock contention across shards, it is directly removed from the cross-shard penalty in the subsequent hypergraph partitioning cost calculation.

[0046] The second category (strong state application hubs): If a candidate node does not meet the above global credential conditions (e.g., components requiring strong consistency such as multi-party collaborative scheduling centers and shared resource pools), it is determined to be an application logic hub with strong state dependencies and sequential execution requirements. It is then classified into the "fixed node set" (denoted as...). ).

[0047] The implementation mechanism of the anchor point primary sharding strategy: A "home sharding" mode is adopted for this type of hub. That is, each strongly stateful application hub is assigned a unique primary shard as its "anchor point," and only this primary shard is granted exclusive write permissions to the core state of that component. Cross-shard calls from other non-home shards cannot directly modify its state; instead, they must be routed to the primary shard via an asynchronous message mechanism, where the primary shard executes the requests serially to ensure state consistency.

[0048] Remaining nodes: Nodes that were not selected as candidate hubs are included in the "MovableNodes" set.

[0049] S3: After removing the global credential hub from the unified hypergraph model, extract the local induced sub-hypergraph that contains only the strong state application hub. Use the offline multi-level hypergraph partitioning algorithm to pre-allocate the local induced sub-hypergraph, determine the main partition to which each strong state application hub belongs, and use each main partition as the guiding anchor point for the subsequent streaming partitioning stage. In one specific embodiment, the offline multi-level hypergraph partitioning algorithm aims to minimize cross-shard communication cost while satisfying load balancing constraints; wherein, the objective function for minimizing cross-shard communication cost is:

[0050] Wherein, the range of summation This indicates that all hyperedges in the local induced hypergraph have been traversed. For super-edge In mapping relationship The number of independent shards spanned below; The load balancing constraints are:

[0051] Among them, the summation term This represents the sum of the total load weights of all strong-state application hub nodes participating in the offline partitioning.

[0052] Specifically, step S3 includes: From the constructed global unified hypergraph, extract only the "strong state application hub" mentioned above. The local induced sub-hypergraph of the region is extracted as follows: vertices belonging to strong state application hubs are retained in the hyperedges, and other nodes are removed; hyperedges with a cardinality less than 2 after processing are filtered out.

[0053] Using offline multilevel hypergraph partitioning algorithms (such as the hMETIS algorithm), find an optimal mapping relationship between nodes and shards. : These application hubs with strong business relevance are pre-assigned to K shards of the system, with mapping relationships... Represents a node The assigned shard number.

[0054] - Optimize the objective function: minimize the total effective cross-slice communication cost. The range of summation is given. This indicates traversing all hyperedges in the local induced hypergraph; For super-edge In mapping relationship The number of independent segments spanned below (i.e., effective slice size, which does not include the global credential hub node mentioned above).

[0055] Constraint parameter settings and load balancing calculation: Input parameters are the induced hypergraph, number of fragments K, and allowable imbalance factor. Computational load balancing means that the partitioning result must meet a hard constraint: the total computational load of any partition k must be... Among them, the summation term This represents the sum of the total load weights of all strong-state application hub nodes participating in the offline partitioning.

[0056] The partitioning results are locked within this cycle and serve as "gravitational anchors" for subsequent streaming partitioning.

[0057] S4: For the movable ordinary nodes in the vertex set V other than the global credential hub and the strong state application hub, a streaming greedy algorithm is used to allocate them sequentially. For each node to be allocated, the comprehensive score of its allocation to each candidate shard is calculated, and it is allocated to the shard with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the real-time load of the shard. In one specific embodiment, the formula for calculating the comprehensive score is: =

[0058] in: The positive gravity score is calculated using the following formula: =

[0059] Wherein, the range of summation This indicates traversing the entire superatlas. All nodes containing pending allocation Historical super-border, For containing nodes to be assigned Historical super-border, This represents the computing power weight of the hyperedge. This superedge has already been allocated to a partition. The number of neighboring nodes, This represents the total number of nodes contained in the superedge. For the guiding coefficient, if the superedge Includes fragments that have been pre-allocated offline. Strong state application hub anchors will grant multiplied rewards. Let it be a constant. ,otherwise Set to 1; The negative load penalty is calculated using the following formula: =

[0060] in, As a penalty regulator, This represents the accumulated computing load of shard k under the current allocation progress. The system's average expected load is the total load of all nodes in the network. Divide by the number of fragments K.

[0061] Specifically, step S4 includes: For a massive number of "movable ordinary nodes," a streaming greedy algorithm with a computational complexity of only O(N) is used to allocate them sequentially. For each ordinary node to be allocated... Calculate its allocation to the candidate fragments. Overall score It is then assigned to the system shard with the highest score.

[0062] The formula for the overall score is: =

[0063] The definitions of each calculation item are as follows: Positive gravity score The formula is: = The range of summation is given. This indicates traversing the entire superatlas. All nodes containing pending allocation Historical super-border; For containing nodes to be assigned Historical super-border, This represents the computing power weight of the hyperedge; This superedge has already been allocated to a partition. The number of neighboring nodes; This represents the total number of nodes contained in the superedge. For the guiding coefficient, if the superedge Includes fragments that have been pre-allocated offline. The "strong state application hub" anchor point is then given multiplied rewards. Let it be a constant. ,like 1.5; otherwise Set to 1. This setting encourages ordinary nodes to prioritize following the core business components they interact with most frequently.

[0064] Negative load penalty (Load Penalty): The formula is = .in, As a penalty regulator; This represents the accumulated computing load of shard k under the current allocation progress. The system's average expected load (its value is the total load of all nodes in the entire network). (Divide by the number of shards K). This term constrains the shard load in real time. If the target shard load approaches or exceeds the system average upper limit, the penalty term increases sharply, prompting subsequent nodes to flow to shards that are in an idle state.

[0065] Specific example of calculating the overall score: Assume the system has K=2 shards (shard 1 and shard 2). Set system parameters: anchor reward coefficient β=1.5, penalty adjustment factor α=10, and average expected load L_avg=1000.

[0066] Now we process the ordinary node u to be assigned. Querying the records shows that node u only participated in one historical superedge e_1.

[0067] The superedge e_1 contains 4 nodes {u, A, B, C} (i.e., cardinality |e_1|=4), and the total computational power consumption weight of the transactions it represents is ω_E(e_1)=120.

[0068] Suppose that nodes A and B are strong stateful application hubs that have been pre-assigned offline to "shard 1" (i.e., there are anchor points in shard 1); while node C is a regular node that has been streamed to "shard 2".

[0069] At this point, the scores for node u assigned to the two shards are calculated based on the system state: 1. Calculate the score S_curr(u, 1) assigned to "shard 1": - Since there are already two nodes (A and B) inside the superedge in partition 1, n_1 = 2.

[0070] - Fragment 1 contains anchor points, so the guiding coefficient δ(e_1, 1) = β = 1.5.

[0071] - Positive gravitational force A(u, 1) = 120 * 1.5 * [2 / (4-1)] = 120 * 1.5 * (2 / 3) = 120.

[0072] - Assuming that the current shard 1 has accumulated a load of L_1=500, then the negative penalty P(1) = 10 * (500 / 1000) = 5.

[0073] - Final score S_curr(u, 1) = 120 - 5 = 115 points.

[0074] 2. Calculate the score S_curr(u, 2) assigned to "shard 2": - Since there is already one node (C) inside the superedge in partition 2, n_2 = 1.

[0075] - Fragment 2 does not contain the anchor point of the superedge, so the guiding coefficient δ(e_1, 2) = 1.

[0076] - Positive gravitational force A(u, 2) = 120 * 1 * [1 / (4-1)] = 120 * 1 * (1 / 3) = 40.

[0077] - Assuming that the current shard 2 has accumulated a load of L_2=800, then the negative penalty P(2) = 10 * (800 / 1000) = 8.

[0078] - Final score S_curr(u, 2) = 40 - 8 = 32 points.

[0079] Since 115 points > 32 points, the system ultimately decides to assign node u to shard 1, thus achieving the goal of "nodes following high-frequency interactive core business components (A, B)".

[0080] Example 2 Please refer to Figure 1 This embodiment provides a distributed sharding and load balancing method based on dynamic hypergraph partitioning, the method including: S1: Obtain historical transaction records and construct a unified hypergraph model H = (V, E, ... , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; S2: Calculate the total load weight of each vertex in the vertex set V, extract the top-ranked vertices as candidate hub nodes, and classify each candidate hub node into a global credential hub or a strong state application hub according to the business logic characteristics of each candidate hub node. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. S3: After removing the global credential hub from the unified hypergraph model, extract the local induced sub-hypergraph that contains only the strong state application hub. Use the offline multi-level hypergraph partitioning algorithm to pre-allocate the local induced sub-hypergraph, determine the main partition to which each strong state application hub belongs, and use each main partition as the guiding anchor point for the subsequent streaming partitioning stage. S4: For the movable ordinary nodes in the vertex set V other than the global credential hub and the strong state application hub, a streaming greedy algorithm is used to allocate them sequentially. For each node to be allocated, the comprehensive score of its allocation to each candidate shard is calculated, and it is allocated to the shard with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the shard's real-time load.

[0081] This embodiment uses a distributed collaborative processing system with four shards as an example to illustrate the data sharding and load balancing method based on dynamic hypergraph partitioning described in this invention. This system contains a large number of external request nodes and multiple internal business component nodes. During the execution of business requests, multi-level cascading calls may be triggered. The system performs dynamic repartitioning once according to a fixed scheduling cycle. Each scheduling cycle can correspond to a preset time window, a preset batch number, or a preset block range.

[0082] In this embodiment, the system first collects historical request execution records from the previous scheduling cycle. Each historical request record includes at least: the identifier of the request initiating node, the identifier of the accessed business component, the internal cascading call chain that occurred during the request execution, and the resource consumption value corresponding to the request. The resource consumption value can be processor time, number of executed instructions, scheduling weight, memory overhead, or a uniformly converted computing resource value. In a preferred embodiment, all nodes involved in a complete business request are mapped together as a hyperedge, and the resource consumption value of the request is used as the hyperedge weight; simultaneously, the hyperedge weight is distributed to each node within the hyperedge according to the number of participating nodes to generate node load weights.

[0083] For example, within a certain scheduling period, if service request R1 is initiated by user node U1, sequentially calling service components A, B, and C, then a hyperedge e1 = {U1, A, B, C} is constructed. If the resource consumption value corresponding to this request is 120, then the weight of hyperedge e1 is recorded as 120, and 120 / 4 is included in the node load weights of U1, A, B, and C respectively. As another example, if service request R2 is initiated by user node U2, calling service components A and D, then a hyperedge e2 = {U2, A, D} is constructed, and its resource consumption value is included in the corresponding node in the same way. Through this modeling method, the multi-node cascading call relationship can be fully preserved, rather than breaking down a single atomic request into several binary edges.

[0084] After completing the unified hypergraph construction, the system calculates the cumulative load of each node within the current scheduling cycle and sorts them from highest to lowest cumulative load, selecting the top-ranked nodes with a predetermined percentage of high load as candidate hub nodes. In this embodiment, the predetermined percentage can be 0.05%, but is not limited to this. For each candidate hub node, the system further analyzes the operational characteristics of its historical requests. These operational characteristics can be identified by request identifier, interface number, function signature, call type, input parameter pattern, or a predefined rule base.

[0085] If most operations performed by a candidate hub node satisfy the commutative law or are independent of each other and do not cause shared write conflicts, such as credential posting, authorization registration, quota updates, and quota revocation for independent accounts, then this node is identified as a first-type hub node, i.e., a global credential hub. A global asynchronous state sharding strategy is adopted for this type of node: its global description information is visible in each shard logic, while specific state entries are stored in the local shard according to the request initiator's affiliation, so that access to this type of node between different shards does not require a strong synchronization locking protocol. Accordingly, in subsequent partitioning cost calculations, access relationships involving this type of node are not included or are only weakly included in cross-shard penalties to eliminate pseudo-cross-shard traffic caused by high-frequency public resource access.

[0086] If a candidate hub node does not meet the above conditions, but its internal state has strong dependencies and the order of request execution affects subsequent results (e.g., shared business pools, collaborative scheduling centers, unified matching engines, complex rule calculation units), it is identified as a second type of hub node, namely a strong-state application hub. An anchor-point primary sharding strategy is adopted for this type of node, that is, each strong-state application hub is pre-assigned a primary shard, and its main write permissions and core state are retained in this primary shard, thereby ensuring that strong-state requests for the same business component are completed within the same shard as much as possible.

[0087] In this embodiment, after removing the global credential hub from all nodes, the system extracts a local induced sub-hypergraph containing only the strong-state application hub and pre-allocates this local induced sub-hypergraph using an offline hypergraph partitioning algorithm. For example, the system has six highly correlated business components A, B, C, D, E, and F. A, B, and C jointly participate in a large number of business requests, D and E jointly participate in a large number of business requests, while F is relatively independent. After offline partitioning, A, B, and C can be pre-allocated to the first shard, D and E to the second shard, and F to the third shard. The fourth shard is used to absorb the load of subsequent ordinary nodes. This pre-allocation result remains fixed within the current scheduling cycle and serves as a guiding anchor point for the subsequent streaming allocation phase.

[0088] Subsequently, the system performs streaming dynamic partitioning on the remaining movable ordinary nodes. These movable ordinary nodes include ordinary user nodes, long-tail service component nodes, and other nodes not identified as hubs. The system processes each node to be assigned sequentially according to a predetermined order and calculates the comprehensive score for each node when it is assigned to each shard. The comprehensive score includes at least two parts: a positive gravity score and a negative load penalty.

[0089] The positive gravity score characterizes the strength of the business association between the node to be assigned and existing nodes within the target shard. If a strong-state application hub, pre-assigned to a target shard, already exists in the historical hyperedges participated in by the node to be assigned, a guidance reward is added to that target shard, causing the node to be assigned to prioritize following its frequently interacting core business components. The negative load penalty characterizes the congestion level when the current cumulative load of the target shard approaches the system average load; the higher the load of the target shard, the greater the penalty, to avoid too many nodes concentrating on the same shard.

[0090] For example, a regular node U3 primarily interacts with business components A and B in its historical records, and A and B have already been pre-assigned to shard 1. Simultaneously, node U3 participates a small number of requests related to business component D. When calculating U3's shard score, shard 1 receives a higher positive attraction score because it contains both A and B anchor points. If the current load on shard 1 has not yet exceeded the load balancing threshold, the system will assign U3 to shard 1. As another example, although a regular node U4 has some association with A, its load on shard 1 is significantly higher than other shards. U4 also has a strong connection with business component F in shard 3. Therefore, under the negative load penalty, U4 can be assigned to shard 3 to achieve overall load balancing.

[0091] In a preferred embodiment, the system maintains the cumulative load value of each shard in real time and updates the cumulative load value of the corresponding shard immediately after the allocation of each ordinary node, in order to support online decision-making for subsequent nodes. Thus, this invention can achieve rapid reallocation of massive numbers of ordinary nodes while maintaining linear time complexity.

[0092] This embodiment may further include a repartition result distribution and execution phase. After completing the sharding calculation for the current scheduling cycle, the system generates a mapping table from nodes to target shards and distributes the mapping table to each sharding control module. For nodes whose mappings change, the system can use incremental migration to transfer their state data, index information, or cache copies; for strongly stateful application hubs, their primary shards are switched only in adjacent scheduling cycles when migration conditions are met, to avoid additional overhead caused by frequent migrations.

[0093] Through the above implementation methods, the present invention has at least the following practical effects: First, by using hyperedges to maintain the atomic association of multiple nodes in a single business request, the proportion of cross-shard collaborative requests can be effectively reduced; Second, by using a resource consumption weighting method to measure the actual load of nodes, computational hotspots caused by dividing based solely on the number of requests can be avoided; Third, by distinguishing between the global credential hub and the strong state application hub, high-frequency public resource access can be separated from the cost of strong synchronization across shards; Fourth, through the two-stage architecture of "offline pre-allocation + online streaming follow-up," the rapid scheduling requirements of large-scale dynamic systems can be adapted.

[0094] Example 3 This embodiment provides a distributed sharding and load balancing system based on dynamic hypergraph partitioning, the system comprising: The hypergraph building module is used to obtain historical transaction records and construct a unified hypergraph model H = (V, E, ... , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; The node classification module is used to calculate the total load weight of each vertex in the vertex set V, extract the top-ranked vertices as candidate hub nodes, and classify each candidate hub node into a global credential hub or a strong state application hub according to the business logic characteristics of each candidate hub node. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. The offline pre-allocation module is used to extract a local induced sub-hypergraph containing only the strong state application hub after removing the global credential hub from the unified hypergraph model, and to pre-allocate the local induced sub-hypergraph using an offline multi-level hypergraph partitioning algorithm to determine the main partition to which each strong state application hub belongs, and to use each main partition as a guiding anchor point for the subsequent streaming partitioning stage. The streaming dynamic partitioning module is used to sequentially allocate movable ordinary nodes in the vertex set V, excluding the global credential hub and the strong state application hub, using a streaming greedy algorithm. For each node to be allocated, the module calculates its comprehensive score for allocation to each candidate partition and allocates it to the partition with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the real-time load of the partition.

[0095] Example 4 This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the distributed sharding and load balancing method based on dynamic hypergraph partitioning described in Embodiment 1 or 2. Example 5 This embodiment provides a computer device, including a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor. When the computer program is executed by the processor, it implements the steps of the distributed sharding and load balancing method based on dynamic hypergraph partitioning described in Embodiment 1 or Embodiment 2.

[0096] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A distributed sharding and load balancing method based on dynamic hypergraph partitioning, characterized in that, The method includes: Obtain historical transaction records and construct a unified hypergraph model H = (V, E, , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; The total load weight of each vertex in the vertex set V is calculated, and the top-ranked vertices with a preset proportion are extracted as candidate hub nodes. Based on the business logic characteristics of each candidate hub node, they are classified as global credential hubs or strong state application hubs. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. After removing the global credential hub from the unified hypergraph model, a local induced sub-hypergraph containing only the strong state application hub is extracted. The local induced sub-hypergraph is pre-assigned using an offline multi-level hypergraph partitioning algorithm to determine the primary partition to which each strong state application hub belongs, and each primary partition is used as a guiding anchor point for the subsequent streaming partitioning stage. For the movable ordinary nodes in the vertex set V other than the global credential hub and the strong state application hub, a streaming greedy algorithm is used to allocate them sequentially. For each node to be allocated, the comprehensive score of its allocation to each candidate shard is calculated, and it is allocated to the shard with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the shard's real-time load.

2. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 1, characterized in that, The vertex set V includes external request nodes and internal logical component nodes. Each hyperedge e in the hyperedge set E corresponds to a distributed transaction and contains all nodes that interact within that transaction. The hyperedge weight set... The weights of each hyperedge in ( The actual computing power consumption of this distributed transaction is represented by the vertex load weight set. Vertex load weights ( It is obtained by accumulating the weights of each hyperedge containing the vertex after amortization.

3. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 2, characterized in that, Vertex load weight ( The calculation method for ) is as follows: in, Let E(v) be the set of all superedges containing vertex v.

4. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 1, characterized in that, The methods for classifying candidate hub nodes into global credential hubs or strong state application hubs based on their business logic characteristics include: If a candidate hub node contains an operation in its historical transactions that matches the preset voucher operation feature code rule base, and the cumulative computing power consumption value of the operation accounts for a proportion of the node's total consumption value that is greater than a preset threshold τ, then it is determined to be a global voucher hub. Otherwise, it is determined to be a strong state application hub.

5. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 1, characterized in that, The asynchronous lock-free state management strategy is as follows: The voucher quota status of the global voucher hub is stored in the respective shards according to the request initiator. When cross-shard transfer occurs, the source shard directly deducts the voucher quota and generates a status certificate. The target shard verifies the certificate and increases the corresponding quota. There is no need to execute the cross-shard two-phase commit protocol. The anchor point master sharding strategy is as follows: Each strong-state application hub is assigned a unique primary shard, and only this primary shard is granted exclusive write permissions to its core state. Cross-shard call requests from other shards are forwarded to this primary shard for unified execution via asynchronous message routing.

6. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 1, characterized in that, The offline multi-level hypergraph partitioning algorithm aims to minimize cross-shard communication costs while satisfying load balancing constraints; wherein, the objective function for minimizing cross-shard communication costs is: Wherein, the range of summation This indicates that all hyperedges in the local induced subgraph have been traversed. For super-edge In mapping relationship The number of independent shards spanned below; The load balancing constraints are: Among them, the summation term This represents the sum of the total load weights of all strong-state application hub nodes participating in the offline partitioning.

7. The distributed sharding and load balancing method based on dynamic hypergraph partitioning according to claim 1, characterized in that, The formula for calculating the overall score is as follows: = in: The positive gravity score is calculated using the following formula: = Wherein, the range of summation This indicates traversing the entire superatlas. All nodes containing pending allocation Historical super-border, For containing nodes to be assigned Historical super-border, This represents the computing power weight of the hyperedge. This superedge has already been allocated to a partition. The number of neighboring nodes, This represents the total number of nodes contained in the superedge. For the guiding coefficient, if the superedge Includes fragments that have been pre-allocated offline. Strong state application hub anchors will grant multiplied rewards. Let it be a constant. ,otherwise Set to 1; The negative load penalty is calculated using the following formula: = in, As a penalty regulator, This represents the accumulated computing load of shard k under the current allocation progress. The system's average expected load is the total load of all nodes in the network. Divide by the number of fragments, K.

8. A distributed sharding and load balancing system based on dynamic hypergraph partitioning, characterized in that, The system includes: The hypergraph building module is used to obtain historical transaction records and construct a unified hypergraph model H = (V, E, ... , ), where V is the vertex set and E is the hyperedge set. For the set of vertex load weights, For the set of superedge weights; The node classification module is used to calculate the total load weight of each vertex in the vertex set V, extract the top-ranked vertices as candidate hub nodes, and classify each candidate hub node into a global credential hub or a strong state application hub according to the business logic characteristics of each candidate hub node. The global credential hub adopts an asynchronous lock-free state management strategy, and the strong state application hub adopts an anchor point master sharding strategy. The offline pre-allocation module is used to extract a local induced sub-hypergraph containing only the strong state application hub after removing the global credential hub from the unified hypergraph model, and to pre-allocate the local induced sub-hypergraph using an offline multi-level hypergraph partitioning algorithm to determine the main partition to which each strong state application hub belongs, and to use each main partition as a guiding anchor point for the subsequent streaming partitioning stage. The streaming dynamic partitioning module is used to sequentially allocate movable ordinary nodes in the vertex set V, excluding the global credential hub and the strong state application hub, using a streaming greedy algorithm. For each node to be allocated, the module calculates its comprehensive score for allocation to each candidate shard and allocates it to the shard with the highest comprehensive score. The comprehensive score includes a positive gravity score based on the historical hyperedge association strength and a negative load penalty term based on the real-time load of the shard.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the distributed sharding and load balancing method based on dynamic hypergraph partitioning as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The method includes a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the distributed sharding and load balancing method based on dynamic hypergraph partitioning as described in any one of claims 1 to 7.

Images