Fragmented block chain account distribution method and system

Abstract

The invention provides a fragmented block chain account allocation method and system, and the method comprises the steps: constructing a block chain environment according with a Markov decision process, and enabling the block chain environment to serve as an interaction environment of deep reinforcement learning; designing a multi-agent near-end strategy optimization Shard-MAPPO architecture in the block chain fragmentation system according to the initially set number of the fragments; downloading block chain transaction data from a network, extracting information to construct the block chain transaction data, designing an algorithm to construct an account relation graph, and segmenting the relation graph; a multi-agent model is trained under the Shard-MAPPO architecture, and the model is stored after being converged; and the model is applied, and balanced distribution of transaction accounts is realized through competition and cooperation among multiple agents in the model. The technical problems that load balancing is difficult to achieve, the cross-piece transaction proportion is too high, and the action requirement of a complex dynamic system cannot be met are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of sharded blockchain transaction account allocation based on multi-agent deep reinforcement learning, and in particular to a sharded blockchain account allocation method and system. Background Art

[0002] Blockchain is a chain structure composed of a series of data blocks. Each data block contains a batch of transaction data. The security and integrity of the data are guaranteed by cryptographic technology, and the decentralization and immutability of the data are guaranteed by consensus protocols. Therefore, the characteristics of blockchain include decentralization, immutability, transparency, openness, security and reliability. The decentralized nature of blockchain makes data storage and management no longer rely on a single institution or individual, but is jointly maintained and verified by multiple nodes in the network, thereby improving the stability and reliability of the system. This feature has made the application of blockchain in the financial field attract much attention; the immutability of blockchain means that once the data is recorded on the blockchain, it cannot be tampered with or deleted. Each data block contains the hash value of the previous data block, and the integrity of the data chain ensures that the data cannot be tampered with. This feature has a wide range of applications in the fields of data security and tamper-proofing; the openness and transparency of blockchain means that all transaction records can be publicly viewed, and anyone can participate in the blockchain network as a node to view transaction records and verify the legitimacy of transactions. This feature gives blockchain a huge advantage and prospect in improving transparency and trust. With the rapid development of the digital economy and people's increasing attention to data security and privacy protection, blockchain technology has been widely used and promoted in the fields of finance, supply chain, Internet of Things, copyright protection, etc. With the increase in applications, the system's transaction volume per unit time (TPS) gradually restricts the application of blockchain systems. In order to break through the bottleneck of blockchain system scalability, it is urgent to study related technologies and provide blockchain systems that meet business needs for related industries. The focus of this patent research is the scalability technology of blockchain systems.

[0003] Sharding is a blockchain expansion technology that divides the entire blockchain network into multiple areas. Each area is an independent area with a certain number of network nodes and transaction data. The shards process transactions independently and in parallel, and communicate and trade with other shards at the same time. This method effectively reduces the burden of the entire blockchain network, and parallel processing of transactions improves the overall throughput performance of the blockchain system. In a transfer transaction, if the payment account and the payment account are in the same shard, it is called an "intra-shard transaction", and if they are not in the same shard, it is called a "cross-shard transaction". When allocating transactions, the existing system will allocate massive transactions to one shard, resulting in a large transaction load in a certain shard, but the load of other shards is idle, resulting in excessive load in individual shards, causing transaction congestion in the shard, increasing the delay in processing transactions, and then reducing throughput, lowering the application performance of the entire system. Therefore, load balancing of transactions between shards is an important part of breaking through the scalability of blockchain. By balancing the load between shards, each shard can be guaranteed to process transactions efficiently and without congestion, thereby improving the throughput of the blockchain system.

[0004] In response to this problem, there have been some attempts to solve this problem and have achieved certain results. For example, the existing invention patent application document "Blockchain Sharding Load Balancing Method and Device" with publication number CN113157457A, the existing method includes: determining the load status of each shard according to the number of unprocessed transactions in each shard in the blockchain after executing transaction verification; determining the node rating of each node according to its own hardware performance, transaction processing time and transaction verification results of each node in each shard after executing the verification transaction; performing shard node reallocation operations according to the load status of each shard and the node rating of each node in each shard, and performing transaction verification operations on the unprocessed transactions in the shard after the shard node reallocation operations are completed, and packaging the corresponding generated micro-blocks on the chain after the transaction verification operations are completed. However, the aforementioned existing solutions are for non-DRL-based solutions, which require the construction of accurate environmental models, which is difficult for complex dynamic systems; in addition, in sequential decision problems, they will incur high computational overhead to make optimal decisions, which is not timely for real-time decision making systems. There are also related works for DRL-based solutions, but they all have certain limitations. For example, Skychain is a centralized execution solution, and the system has a single agent failure, which in turn causes the overall system to crash; Deep-RL's action space has only one action, which cannot meet complex action requirements.

[0005] In summary, the existing technology has technical problems such as load balancing is difficult to achieve, the proportion of cross-shard transactions is too high, and it cannot meet the action requirements of complex dynamic systems. Summary of the invention

[0006] The technical problem to be solved by the present invention is: how to solve the technical problems existing in the prior art that load balancing is difficult to achieve, the proportion of cross-slice transactions is too high, and the action requirements of complex dynamic systems cannot be met.

[0007] The present invention adopts the following technical solutions to solve the above technical problems: A sharded blockchain account allocation method includes:

[0008] S1. Based on Markov decision process, build a blockchain environment as a deep reinforcement learning interactive environment;

[0009] S2. In the deep reinforcement learning interactive environment, obtain and build the multi-agent proximal strategy optimization architecture Shard-MAPPO for the blockchain sharding system based on the initial set number of shards;

[0010] S3, downloading transaction information from the network, constructing blockchain transaction data, constructing an account relationship graph based on the blockchain transaction data, and segmenting and processing the account relationship graph;

[0011] S4. Under the multi-agent proximal strategy optimization framework Shard-MAPPO, train the multi-agent model until the multi-agent model converges, and obtain and save the applicable multi-agent model;

[0012] S5. In the applicable multi-agent model, multi-agent competition operations and multi-agent cooperation operations are performed to evenly distribute transaction accounts in the account relationship diagram.

[0013] The present invention proposes a multi-agent reinforcement learning scheme through centralized training and distributed execution to achieve balanced account allocation in the sharding system. The graph segmentation algorithm maintains the transaction relationship between accounts while avoiding the problem of excessive cross-shard transactions. The account allocation strategy designed by the present invention based on multi-agent deep reinforcement learning enables cooperation and competition between agents, and evenly distributes transaction subgraphs to different shards to achieve the purpose of load balancing between shards. In addition, the transaction graph segmentation algorithm retains the transaction connection between accounts while reducing the cross-shard transaction ratio.

[0014] In a more specific technical solution, S1 includes:

[0015] S11. In a similar blockchain environment, set the reset state as the reset environment for each new epoch;

[0016] S12, within the epoch, select and execute the appropriate action action according to the current state;

[0017] S13, switch the current state state to the next state next-state, and give reward reward;

[0018] S14. Collect tuples as training data, and use them to train and update the preset network model until the preset network model converges to obtain a deep reinforcement learning interactive environment.

[0019] In a specific technical solution, in S14, the tuple includes: current state state, action, reward reward and next state next-state.

[0020] In the specific technical solution, in S1, the state space in the deep reinforcement learning interactive environment includes the number of subgraphs to be allocated, the number of accounts to be allocated, the total system load, and the standard deviation between slices.

[0021] In the specific technical solution, S2 includes:

[0022] S21. According to the number of initially set zones, an zone control scheduling module is added for the preset zones;

[0023] In S22, in the deep reinforcement learning interactive environment, each preset area corresponds to an intelligent agent, and the actor-critic module Actor-Critic is designed to construct the multi-agent proximal strategy optimization architecture Shard-MAPPO.

[0024] Specifically, lines 1-3 in the following code initialize the actor and critic network for the agent in each patch, then deploy a random process for action exploration, initialize the global state and obtain the input subgraph; next, the max-episode-length iterative process is executed, and each agent selects an action according to the current policy; lines 4-6 indicate that after assigning a new action to each agent, the performance is tested using the set reward; at the same time, the current subgraph state, the current moment action, the system performance and the corresponding reward are calculated and collected, and it is determined whether to terminate the round, and then stored in the replay buffer in the form of a tuple; lines 7-9 indicate that the agent selects the best action a according to the current state s until an iteration is completed, and then updates the state s' to the initial state s; lines 10-18 indicate that after executing a whole iteration, each agent will perform the following steps to update the actor and critic network.

[0025]

[0026] This invention adopts the centralized training and distributed execution algorithm for the first time to realize the account subgraph division. All agents are centrally trained in the Critic network and distributedly executed in the Actor network, avoiding the failure of the entire system caused by the downtime of a single agent Actor network, making the model more robust.

[0027] The present invention adopts a multi-agent deep reinforcement learning scheme to solve the account allocation problem. Its basic idea is to construct the accounts and transactions in the transaction pool into a global graph, divide the transaction graph through the Linear-Deterministic-Greedy (LDG) graph segmentation algorithm, and then allocate accounts through Shard-MAPPO agents. Shard-MAPPO uses the Actor-Critc algorithm to learn states and rewards and provide optimal actions. Through the relationship of cooperation and competition between agents, accounts are allocated to the optimal areas to achieve load balancing between the blockchain system areas.

[0028] In the specific technical solution, S3 includes:

[0029] S31. Download transaction information according to a preset format;

[0030] S32. Extract the sending account from, the receiving account to, and the transaction unique ID from the transaction information to obtain the account pair (from, to);

[0031] S33, counting the number of transactions performed by each account pair (from, to), and saving the data as account relationship data in a preset format;

[0032] S34. construct an account relationship diagram based on the account relationship data;

[0033] S35. Design a Linear Deterministic Greedy Partition algorithm to segment the account relationship graph and retain the transaction connection between accounts.

[0034] Specifically, in the aforementioned LDGP algorithm, the input G is the transaction graph consisting of all accounts, N represents the number of subgraphs that need to be divided, and the output P is each subgraph, which is composed of some accounts.

[0035] In the algorithm execution code below, lines 5-9 indicate the calculation of the degree of each graph node, lines 12-13 indicate initializing the minimum degree of the current node and the corresponding partition index; line 14 indicates traversing all neighbor nodes of the current node; lines 15-17 indicate that if a neighbor node has been assigned a partition index, then the partition index of the neighbor node is obtained and the degree of the partition is calculated; lines 18-20 indicate updating the minimum degree and the corresponding partition index; lines 24-28 indicate that if a partition index that meets the conditions is found, the current node is assigned to the partition, otherwise, the current node is assigned to a new partition.

[0036]

[0037] In the sharded blockchain environment, the existing method allocates individual accounts to corresponding shards in isolation, ignoring the transaction relationship between accounts, which will cause the problem of excessive cross-shard transaction ratio. In the construction of the sharded blockchain environment, the present invention innovatively constructs a global account transaction graph, and designs a Linear Deterministic Greedy Partition algorithm to split the account relationship graph, retaining the transaction connection between accounts to the greatest extent, and reducing the cross-shard transaction ratio after allocating the account subgraph.

[0038] In a specific technical solution, in S34, the sending account from and the receiving account to are set as nodes of the account relationship graph, and the transaction quantity between the sending account from and the receiving account to is set as an edge of the account relationship graph.

[0039] In the specific technical solution, S4 includes:

[0040] S41. Assume that the blockchain sharding system has N shards, corresponding to N intelligent agents. For each intelligent agent, the initial intelligent agent strategy and intelligent agent value function are provided;

[0041] S42, performing model training on the multi-agent model according to a preset number of training times. In each model training, all agents interact in a deep reinforcement learning interactive environment and obtain their respective state data;

[0042] S43. For each agent, based on the current agent value function, calculate the advantage function estimate, maximize the agent's PPO cutoff target, and obtain the update strategy;

[0043] S44. Use the mean square error loss function to optimize the agent value function.

[0044] The present invention adopts a multi-agent architecture to solve the account allocation problem of sharded blockchain. Each agent corresponds to a shard. The agents process the allocated account subgraphs through competition and cooperation, thereby achieving load balancing between sharded blockchain systems and improving the overall performance of the blockchain system.

[0045] In the specific technical solution, S5 includes:

[0046] S51, applying the multi-agent model to the account relationship diagram to be allocated;

[0047] S52, the multi-agent system performs inter-agent competition operation and multi-agent cooperation operation according to the state change data;

[0048] S53. Obtain an account subgraph from the account relationship graph;

[0049] S54, assigning different account sub-graphs to different areas;

[0050] S55, making the areas with transaction accounts greater than or equal to a preset threshold compete for the account subgraph to be allocated;

[0051] S56: Make the areas with transaction accounts less than a preset threshold to collaborate and compete for the account subgraphs to be allocated.

[0052] In a specific technical solution, a sharded blockchain account allocation system includes:

[0053] An interactive environment building module, which is used to build a blockchain environment based on the Markov decision process as a deep reinforcement learning interactive environment;

[0054] The strategy optimization architecture building module is used to obtain and build the multi-agent proximal strategy optimization architecture Shard-MAPPO for the blockchain sharding system according to the initial set number of shards in the deep reinforcement learning interactive environment. The strategy optimization architecture building module is connected with the interactive environment building module;

[0055] The account relationship diagram construction and segmentation module is used to download transaction information from the network, construct blockchain transaction data, construct an account relationship diagram based on the blockchain transaction data, and segment and process the account relationship diagram;

[0056] The model training module is used to train the multi-agent model under the multi-agent proximal policy optimization architecture Shard-MAPPO until the multi-agent model converges, obtain and save the applicable multi-agent model, and the model training module is connected with the account relationship graph construction segmentation module and the policy optimization architecture construction module;

[0057] The trading account allocation module is used to perform multi-agent competition operations and multi-agent cooperation operations in an applicable multi-agent model, so as to evenly allocate trading accounts in the account relationship diagram. The trading account allocation module is connected to the model training module.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] The present invention proposes a multi-agent reinforcement learning scheme through centralized training and distributed execution to achieve balanced account allocation in the sharding system. The graph segmentation algorithm maintains the transaction relationship between accounts while avoiding the problem of excessive cross-shard transactions. The account allocation strategy designed by the present invention based on multi-agent deep reinforcement learning enables cooperation and competition between agents, and evenly distributes transaction subgraphs to different shards to achieve the purpose of load balancing between shards. In addition, the transaction graph segmentation algorithm retains the transaction connection between accounts while reducing the cross-shard transaction ratio.

[0060] This invention adopts the centralized training and distributed execution algorithm for the first time to realize the account subgraph division. All agents are centrally trained in the Critic network and distributedly executed in the Actor network, avoiding the failure of the entire system caused by the downtime of a single agent Actor network, making the model more robust.

[0061] The present invention adopts a multi-agent deep reinforcement learning scheme to solve the account allocation problem. Its basic idea is to construct the accounts and transactions in the transaction pool into a global graph, divide the transaction graph through the Linear-Deterministic-Greedy (LDG) graph segmentation algorithm, and then allocate accounts through Shard-MAPPO agents. Shard-MAPPO uses the Actor-Critc algorithm to learn states and rewards and provide optimal actions. Through the relationship of cooperation and competition between agents, accounts are allocated to the optimal areas to achieve load balancing between the blockchain system areas.

[0062] In the sharded blockchain environment, the existing method allocates individual accounts to corresponding shards in isolation, ignoring the transaction relationship between accounts, which will cause the problem of excessive cross-shard transaction ratio. In the construction of the sharded blockchain environment, the present invention innovatively constructs a global account transaction graph, and designs a Linear Deterministic Greedy Partition algorithm to split the account relationship graph, retaining the transaction connection between accounts to the greatest extent, and reducing the cross-shard transaction ratio after allocating the account subgraph.

[0063] The present invention adopts a multi-agent architecture to solve the account allocation problem of sharded blockchain. Each agent corresponds to a shard. The agents process the allocated account subgraphs through competition and cooperation, thereby achieving load balancing between sharded blockchain systems and improving the overall performance of the blockchain system.

[0064] In terms of aggregating accounts, the present invention constructs accounts with transaction relationships into a graph structure, and uses a graph partitioning algorithm to divide the complete graph structure into subgraphs. Each subgraph contains some accounts with the most close transactions. The advantage of this is that cross-shard transactions can be minimized.

[0065] At the same time, in terms of allocation algorithm, this patent uses a deep reinforcement learning algorithm to control the allocation of subgraphs to different areas. Compared with the existing random division or dynamic methods, the use of reinforcement learning algorithms to dynamically allocate accounts can adapt to complex and dynamically changing area states;

[0066] The present invention designs a multi-agent reinforcement learning algorithm, which further improves on the single-agent algorithm and introduces the idea of ​​swarm intelligence, which can effectively alleviate the system downtime problem caused by the failure of a single agent.

[0067] The present invention solves the technical problems existing in the prior art that load balancing is difficult to achieve, the cross-chip transaction ratio is too high, and the action requirements of complex dynamic systems cannot be met. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a schematic diagram of the basic steps of a sharded blockchain account allocation method according to Embodiment 1 of the present invention;

[0069] Figure 2 This is a schematic diagram of a deep reinforcement learning system model according to Embodiment 1 of the present invention;

[0070] Figure 3 This is a schematic diagram of the Shard-Actor Critic architecture of Example 1 of the present invention;

[0071] Figure 4 This is a schematic diagram of specific implementation steps of a sharded blockchain account allocation method according to Embodiment 2 of the present invention;

[0072] Figure 5 Schematic diagram of the graph construction and graph segmentation principles of Example 2 of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Example 1

[0075] like Figure 1 As shown, a sharded blockchain account allocation method provided by the present invention includes the following basic steps:

[0076] Step S1: construct a blockchain environment that conforms to the Markov decision process as an interactive environment for deep reinforcement learning;

[0077] In this embodiment, the account allocation strategy based on multi-agent deep reinforcement learning adopts a multi-dimensional action space, which is more suitable for complex environments. The environment in which the multi-agents are located is highly dynamic. When the state of each agent changes, the environment changes dynamically relative to other agents.

[0078] like Figure 2 As shown, in this embodiment, in the scope of deep reinforcement learning, the agent needs to interact with the environment, and the constructed environment must comply with the environment of the Markov decision process. Specifically, the next state of the system depends only on the current state and has nothing to do with the past state. The environment sets the reset state as the reset environment for each new epoch, selects the appropriate action (action) according to the current state (state) in the epoch, switches to the next state (next-state) after executing the action and gives a reward (reward). Collect tuples as training data and update the network until the model converges. In this embodiment, the aforementioned tuple content includes but is not limited to: state, action, reward, next state, whether to end. Specifically, the state space includes four states, the number of subgraphs to be allocated, the number of accounts to be allocated, the total system load, and the standard deviation between slices. Each time the selected optimal action is executed, the system is updated to the next state, the reward value is set to TPS, and the end mark is a fixed number of training times. Collect the above five tuples and train until the model converges.

[0079] Step S2: Design the Shard-MAPPO architecture in the blockchain sharding system according to the number of shards initially set;

[0080] like Figure 3As shown, in this embodiment, according to the initially set number of shards, a shard control scheduling module is added, each shard corresponds to an agent in the blockchain environment, and an actor-critic module is designed. The designed Shard-MAPPO is used as the multi-agent proximal policy optimization (MAPPO) architecture of this patent.

[0081] Step S3: download blockchain transaction data from the network, extract information to construct blockchain transaction data, design an algorithm to construct an account relationship diagram, and segment the relationship diagram;

[0082] like Figure 4 As shown, in this embodiment, the real transaction data of Ethereum in CSV format is downloaded from the network, and the sending account (from), receiving account (to), and transaction unique ID (hash) are extracted from it, and the number of transactions performed by each account pair (from, to) is counted and saved in a new CSV format. An account relationship graph is constructed based on the above CSV, and the nodes of the relationship graph are the sending account or the receiving account, and the edges are the number of transactions between the two accounts. The Linear Deterministic Greedy Partition algorithm is designed to split the account relationship graph, retain the transaction connection between accounts to the greatest extent, and reduce the number of cross-shard transactions after allocating the account subgraph.

[0083] Step S4: Train the multi-agent model under the Shard-MAPPO architecture, and save the model after it converges;

[0084] In this embodiment, the blockchain sharding system is set to have N shards, corresponding to N agents, and each agent is initialized with its own strategy and value function; the algorithm is trained K times in total, and in each training, all agents interact in the environment and obtain their own state data respectively. For each agent, the advantage function is estimated based on the current value function, and its strategy is updated by maximizing its PPO truncated target, and finally the mean square error loss function is used to optimize its value function. Training is performed until convergence, and the model and parameters are saved.

[0085] Step S5: Apply the model to achieve balanced distribution of transaction accounts through competition and cooperation among multiple agents in the model.

[0086] In this embodiment, the trained model is called and applied to the account graph to be allocated. The multi-agent system competes and cooperates according to the change of state, and allocates different account subgraphs to different areas. Areas with more accounts compete for the account subgraphs to be allocated, and areas with fewer accounts cooperate to jointly strive for account subgraphs, thereby achieving the purpose of balancing the transaction account load and reducing the cross-slice transaction ratio.

[0087] Example 2

[0088] like Figure 4 As shown, in this embodiment, a sharded blockchain account allocation method also includes the following specific implementation steps:

[0089] Step S101: In constructing a blockchain environment that conforms to the Markov decision process, design a method that conforms to the blockchain logic;

[0090] In this embodiment, the reset method is used to obtain the number of initialized subgraphs, the number of accounts, the system state value when resetting the environment, the Boolean values ​​of the first subgraph division and the allocated subgraph; the step method designs the Execute_Subgraph_Distribution_based_on_Action method according to the selected action to perform account graph division, and the method Send_Subgraph_of_Account_to_Shard to send the account subgraph to different shards, calculates the number of accounts and load received by the corresponding shard as the next stage state (next state), and calculates the throughput as a reward according to the formula based on the load transaction processed this time; the Check_Done method determines whether to terminate the epoch iteration of this round by monitoring the number of account subgraphs in the environment; in order to facilitate the description of the shard status, the shard Shard class is designed, and the attributes shard index and shard load are defined, the method Add_Account_Addresses to add accounts is defined, and the method Get_Transactions_Amount_from_Shard to obtain the load from the shard is defined.

[0091] Step S102: design the corresponding number of agents according to the number of areas initially set;

[0092] In this embodiment, the Shard-MAPPO architecture sets an agent for each agent, and each agent is composed of an Actor-Critic network. When training the Critic, all agent states and action information are used as input. In order to avoid the dimensional disaster caused by the increase of input dimensions, the Shard-MAPPO architecture is optimized to construct the MAPPO architecture, and a scheduler module is added to the original MAPPO architecture to further abstract and extract the state observation of all agents, and calculate the average value of each observed shard state, and then abstractly combine these arrays into a new array, so that the length of each abstract array is constant.

[0093] Step S103, download the real Ethereum data set from the network, extract the transaction initiator: from, the transaction recipient: to, and the transaction hash: hash field, and count the number of transactions between the same transaction pair: from and to, and save it in csv format as data input for the sharded blockchain environment;

[0094] Step S104: According to the aforementioned account transaction graph, design the algorithm Build_Account_Graph to construct the account global graph, and design the algorithm Linear Deterministic Greedy Partition to partition the account global graph;

[0095] like Figure 5 As shown, in this embodiment, the algorithm divides accounts with associated edges into the same area. Figure 5 , nodes represent the sending and receiving accounts of the transaction parties, and edges represent the transaction quantities. Different colors represent different subgraphs.

[0096] Step S105: train the Shard-MAPPO model, set K rounds of training, and in each round, put the five-tuple of <state, action, reward, next state, whether to end training> into the experience pool and randomly sample it as a training set;

[0097] Step S106: After each round of training, the Critic network is updated, all agents learn collectively, the Actor network is updated, actions are distributed according to the learned strategy, and the rewards of each round are recorded. After the model converges, the model and parameters are saved;

[0098] Step S107, loading the model and parameters. There is a relationship of competition and cooperation between the trained model agents. The areas with large load differences compete with each other for account subgraphs, and the areas with small differences cooperate with each other to obtain account subgraphs, ultimately achieving a balanced distribution of transaction loads between sharded blockchains.

[0099] The present invention is designed based on the Multi-Agent Proximal Policy Optimization (MAPPO) deep reinforcement learning algorithm. The actor-critic module is modified in the original algorithm, and a blockchain sharding module is added. At the same time, a blockchain sharding environment that conforms to the Markov decision process (MDP) is designed. In this environment, balanced distribution of transaction accounts is achieved through competition and collaboration among multiple agents.

[0100] This paper implements an account allocation strategy based on multi-agent deep reinforcement learning. The main innovation is that a new multi-agent architecture is proposed to solve the account allocation problem of sharded blockchain, a centralized training and distributed execution algorithm is designed to realize account subgraph partitioning, and a sharded blockchain environment that conforms to the Markov decision process is constructed, which facilitates the subsequent use of deep reinforcement learning methods to solve sharded blockchain related problems.

[0101] The present invention proposes a multi-agent reinforcement learning scheme through centralized training and distributed execution to achieve balanced account allocation in the sharding system. The graph segmentation algorithm maintains the transaction relationship between accounts while avoiding the problem of excessive cross-shard transactions. The account allocation strategy designed by the present invention based on multi-agent deep reinforcement learning enables cooperation and competition between agents, and evenly distributes transaction subgraphs to different shards to achieve the purpose of load balancing between shards. In addition, the transaction graph segmentation algorithm retains the transaction connection between accounts while reducing the cross-shard transaction ratio.

[0102] This invention adopts the centralized training and distributed execution algorithm for the first time to realize the account subgraph division. All agents are centrally trained in the Critic network and distributedly executed in the Actor network, avoiding the failure of the entire system caused by the downtime of a single agent Actor network, making the model more robust.

[0103] The present invention adopts a multi-agent deep reinforcement learning scheme to solve the account allocation problem. Its basic idea is to construct the accounts and transactions in the transaction pool into a global graph, divide the transaction graph through the Linear-Deterministic-Greedy (LDG) graph segmentation algorithm, and then allocate accounts through Shard-MAPPO agents. Shard-MAPPO uses the Actor-Critc algorithm to learn states and rewards and provide optimal actions. Through the relationship of cooperation and competition between agents, accounts are allocated to the optimal areas to achieve load balancing between the blockchain system areas.

[0104] In the sharded blockchain environment, the existing method allocates individual accounts to corresponding shards in isolation, ignoring the transaction relationship between accounts, which will cause the problem of excessive cross-shard transaction ratio. In the construction of the sharded blockchain environment, the present invention innovatively constructs a global account transaction graph, and designs a Linear Deterministic Greedy Partition algorithm to split the account relationship graph, retaining the transaction connection between accounts to the greatest extent, and reducing the cross-shard transaction ratio after allocating the account subgraph.

[0105] The present invention adopts a multi-agent architecture to solve the account allocation problem of sharded blockchain. Each agent corresponds to a shard. The agents process the allocated account subgraphs through competition and cooperation, thereby achieving load balancing between sharded blockchain systems and improving the overall performance of the blockchain system.

[0106] The present invention solves the technical problems existing in the prior art that load balancing is difficult to achieve, the cross-chip transaction ratio is too high, and the action requirements of complex dynamic systems cannot be met.

[0107] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sharded blockchain account allocation method, characterized in that: The method comprises: S1. Based on Markov decision process, build a blockchain environment as a deep reinforcement learning interactive environment; S2. In the deep reinforcement learning interactive environment, obtain and build a multi-agent proximal strategy optimization architecture Shard-MAPPO for the blockchain sharding system according to the initial set number of shards; S3. Download transaction information from the network, construct blockchain transaction data, construct an account relationship graph based on the blockchain transaction data, and segment and process the account relationship graph; S4, training the multi-agent model under the multi-agent proximal policy optimization architecture Shard-MAPPO until the multi-agent model converges, and obtaining and saving an applicable multi-agent model; S5. In the applicable multi-agent model, multi-agent competition operations and multi-agent cooperation operations are performed to evenly distribute transaction accounts in the account relationship diagram.

2. A sharded blockchain account allocation method according to claim 1, characterized in that: The S1 includes: S11. In a blockchain environment similar to the above, set the reset state as the reset environment for each new epoch; S12, within the epoch, select and execute an appropriate action action according to the current state state; S13, switching the current state state to the next state next-state, and giving a reward reward; S14. Collect tuples as training data, and use them to train and update a preset network model until the preset network model converges, thereby obtaining the deep reinforcement learning interactive environment.

3. A sharded blockchain account allocation method according to claim 2, characterized in that: In S14, the tuple includes: the current state state, the action action, the reward reward and the next state next-state.

4. A sharded blockchain account allocation method according to claim 1, characterized in that: In S1, the state space in the deep reinforcement learning interactive environment includes the number of subgraphs to be allocated, the number of accounts to be allocated, the total system load, and the standard deviation between slices.

5. A sharded blockchain account allocation method according to claim 1, characterized in that: The S2 includes: S21. According to the initially set number of zones, an zone control scheduling module is added for the preset zones; In S22, in the deep reinforcement learning interactive environment, each of the preset areas corresponds to an intelligent agent, and an actor-critic module Actor-Critic is designed to construct the multi-agent proximal policy optimization architecture Shard-MAPPO.

6. A sharded blockchain account allocation method according to claim 1, characterized in that: The S3 includes: S31, downloading the transaction information according to a preset format; S32. Extract the sending account from, the receiving account to, and the transaction unique ID from the transaction information to obtain an account pair (from, to); S33, counting the number of transactions performed by each account pair (from, to), and saving the data as account relationship data in the preset format; S34, constructing the account relationship graph according to the account relationship data; S35. Design a Linear Deterministic Greedy Partition algorithm to segment the account relationship graph and retain transaction connections between accounts.

7. A sharded blockchain account allocation method according to claim 6, characterized in that: In S34, the sending account from and the receiving account to are set as nodes of the account relationship graph, and the transaction quantity between the sending account from and the receiving account to is set as an edge of the account relationship graph.

8. A sharded blockchain account allocation method according to claim 1, characterized in that: The S4 includes: S41. Assume that the blockchain sharding system has N shards, corresponding to N intelligent agents, and for each of the intelligent agents, an initial intelligent agent strategy and an intelligent agent value function; S42, performing model training on the multi-agent model according to a preset number of training times, wherein in each model training, all agents interact in the deep reinforcement learning interactive environment and obtain their respective state data; S43, for each of the intelligent agents, based on the current value function of the intelligent agent, calculate the advantage function estimate, maximize the PPO cutoff target of the intelligent agent, and obtain an update strategy; S44. Optimize the agent value function using the mean square error loss function.

9. A sharded blockchain account allocation method according to claim 1, characterized in that: The S5 includes: S51, applying the applicable multi-agent model to the account relationship diagram to be allocated; S52, the multi-agent system performs the multi-agent competition operation and the multi-agent cooperation operation according to the state change data; S53, obtaining an account subgraph from the account relationship graph; S54, allocating different account subgraphs to different areas; S55, making the areas where the transaction accounts are greater than or equal to a preset threshold compete for the account subgraph to be allocated; S56: Make the area where the transaction account is smaller than the preset threshold compete collaboratively for the account subgraph to be allocated.

10. A sharded blockchain account allocation system, characterized in that: The system comprises: An interactive environment building module, which is used to build a blockchain environment based on the Markov decision process as a deep reinforcement learning interactive environment; A strategy optimization architecture building module is used to obtain and build a multi-agent proximal strategy optimization architecture Shard-MAPPO for the blockchain sharding system according to the initial set number of shards in the deep reinforcement learning interactive environment. The strategy optimization architecture building module is connected to the interactive environment building module; An account relationship graph construction and segmentation module is used to download transaction information from the network based on the account relationship, construct blockchain transaction data, construct an account relationship graph based on the blockchain transaction data, and segment and process the account relationship graph; A model training module, used to train a multi-agent model under the multi-agent proximal policy optimization framework Shard-MAPPO until the multi-agent model converges, obtain and save an applicable multi-agent model, and the model training module is connected with the account relationship graph construction segmentation module and the policy optimization framework construction module; The transaction account allocation module is used to perform multi-agent competition operations and multi-agent cooperation operations in the applicable multi-agent model, so as to evenly allocate transaction accounts in the account relationship diagram. The transaction account allocation module is connected to the model training module.

Images