Block chain fragmentation strategy optimization method based on model reinforcement learning

Abstract

The invention discloses a block chain fragmentation strategy optimization method based on model reinforcement learning, and relates to the technical field of block chain fragmentation. Obtaining current state data of the block chain, and inputting the current state data into the initial policy network to obtain behavior data of the block chain; performing state transition on the block chain according to the behavior data of the block chain to obtain state data of the block chain at the next moment; then obtaining initial state data of the prediction model, and inputting the initial state data into a cross entropy algorithm to obtain an optimal behavior trajectory of the block chain; obtaining an initial state behavior pair from the optimal behavior track, and training an initial strategy network through the initial state behavior pair to obtain an optimal strategy network; the optimal strategy network is used for generating an optimal fragmentation strategy of the to-be-fragmented block chain. According to the method, the sampling efficiency during learning of the optimal fragmentation strategy of the block chain can be improved, so that the throughput of the block chain is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of blockchain sharding, and in particular to a blockchain sharding strategy optimization method based on model reinforcement learning. Background Art

[0002] As a distributed ledger maintained by multiple participants, blockchain is tamper-proof and cannot be forged. Globally, blockchain technology has attracted widespread attention from academia and industry. However, when applied to large-scale systems, the main challenge facing blockchain is its scalability.

[0003] In the prior art, blockchain networks are usually divided into multiple parallel processing shards through blockchain sharding technology (such as Zilliqa) to improve the throughput of the blockchain. For example, Model-Free Reinforcement Learning (MFRL) is used to solve the Optimal Blockchain Sharding (OBCS) problem to achieve dynamic sharding.

[0004] However, in the existing blockchain sharding methods, MFRL usually requires a lot of interaction with the blockchain system, which to a certain extent limits the improvement of sampling efficiency and results in low blockchain throughput. Summary of the invention

[0005] Based on this, it is necessary to provide a blockchain sharding strategy optimization method based on model reinforcement learning to address the above technical problems, which improves the sampling efficiency when learning the optimal sharding strategy of the blockchain, thereby improving the throughput of the blockchain.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a blockchain sharding strategy optimization method based on model reinforcement learning, comprising:

[0008] Obtain the current state data of the blockchain and input the current state data into the initial policy network to obtain the behavior data of the blockchain; the behavior data of the blockchain is used to guide the blockchain sharding;

[0009] Transfer the state of the blockchain according to the behavior data of the blockchain to obtain the state data of the blockchain at the next moment;

[0010] Obtaining the initial state data of the prediction model, and inputting the initial state data into the cross entropy algorithm, calling the prediction model through the cross entropy algorithm to predict the state data at the future moment, and calling the model strategy network to predict the behavior data corresponding to the state data, to obtain the optimal behavior trajectory of the blockchain; the optimal behavior trajectory includes the state data and behavior data of the blockchain at multiple moments;

[0011] The initial state behavior pair is obtained from the optimal behavior trajectory, and the initial policy network is trained through the initial state behavior pair to obtain the optimal policy network; the optimal policy network is used to generate the optimal sharding strategy for the blockchain to be sharded.

[0012] Preferably, the current state data is input into the initial policy network to obtain the behavior data of the blockchain, including:

[0013] Input the current state data into the initial strategy network to obtain the probability value of each discrete behavior of the blockchain; discrete behaviors include the block size of the blockchain, the number of shards, and the time required to generate a block;

[0014] According to the probability value of each discrete behavior, the behavior data of the blockchain is determined.

[0015] Preferably, the cross entropy algorithm is used to call the prediction model to predict the state data at the future moment, and the model strategy network is called to predict the behavior data corresponding to the state data, so as to obtain the optimal behavior trajectory of the blockchain, including:

[0016] The prediction model is called by the cross entropy algorithm to predict the state data at the future moment, and the model strategy network is called to predict the behavior data corresponding to the state data, so as to obtain multiple different behavior trajectories; each behavior trajectory includes the state data, behavior data and corresponding reward value of the blockchain at multiple moments;

[0017] Determine the total reward value of each behavior trajectory based on multiple reward values ​​in each behavior trajectory;

[0018] A preset number of behavior trajectories with the highest total return value are determined as the optimal behavior trajectory of the blockchain.

[0019] Preferably, the cross entropy algorithm is used to call the prediction model to predict the state data at the future moment, and the model strategy network is called to predict the behavior data corresponding to the state data, so as to obtain multiple different behavior trajectories, including:

[0020] For any behavior trajectory, the initial state data is input into the model strategy network to obtain the behavior data corresponding to the initial state data, and the reward value of the behavior data is calculated;

[0021] Input the initial state data into the prediction model to obtain the state data at the next moment;

[0022] According to the state data at the next moment, determine the behavior data and reward value at the next moment;

[0023] Determine the behavior trajectory based on the state data, behavior data and reward value at each moment.

[0024] Preferably, calculating the reward value of the behavior data includes:

[0025] Determine whether the blockchain corresponding to the behavior data meets the preset constraints;

[0026] When the blockchain corresponding to the behavior data satisfies the constraint conditions, the behavior data is substituted into the preset reward function to obtain the reward value;

[0027] When the blockchain corresponding to the behavior data does not meet the constraint conditions, the reward value is determined to be 0;

[0028] The constraints are:

[0029]

[0030] TI+TI K ≤uTI,K=1,2,...,K max

[0031] Where N is the number of nodes in the blockchain, P t is the number of malicious nodes in the blockchain, TI is the time required to generate a block, and TI K is the total consensus time when there are K shards in the blockchain, and u is the block interval.

[0032] Preferably, the method further comprises:

[0033] The model policy network is trained through the initial state behavior pairs to update the model policy network.

[0034] Preferably, obtaining the initial state-behavior pair from the optimal behavior trajectory includes:

[0035] Obtaining behavior data corresponding to the initial state data in the optimal behavior trajectory;

[0036] An initial state behavior pair is determined based on the initial state data and the corresponding behavior data.

[0037] Preferably, the process of constructing the prediction model includes:

[0038] The blockchain interacts with the initial policy network for multiple time steps to obtain multiple data quadruplets, and the multiple data quadruplets are stored in the first experience playback buffer; each array quadruplet includes the current state data, behavior data, reward value and state data of the blockchain at the next moment;

[0039] The initial prediction model is trained according to the multiple data quadruplets in the first experience playback buffer to obtain a prediction model.

[0040] Preferably, the method further comprises:

[0041] After obtaining the state data of the blockchain at the next moment, substitute the behavior data of the blockchain into the reward function to obtain the reward value, and store the current state data, behavior data, reward value and state data of the blockchain at the next moment as a data quadruple in the second experience playback buffer;

[0042] The prediction model is trained by using the data quadruple in the first experience replay buffer and the second experience replay buffer to obtain an updated prediction model; the updated prediction model is used for the state data prediction of the blockchain.

[0043] Preferably, obtaining the initial state data of the prediction model includes:

[0044] A state data of the blockchain is randomly selected from the first experience replay buffer and the second experience replay buffer as the initial state data of the prediction model.

[0045] The present invention provides a blockchain sharding strategy optimization device based on model reinforcement learning, comprising:

[0046] The acquisition module is used to obtain the current state data of the blockchain and input the current state data into the initial strategy network to obtain the behavior data of the blockchain; the behavior data of the blockchain is used to guide the blockchain sharding;

[0047] The state transfer module is used to transfer the state of the blockchain according to the behavior data of the blockchain and obtain the state data of the blockchain at the next moment;

[0048] A solution module is used to obtain the initial state data of the prediction model, and input the initial state data into the cross entropy algorithm, and call the prediction model to predict the state data at the future moment through the cross entropy algorithm, and call the model strategy network to predict the behavior data corresponding to the state data, so as to obtain the optimal behavior trajectory of the blockchain; the optimal behavior trajectory includes the state data and behavior data of the blockchain at multiple moments;

[0049] The training module is used to obtain the initial state behavior pair from the optimal behavior trajectory, and train the initial policy network through the initial state behavior pair to obtain the optimal policy network; the optimal policy network is used to generate the optimal sharding strategy for the blockchain to be sharded.

[0050] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the blockchain sharding strategy optimization method based on model reinforcement learning is implemented.

[0051] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned blockchain sharding strategy optimization method based on model reinforcement learning is implemented.

[0052] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0053] In the present invention, first, the server interacts with the blockchain to obtain the state data of the blockchain in the real environment, and then further learns the optimal behavior trajectory of the blockchain through the cross entropy algorithm. The cross entropy algorithm can be used to optimize the discrete behavior space, thereby exploring and determining an optimal behavior trajectory that can maximize the throughput of the blockchain sharding system. In addition, in the decision interval, most of the time is used to learn the optimal strategy through the cross entropy algorithm. The cross entropy algorithm generates a model data stream based on the current simulation environment, rather than data directly obtained from the real environment. Since the behavior data generation step is only the forward transfer of the policy network, its time cost is negligible compared to the length of the decision interval. Therefore, such an implementation process effectively combines the simulation environment with the blockchain, reduces the number of interactions between the server and the real environment, and thus greatly improves the sampling efficiency when learning the optimal sharding strategy of the blockchain, thereby improving the throughput of the blockchain. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0055] Figure 1 A flowchart of a blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention;

[0056] Figure 2 A flowchart of another blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention;

[0057] Figure 3A schematic diagram of the structure of a blockchain system provided by the present invention;

[0058] Figure 4 A schematic diagram of the structure of a blockchain sharding strategy optimization system based on model reinforcement learning provided by the present invention;

[0059] Figure 5 A flowchart of another blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention;

[0060] Figure 6 A flowchart of another blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention;

[0061] Figure 7 A structural block diagram of a blockchain sharding strategy optimization device based on model reinforcement learning provided by the present invention;

[0062] Figure 8 A schematic diagram of a computer device for implementing a blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0064] The blockchain sharding strategy optimization method based on model reinforcement learning provided in the embodiment of the present application can be applied to a server, which can be a server set on a business platform, or a device such as a desktop computer, a laptop computer, etc. that can execute the solution of the present invention.

[0065] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0066] Figure 1 The following is a flow chart of a blockchain sharding strategy optimization method based on model reinforcement learning in the present invention, which specifically includes the following steps:

[0067] S101, obtain the current state data of the blockchain, and input the current state data into the initial policy network to obtain the behavior data of the blockchain; the behavior data of the blockchain is used to guide the blockchain sharding.

[0068] The state data of the blockchain may include the computing power set of nodes in the blockchain, the transmission rate set between nodes, and the consensus history set. Optionally, the state data of the blockchain at time t may be defined as follows:

[0069] S t =[C t ,V t ,H t ] (1)

[0070] Among them, C t ={C t,i}, i∈N is the computing power set of the node, and N represents the number of nodes in the blockchain. V t = {V t,ij},i,j∈N is the set of transmission rates between nodes. t ={H t,i}, i∈N is the set of consensus history of the previous epoch (epoch usually refers to a time period or batch of block generation); when the consensus opinion of node i is valid, H t,i =0, otherwise, H t,i =1.

[0071] The initial policy network can be a neural network model that predicts the behavior space of the blockchain through the state data of the blockchain. Therefore, the current state data is input into the initial policy network to obtain the behavior data of the blockchain. The current state data is input into the initial policy network to obtain the probability value of each discrete behavior of the blockchain; the discrete behavior includes the block size of the blockchain, the number of shards, and the time required to generate a block; according to the probability value of each discrete behavior, the behavior data of the blockchain is determined.

[0072] The behavior space of the blockchain at time t can be defined as follows:

[0073] a t =[B t ,K t ,TI t ] (2)

[0074] Among them, B t ={1,2,3,...,B max} represents the block size, K t ={1,2,3,...,K max} represents the number of shards of the blockchain, TI t ={1,2,3,...,TI max} represents the time required to generate a block. It should be noted that B t ,K t ,TI t are all discrete variables.max Can be the maximum block size, K max The maximum number of shards that a blockchain can be sharded into, TI max The maximum time required to generate a block.

[0075] Therefore, the current state data is input into the initial strategy network, and the initial strategy network can output the probability value of each discrete behavior, that is, output B t , K t and TI t The probability value of B t For example, B t ={1,2,3}, output B t =1, B t =2 and B t =3 probability value.

[0076] In this way, the behavior data of the blockchain is determined according to the probability value of each discrete behavior, including: for example, for any behavior, the value with the largest probability value is determined as the value of the corresponding behavior, and the value of each behavior is used as the value of the behavior data; or, the value of each discrete behavior is randomly sampled to determine the behavior data of the blockchain.

[0077] S102, transferring the state of the blockchain according to the behavior data of the blockchain to obtain the state data of the blockchain at the next moment.

[0078] Behavioral data through blockchain t Interact with the blockchain system, transfer the blockchain to the next state, and obtain the state data of the blockchain at the next moment.

[0079] At the same time, the reward value of the blockchain is determined according to the behavior data of the blockchain, that is, the behavior data of the blockchain is substituted into the reward function to obtain the reward value, and the current state data, behavior data, reward value and state data of the blockchain at the next moment are taken as the data quadruple (s t ,a t ,r t ,s t+1 ) is stored in the second experience playback buffer; the second experience playback buffer can be used with D RL express.

[0080] S103, obtaining the initial state data of the prediction model, and inputting the initial state data into the cross entropy algorithm, calling the prediction model through the cross entropy algorithm to predict the state data at the future moment, and calling the model strategy network to predict the behavior data corresponding to the state data, to obtain the optimal behavior trajectory of the blockchain.

[0081] Among them, the optimal behavior trajectory includes the state data and behavior data of the blockchain at multiple times.

[0082] Before sharding, a prediction model can be constructed first, and the state data of the blockchain at the next moment can be predicted through the prediction model. Preferably, the construction process of the prediction model includes: interacting the blockchain with the initial policy network for multiple time steps to obtain multiple data quadruplets, and storing the multiple data quadruplets in the first experience replay buffer; each array quadruplet includes the current state data, behavior data, reward value and state data of the blockchain at the next moment; training the initial prediction model according to the multiple data quadruplets in the first experience replay buffer to obtain the prediction model.

[0083] Specifically, the initial policy network can interact with the blockchain system for G time steps to generate a series of data quadruplets (s ) consisting of state data, behavior data, reward value, and next state. g ,a g ,r g ,s g+1 ),g∈{0,1,...,G-2}. These data quadruplets can be stored in the first experience playback buffer D RAND In. D RAND Used to train the prediction model f (prediction model input: current state data; output: state data at the next moment) and sample experience in the initial stage.

[0084] The method of obtaining the initial state data of the prediction model may be to randomly select a state data of the blockchain from the first experience playback buffer and the second experience playback buffer as the initial state data of the prediction model. For example, randomly select an experience (s i ,a i ,r i ,s i+1 ), i is a certain moment, and s i As the initial state data s'0 of the prediction model rolling, starting from s'0, the behavior trajectory ζ = (s i ,a'0,r'0,s' i+1 ...,s i+T-1 ,a' T-1 ,r' T-1 ,s' i+T ).

[0085] Preferably, if Figure 2As shown, the initial state data is input into the cross entropy algorithm, the prediction model is called by the cross entropy algorithm to predict the state data at the future moment, and the model strategy network is called to predict the behavior data corresponding to the state data, so as to obtain the optimal behavior trajectory of the blockchain, including the following steps:

[0086] S201, input the initial state data into the cross entropy algorithm, call the prediction model through the cross entropy algorithm to predict the state data at the future moment, and call the model strategy network to predict the behavior data corresponding to the state data, and obtain multiple different behavior trajectories.

[0087] Among them, each behavior trajectory includes the state data, behavior data and corresponding reward value of the blockchain at multiple times.

[0088] Preferably, for any behavior trajectory, the initial state data can be input into the model strategy network to obtain the behavior data corresponding to the initial state data, and the reward value of the behavior data can be calculated, and then the initial state data can be input into the prediction model to obtain the state data at the next moment; and based on the state data at the next moment, the behavior data and reward value at the next moment can be determined; finally, the behavior trajectory can be determined based on the state data, behavior data and reward value at each moment.

[0089] The model policy network can be a policy network with the same structure as the initial policy network but with different weight parameters; from the initial state data s' j Start by setting the initial state data s' j Input to the model policy network π * In the model policy network π * Output behavior data a' j , and calculate the behavior data a' j The reward value r' j , and then use the prediction model f to predict the state data s' at the next moment j+1 , and determine the behavior data and reward value at the next moment based on the state data at the next moment. This process can last for T steps to generate a complete behavior trajectory ζ=(s i ,a'0,r'0,s' i+1 ...,s i+T-1 ,a' T-1 ,r' T-1 ,s' i+T ).

[0090] By repeating this complete process S times, S different behavior trajectories can be obtained. It should be noted that the initial state data corresponding to each behavior trajectory can be different, and multiple different behavior trajectories can be constructed by obtaining different initial state data.

[0091] Optionally, calculating the reward value of the behavior data includes: determining whether the blockchain corresponding to the behavior data satisfies preset constraints; when the blockchain corresponding to the behavior data satisfies the constraints, substituting the behavior data into a preset reward function to obtain a reward value; when the blockchain corresponding to the behavior data does not satisfy the constraints, determining the reward value to be 0.

[0092] The reward function is shown in formula (3).

[0093]

[0094] Among them, r t represents the reward value of the state data at time t, B h represents the size of the block header, b represents the average size of a transaction, (B t -B h ) represents the size of transactions processed in each block.

[0095] The constraints are:

[0096]

[0097] TI+TI K ≤uTI,K=1,2,...,K max (6)

[0098] Where N is the number of nodes in the blockchain, P t is the number of malicious nodes in the blockchain, TI is the time required to generate a block, and TI K is the total consensus time when there are K shards in the blockchain, and u is the block interval.

[0099] It should be noted that formulas (4) and (5) are security constraints, which aim to ensure that after sharding, malicious nodes in the shard will not threaten the security of the entire shard, and blocks generated by malicious nodes will not be uploaded to the blockchain. Formula (6) is a delay constraint, which aims to meet the final property of the blockchain, that is, the delay should be completed within several consecutive block intervals u, TI K Represents the consensus time within the blockchain shard, and the probability of malicious nodes P t The calculation of depends on the consensus history.

[0100] S202, determining a total reward value of each behavior trajectory according to multiple reward values ​​in each behavior trajectory.

[0101] For any behavior trajectory, multiple reward values ​​of each behavior trajectory can be substituted into the objective function to obtain the objective function value, and the objective function value obtained for each behavior trajectory is used as the total reward value of each behavior trajectory. The objective function is shown in formula (7).

[0102]

[0103] Where T represents the number of reward values, γ t Represents the discount factor corresponding to the behavioral data at time t.

[0104] S203, determining a preset number of behavior trajectories with the highest total return value as the optimal behavior trajectory of the blockchain.

[0105] A preset number of behavior sequences with the highest total return value are selected from the generated behavior trajectories as the optimal behavior trajectory; for example, e behavior sequences with the highest total return are selected from the generated S trajectories to form the optimal behavior trajectory.

[0106] S104, obtaining an initial state-behavior pair from the optimal behavior trajectory, and training an initial policy network through the initial state-behavior pair to obtain an optimal policy network.

[0107] Among them, the optimal strategy network is used to generate the optimal sharding strategy for the blockchain to be sharded.

[0108] Acquiring an initial state behavior pair from an optimal behavior trajectory includes: acquiring behavior data corresponding to the initial state data in the optimal behavior trajectory; and determining the initial state behavior pair according to the initial state data and the corresponding behavior data.

[0109] The initial state behavior pair may include a state data and behavior data. The initial state data may be used as the state data in the initial state behavior pair, and the behavior data corresponding to the behavior trajectory with the highest total return value among the behavior data corresponding to the initial state data in the optimal behavior trajectory may be used as the behavior data in the initial state behavior pair. That is, the initial state behavior pair is: (s i ,a'0).

[0110] The initial state behavior pair can be stored in the third experience playback buffer D model In, and according to D elite The initial state behavior in the training initial policy network The initial policy network that has been trained Determine the optimal policy network. Optionally, during the training process, the initial policy network can be optimized using the loss function defined below:

[0111]

[0112] Among them, θ is the initial policy network Parameters.

[0113] Optionally, taking the preset number as e as an example, the initial state behavior pair may also include the initial state data and all behavior data corresponding to the initial state data in the optimal behavior trajectory, that is, the initial state behavior pair is: The model policy network can be trained by the initial state behavior pair to update the model policy network. Specifically, the initial state behavior pair can be stored in the elite experience playback buffer D elite According to formula (9) and D elite The initial state behavior in the training model policy network π * .

[0114]

[0115] Among them, θ - is the policy network π * The parameter α is the learning rate. This process improves π * In D elite (represents the probability of taking an action in the high-reward action space). By iterating the entire process Z times, the cross entropy algorithm can better explore the solution space and approach the optimal solution to the OBCS problem. During the iteration process, the value of parameter S gradually decreases, allowing the algorithm to pay more attention to the high-reward area in the later stage. After the last iteration, the model selects the optimal action trajectory ζ with the best action sequence.

[0116] After obtaining the optimal strategy network, the prediction model can also be trained using the data quadruple in the first experience replay buffer and the second experience replay buffer to obtain an updated prediction model; the updated prediction model is used for predicting the state data of the blockchain.

[0117] Optionally, the state data of the blockchain to be sharded can be input into the optimal strategy network to obtain the sharding strategy of the blockchain to be sharded.

[0118] The blockchain to be sharded is the blockchain to be traded; the state data of the blockchain to be sharded is input into the optimal strategy network to obtain the probability value of each discrete behavior of the blockchain to be sharded; the probability value of each discrete behavior is determined as the sharding strategy of the blockchain to be sharded.

[0119] Optionally, the value of each discrete behavior of the blockchain to be sharded can be determined according to the probability value of the discrete behavior, that is, the value with the largest probability value in each discrete behavior is used as the value of each discrete behavior, and then the value of each discrete behavior is used as behavior data to determine the sharding strategy of the blockchain to be sharded.

[0120] After obtaining the sharding strategy of the blockchain to be sharded, obtain the time taken by each node in the blockchain to be sharded to solve the proof-of-work problem, and sort the nodes in ascending order of time; assign the nodes with a certain number of time in the front to the directory committee; and assign the remaining nodes to different shards of the blockchain to be sharded according to the last preset number of bits of their respective identifiers.

[0121] Specifically, at time t, each node in the blockchain to be sharded participates in solving the Proof of Work (PoW) problem according to its computing power, and the time it takes for each node to solve the Proof of Work problem is determined. A specific number of nodes that can solve the PoW problem faster are assigned to the Directory Committee (DC), and the remaining nodes are assigned to different shards according to the last preset number of digits of their respective identifiers (ID) (for example, if the preset number is 2, the last two digits represent the shard number).

[0122] like Figure 3 As shown, Figure 3 The blockchain system consists of N nodes and K shards. In each shard, a block producer is designated as the master node (faster / random in solving puzzles), responsible for packaging transactions into blocks and executing the Practical Byzantine Fault Tolerance (PBFT) consensus algorithm for verification to defend against potential Byzantine node attacks.

[0123] After the block verification is completed within the shard, the verified block will be transmitted to the DC. As the master node, the DC is responsible for centrally executing the PBFT consensus and taking on the final block packaging process to ensure the finality and consistency of the block. Once the block passes the PBFT consensus verification, the DC will package it into a final block and broadcast it to the entire blockchain network. At the same time, the DC will synchronize the final block information with each shard node to ensure real-time update and synchronization of system data.

[0124] The blockchain sharding strategy optimization method based on model reinforcement learning in the present invention is essentially a sharding control algorithm (Model-Based Policy Optimization with Batch Sampling, MBPOBS) based on model-based policy optimization (MBPO). Figure 4 As shown, Figure 4 The present invention is a structural schematic diagram of a blockchain sharding strategy optimization system based on model reinforcement learning, which is a blockchain sharding strategy optimization method based on model reinforcement learning. The system controls the optimal sharding of the blockchain through two experience streams: real experience stream and simulated experience stream. The real experience stream refers to the experience gained from real interactions within the blockchain, which is directly used to train the prediction model; while the simulated experience stream is generated by the prediction model and is used to learn the optimal strategy. This method can reduce the number of real interactions and improve sampling efficiency.

[0125] Specifically, in the blockchain environment, the state data st Input to the optimal policy network In the above example, we determine the behavior data a t , then the state data s t Execute behavior data a t , through sharding information and network monitoring, calculate the consensus inconsistency and malicious node rate, and then calculate the reward value r t and the state data s at the next moment t+1 , thus obtaining the data quadruple (s t ,a t ,r t ,s t+1 ), and the data quadruple (s t ,a t ,r t ,s t+1 ) is stored in the experience playback buffer D RL middle.

[0126] Can be in D RL Randomly sample M state data M t > and input it into the model policy network π * , get the corresponding behavior data And through sharding information and network monitoring, calculate the consensus inconsistency and malicious node rate, and then calculate the reward value r t ′, and through the prediction model f the state data s′ at the next moment t+1 , the process iterates and calculates T steps to generate a complete behavior trajectory. The construction process of the behavior trajectory can be iterated S times to generate S behavior trajectories, and calculate the total reward of each behavior trajectory, and select the top e behavior sequences with the highest total reward as the optimal behavior trajectory and store it in the elite experience playback buffer D elite , D elite For training model policy network π * .

[0127] In addition, the first behavior data and state data of the behavior sequence with the highest total reward are stored in the experience replay buffer D model In, D model Used to train the optimal policy network

[0128] The entire workflow process is as follows Figure 5 As shown, the process is as follows:

[0129] S501, initializing three experience playback buffers D RAND , D RL , D mod el , and initialize the initial policy network and predictive models.​

[0130] S502, interact with the blockchain system for G time steps to generate a series of data quadruplets consisting of state, behavior, reward, and next state, which are then stored in the experience replay buffer D RAND , and through D RAND Train the predictive model.

[0131] S503, obtaining the current state data of the blockchain in the system environment, and inputting the current state data into the initial strategy network to obtain behavior data, and then executing the behavior data in the blockchain and calculating the reward value.

[0132] S504, obtain the state data of the blockchain at the next moment, generate a data quadruple, and store the data quadruple in D RL .

[0133] S505, from D RL and D RAND A data quadruple is randomly selected from the , and the optimal behavior trajectory is obtained by combining the state data in the data quadruple through the cross entropy algorithm.

[0134] S506, storing the initial state behavior pair in the optimal behavior trajectory into D mod el .

[0135] S507, by D mod el Train the initial policy network and pass D RL and D RAND Train the predictive model.

[0136] S508, Clear D mod el .

[0137] S509: Determine the initial policy network as the optimal policy network.

[0138] It should be noted that the above steps S503-S508 need to be iteratively executed N times, and the above steps S505-S506 need to be iteratively executed M times.

[0139] In the blockchain sharding system, the present invention targets three status values ​​with different characteristics: (1) node computing capacity C t , (2) The transmission rate between nodes V t , (3) Consensus history H t , different prediction methods are used. t and V t With randomness and continuity, the prediction model selected Gaussian Processes Regression (GPR) model. tSince the update is based on the known state transition rules and the current state, we directly use the state transition rules in the system for prediction. This method not only ensures the accuracy of the consensus history value, but also avoids the uncertainty caused by randomness.

[0140] Gaussian process regression assumes that state C t and V t The observations of follow a Gaussian process, that is, the joint distribution of any finite number of observations follows a multivariate Gaussian distribution. t and V t After the current state value is obtained, the GPR model f will output a Gaussian distribution, whose mean and variance are determined by the mean function and covariance function respectively. The predicted next state is the mean of the predicted distribution. The covariance function uses the Radial Basis Function (RBF) kernel.

[0141] After making a decision, the remaining time in the decision interval will be used for planning, during which the cross entropy algorithm generates a simulated experience stream based on the model. Since the behavior generation step (step S503) only involves passing through the initial policy network The time cost of performing a forward pass is negligible relative to the length of the decision interval. Therefore, it is assumed that there is enough time for planning. The planning process (step S505) can be performed by calling Figure 6 This is implemented using the CEM algorithm outlined in .

[0142] like Figure 6 As shown, the implementation method of obtaining the optimal behavior trajectory by combining the state data in the data quadruple through the cross entropy algorithm in the above step S505 includes the following steps:

[0143] S601, initialize the model policy network and elite experience replay buffer.

[0144] S602, initialize an array X of length S, a trajectory w, and a trajectory set W.

[0145] S603, clear elite experience replay buffer, X, W and W.

[0146] S604: Use the state data as initial state data.

[0147] S605, input the current state data into the model strategy network, obtain the behavior data, execute the behavior data and calculate the reward value; predict the state data at the next moment according to the prediction model; finally, store the state data, behavior data, reward value and the state data at the next moment into w.

[0148] S606 , calculate the total reward of w, store the total reward in X, and store w in W.

[0149] S607, select the e trajectories with the highest total rewards in X, add the data corresponding to the e trajectories to the elite experience playback buffer, and use the behavior data and state data corresponding to the trajectory with the highest total reward as the initial state behavior pair.

[0150] S608, updating the model policy network through the elite experience playback buffer.

[0151] S609, select the optimal behavior trajectory from W and output it.

[0152] It should be noted that the above steps S603-S608 need to be iteratively executed Z times, steps S604-S606 need to be iteratively executed S times, and step S605 needs to be iteratively executed T times.

[0153] The core idea of ​​this invention is to use model-based reinforcement learning to find higher-quality sharding strategies, thereby solving the OBCS problem more effectively. In addition, MBPOBS innovatively integrates Gaussian process regression to accurately predict future states, and on this basis, imitates learning through the demonstrator of the cross entropy algorithm to learn the best sharding strategy. Compared with MFRL, MBPOBS has faster efficiency, stronger generalization ability and higher stability in learning the optimal strategy.

[0154] When applying the blockchain sharding strategy optimization method based on model reinforcement learning provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0155] The above is a blockchain sharding strategy optimization method based on model reinforcement learning provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding blockchain sharding strategy optimization device based on model reinforcement learning, such as Figure 7 shown.

[0156] Figure 7 The present invention provides a structural block diagram of a blockchain sharding strategy optimization device based on model reinforcement learning, wherein the device 700 includes:

[0157] The acquisition module 701 is used to obtain the current state data of the blockchain and input the current state data into the initial policy network to obtain the behavior data of the blockchain; the behavior data of the blockchain is used to guide the blockchain sharding.

[0158] The state transfer module 702 is used to transfer the state of the blockchain according to the behavior data of the blockchain to obtain the state data of the blockchain at the next moment.

[0159] The solution module 703 is used to obtain the initial state data of the prediction model and input the initial state data into the cross entropy algorithm. The cross entropy algorithm is used to call the prediction model to predict the state data at future moments, and the model strategy network is called to predict the behavior data corresponding to the state data to obtain the optimal behavior trajectory of the blockchain; the optimal behavior trajectory includes the state data and behavior data of the blockchain at multiple moments.

[0160] The training module 704 is used to obtain the initial state behavior pair from the optimal behavior trajectory, and train the initial policy network through the initial state behavior pair to obtain the optimal policy network; the optimal policy network is used to generate the optimal sharding strategy for the blockchain to be sharded.

[0161] For the specific definition of the blockchain sharding device, please refer to the definition of the blockchain sharding strategy optimization method based on model reinforcement learning above, which will not be repeated here. Each module in the above-mentioned blockchain sharding strategy optimization device based on model reinforcement learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The blockchain sharding strategy optimization method based on model reinforcement learning is provided.

[0163] The present invention also provides Figure 8 The structural diagram of the computer device shown in FIG. Figure 8 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The blockchain sharding strategy optimization method based on model reinforcement learning is provided.

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0165] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A blockchain sharding strategy optimization method based on model reinforcement learning, characterized in that: The method comprises: Obtaining current state data of the blockchain, and inputting the current state data into the initial policy network to obtain behavior data of the blockchain; the behavior data of the blockchain is used to guide the blockchain sharding; Transferring the state of the blockchain according to the behavior data of the blockchain to obtain the state data of the blockchain at the next moment; Acquire the initial state data of the prediction model, and input the initial state data into the cross entropy algorithm, call the prediction model through the cross entropy algorithm to predict the state data at the future moment, and call the model strategy network to predict the behavior data corresponding to the state data, and obtain the optimal behavior trajectory of the blockchain; the optimal behavior trajectory includes the state data and behavior data of the blockchain at multiple moments; An initial state behavior pair is obtained from the optimal behavior trajectory, and the initial policy network is trained by the initial state behavior pair to obtain an optimal policy network; the optimal policy network is used to generate an optimal sharding strategy for the blockchain to be sharded.

2. The method according to claim 1, characterized in that The step of inputting the current state data into the initial policy network to obtain the behavior data of the blockchain includes: Input the current state data into the initial strategy network to obtain the probability value of each discrete behavior of the blockchain; the discrete behavior includes the block size, the number of shards and the time required to generate a block of the blockchain; According to the probability value of each discrete behavior, the behavior data of the blockchain is determined.

3. The method according to claim 1 or 2, characterized in that: The method of calling the prediction model to predict the state data at a future moment through the cross entropy algorithm and calling the model strategy network to predict the behavior data corresponding to the state data to obtain the optimal behavior trajectory of the blockchain includes: The cross entropy algorithm is used to call the prediction model to predict the state data at the future moment and to call the model strategy network to predict the behavior data corresponding to the state data, thereby obtaining multiple different behavior trajectories; each behavior trajectory includes the state data, behavior data and corresponding reward value of the blockchain at multiple moments; Determine the total reward value of each behavior trajectory based on multiple reward values ​​in each behavior trajectory; A preset number of behavior trajectories with the highest total return value are determined as the optimal behavior trajectories of the blockchain.

4. The method according to claim 3, characterized in that The cross entropy algorithm is used to call the prediction model to predict the state data at the future moment and to call the model strategy network to predict the behavior data corresponding to the state data, to obtain multiple different behavior trajectories, including: For any behavior trajectory, the initial state data is input into the model strategy network to obtain the behavior data corresponding to the initial state data, and the reward value of the behavior data is calculated; Inputting the initial state data into the prediction model to obtain state data at the next moment; According to the state data at the next moment, determine the behavior data and reward value at the next moment; The behavior trajectory is determined according to the state data, behavior data and reward value at each moment.

5. The method according to claim 4, characterized in that The calculating the reward value of the behavior data includes: Determine whether the blockchain corresponding to the behavior data satisfies preset constraints; When the blockchain corresponding to the behavior data satisfies the constraint condition, substituting the behavior data into a preset reward function to obtain the reward value; When the blockchain corresponding to the behavior data does not satisfy the constraint condition, determining the reward value to be 0; The constraints are: YOU+YOU K ≤uTI,K=1,2,...,K max Where N is the number of nodes in the blockchain, P t is the number of malicious nodes in the blockchain, TI is the time required to generate a block, and TI K is the total consensus time when there are K shards in the blockchain, and u is the block interval.

6. The method according to claim 4, characterized in that The method further comprises: The model policy network is trained by the initial state behavior pair to update the model policy network.

7. The method according to claim 1 or 2, characterized in that: The obtaining of the initial state-behavior pair from the optimal behavior trajectory includes: Obtaining behavior data corresponding to the initial state data in the optimal behavior trajectory; The initial state behavior pair is determined according to the initial state data and the corresponding behavior data.

8. The method according to claim 1 or 2, characterized in that: The process of constructing the prediction model includes: The blockchain interacts with the initial policy network for multiple time steps to obtain multiple data quadruplets, and the multiple data quadruplets are stored in a first experience playback buffer; each array quadruplet includes current state data, behavior data, reward value and state data at the next moment of the blockchain; The initial prediction model is trained according to the multiple data quadruplets in the first experience playback buffer to obtain the prediction model.

9. The method according to claim 8, characterized in that The method further comprises: After obtaining the state data of the blockchain at the next moment, substituting the behavior data of the blockchain into the reward function to obtain the reward value, and storing the current state data, behavior data, reward value and state data of the blockchain at the next moment as a data quadruple in a second experience replay buffer; The prediction model is trained using the data quadruple in the first experience replay buffer and the second experience replay buffer to obtain an updated prediction model; the updated prediction model is used for status data prediction of the blockchain.

10. The method according to claim 9, characterized in that The obtaining of the initial state data of the prediction model includes: A state data of the blockchain is randomly selected from the first experience replay buffer and the second experience replay buffer as the initial state data of the prediction model.

Images