Block chain load balancing method and system based on deep reinforcement learning

Abstract

The invention discloses a block chain load balancing method and system based on deep reinforcement learning, and the method comprises the steps: comprehensively learning a transaction track of an account and a fragment in a block chain network and a cross-fragment transaction proportion, so as to determine an optimal account fragment position of the account in a next transaction period; the working process of the block chain fragment load balancing middleware is as follows: each account transaction record and each fragment processing transaction record of a current transaction period of a fragment block chain network are used as states to be transmitted to a deep reinforcement learning model; the deep reinforcement learning model gives the optimal fragment position of each account in the next transaction cycle through reinforcement learning; the verifier node in the fragments initiates an account migration process according to the decision of the model; and the account and the non-block transaction are transferred to a new fragment, and the next transaction period is started. According to the block chain load balancing method and system, block chain fragment load balancing is realized, and meanwhile, the influence of cross-fragment transaction caused by account migration on system performance is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a blockchain load balancing method and system based on deep reinforcement learning. Background Art

[0002] Industrial Internet of Things (IIoT) is a key technology to promote the intelligent and information-based transformation of traditional manufacturing enterprises. As modern industrial production continues to increase its demand for data security and system availability, blockchain technology has attracted widespread attention from many researchers in the field of industrial Internet of Things due to its excellent high availability and tamper-proof characteristics. "Sharding" is an innovative concept transplanted from the field of cloud computing and cloud databases, which aims to solve the serious scalability problems in traditional blockchain systems by adopting parallel computing. Under the sharding architecture, the original blockchain network is divided into several "committees" called shards, in which the validators in a single committee are only responsible for the consensus and block generation of a subset of blockchain transactions, and all ledgers are only maintained locally by the validators in the shards. In this way, all shards can perform tasks in parallel, thereby improving the scalability of the blockchain system. Due to the huge performance improvement of blockchain networks assisted by sharding technology compared to traditional blockchains, its application in fields such as medical care, Internet of Things, and Internet of Vehicles is becoming more and more in-depth.

[0003] Although the industry has evolved a variety of performance-optimized blockchain sharding solutions, the partitioning and parallel computing concepts of sharding itself also bring two prominent problems that limit its further large-scale application in the industrial Internet of Things production environment. The first bottleneck is how to achieve efficient cross-shard transactions. This is because if the same transaction involves multiple shards, it is necessary to sacrifice performance to ensure data consistency. The second bottleneck is how to balance the load between shards to improve system throughput while making full use of the company's precious computing resources. However, there is still no better solution that can take into account both problems during blockchain operation to improve performance. The existing load balancing strategy ignores the performance impact of the change in the number of cross-shard transactions caused by account migration on the system. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a blockchain load balancing method and system based on deep reinforcement learning. The sharding middleware monitors and learns the workload and cross-shard transaction ratio of blockchain shards and accounts in the production environment, and determines the best shard to which the account belongs, thereby achieving efficient cross-shard transactions and load balancing between shards.

[0005] The present invention proposes a blockchain load balancing method based on deep reinforcement learning, which comprehensively learns the transaction tracks of accounts and shards in the blockchain network and the cross-shard transaction ratio to determine the optimal shard position of the account in the next transaction cycle;

[0006] The working process of the blockchain sharding load balancing middleware is as follows:

[0007] The consortium blockchain transaction monitoring component continuously monitors blockchain network activities and saves them to the in-memory database;

[0008] The shard load balancer obtains the network's account and shard workload data in two consecutive transaction cycles from the database, and converts the account and shard workload data in the transaction cycle into the state and action of reinforcement learning;

[0009] Based on the state and action, the load balancer uses a deep reinforcement learning model to calculate the action with the largest long-term reward as the final prediction result of the account belonging to the new shard;

[0010] For each account that needs to be moved to the new shard, a migration transaction is initiated for the account of the new shard based on the final prediction result. The migration transaction will be routed to the shard to which the account currently belongs, waiting for the validators to reach a consensus;

[0011] The validator node responsible for packaging the migration transaction will initiate a new cross-shard transaction to the new shard of the account. The new cross-shard transaction contains the account information waiting to be migrated and a list of pending transactions related to the account. The account information will then be deleted from the original shard, and the validator of the new shard will broadcast the new account information within the shard so that all validators can process transactions for the new account.

[0012] Furthermore, in the process of calculating the action with the largest long-term reward through the deep reinforcement learning model as the final prediction result of the account belonging to the new shard, the basic parameters of the deep reinforcement learning model are specifically:

[0013] Action space: Blockchain network sets shard S = {S1, S2, ..S i ,,...}, each shard has A={A1,A2,..A j , ...} account set, the S i is the i-th shard, A j is the jth account;

[0014] Status: 1) Status information related to the account, including the account ID, the number of transactions submitted in a certain cycle, the number of transactions that have reached consensus, the proportion of cross-shard transactions in the number of submitted transactions, the most frequently appearing destination shard ID in cross-shard transactions, and the shard ID to which the account belongs; 2) Status information related to the shard, including the shard ID, the total number of transactions processed by the shard in a certain cycle, and the proportion of cross-shard transactions in the total number of transactions processed.

[0015] Furthermore, the basic parameters of the deep reinforcement learning model also include a reward function, which includes a CST reward R1 for the cross-shard transaction ratio and a load uniformity reward R2 for workload data, where:

[0016] The calculation of CST reward R1 is as follows:

[0017] After account A is reallocated, the workload data will be updated according to formula (1); based on the updated workload data, the new cross-shard transaction ratio is calculated according to formula (2); the penalty method is applied and the CST reward R1 is calculated according to formula (3);

[0018] M′(S i )=M(S i )+M(A); (1)

[0019] P′(S i )=(P(S i )×M(S i )+P(A,S i )×M(A)) / M′(S i ); (2)

[0020] R1=-(P′(S i )-P(S i )); (3)

[0021] Among them, M′(S i ) indicates shard S i The workload data updated in a certain period, M(S i ) indicates shard S i The workload data in a certain period, M(A) is the workload data of account A in a certain period, P′(S i ) is the new cross-shard transaction ratio, P(S i ) is the fragment S i The proportion of cross-shard transactions processed in a certain period, M(S i ) is the fragment S i The workload data in a certain period is the total number of transactions processed, P(A, S i ) is the fragment S iThe percentage of the total number of cross-shard transactions sent by account A in a certain period.

[0022] Furthermore, the calculation of the load evenness reward R2 is as follows:

[0023] The workload standard deviation of the current cycle is calculated based on formula (4), and then the standard deviation of the new cycle is calculated in formula (5), and the load uniformity reward R2 is obtained in formula (6) through the penalty method;

[0024]

[0025] R2=-(D'-D); (6)

[0026] Where N is the total number of shards, M(S i ) is the fragment S i Workload data in the current cycle (i.e., the total number of transactions successfully packaged by the shard in the cycle), M'(S i ) is the fragment S i Workload data in the new cycle, is the mean value of workload data in the current cycle, is the mean of the workload data in the new cycle, D is the standard deviation of transactions in all shards of the blockchain in the current cycle, and D' is the standard deviation of transactions in all shards of the blockchain in the new cycle.

[0027] Furthermore, the reward function R of the deep reinforcement learning model is constructed based on the CST reward R1 and the load uniformity reward R2:

[0028] R=w1×R1+w2×R2

[0029] Among them, w1 and w2 are weight coefficients for balancing the cross-shard transaction ratio and workload balance, respectively.

[0030] Furthermore, the Q function of the deep reinforcement learning model is specifically:

[0031]

[0032] Among them, Q(s t ,a t ) represents the state s t Take action a t Q value; Q v (s t ) represents the state s t The value function under Q a (s t ,a t ) represents the state s t Take action a tAdvantage functions over taking other actions; Represents in state s t The average value of the advantage function of taking all possible actions under state s; A represents the set of action spaces of the deep reinforcement model, |A| represents the size of the action space (that is, the number of all possible actions), and a'∈A represents the t List every possible action you can take.

[0033] Furthermore, the training process of the deep reinforcement learning model is as follows:

[0034] The blockchain transaction records of data upload devices in the real industrial blockchain network were collected and cleaned for cleaning and standardization, and grouped according to shards to determine the cross-shard transaction ratio standard;

[0035] Define the target shard set of the transaction based on all shards involved in the account's transaction history in the real blockchain environment;

[0036] The shard and transaction initiation time of the set account are input into the long short-term memory model as raw data to predict the behavior of the set account in the next transaction cycle. The obtained prediction results are verified to ensure that they are consistent with the configuration of the account and shard in the deep reinforcement learning model. If passed, the raw data and the prediction results are combined into an extended data set and provided to the intelligent agent;

[0037] Based on the expanded environmental data, the agent continuously explores the environment, and continuously optimizes the action strategy with the best Q value in different states by combining the environmental feedback brought by the strategy and the enhanced Q function until the deep reinforcement learning model training is completed;

[0038] The deep reinforcement learning model training that has passed the verification test is applied to make decisions in the actual blockchain network.

[0039] A blockchain load balancing system based on deep reinforcement learning, which comprehensively learns the transaction trajectories of accounts and shards in the blockchain network and the cross-shard transaction ratio to determine the best shard position of the account in the next transaction cycle;

[0040] The working process of the blockchain sharding load balancing middleware is as follows:

[0041] The consortium blockchain transaction monitoring component continuously monitors blockchain network activities and saves them to the in-memory database;

[0042] The shard load balancer obtains the network's account and shard workload data in two consecutive transaction cycles from the database, and converts the account and shard workload data in the transaction cycle into the state and action of reinforcement learning;

[0043] Based on the state and action, the load balancer uses a deep reinforcement learning model to calculate the action with the largest long-term reward as the final prediction result of the account belonging to the new shard;

[0044] For each account that needs to be moved to the new shard, a migration transaction is initiated for the account of the new shard based on the final prediction result. The migration transaction will be routed to the shard to which the account currently belongs, waiting for the validators to reach a consensus;

[0045] The validator node responsible for packaging the migration transaction will initiate a new cross-shard transaction to the new shard of the account. The new cross-shard transaction contains the account information waiting to be migrated and a list of pending transactions related to the account. The account information will then be deleted from the original shard, and the validator of the new shard will broadcast the new account information within the shard so that all validators can process transactions for the new account.

[0046] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the blockchain load balancing method as described above when executing the computer program.

[0047] A computer-readable storage medium having a plurality of classification programs stored thereon, wherein the plurality of classification programs are used to be called by a processor and execute the blockchain load balancing method as described above.

[0048] The advantages of the blockchain load balancing method and system based on deep reinforcement learning provided by the present invention are: the workload balancing problem of industrial Internet of Things blockchain shards is modeled as a Markov decision process (Markov Decision Process, MDP) and a deep reinforcement learning load balancing algorithm based on deep double Q network (D3QN) is designed accordingly, which not only considers the workload of accounts and shards, but also integrates the cross-shard transaction characteristics of accounts. This paper also provides LSTM to assist the model training process, thereby improving the accuracy of the load balancing middleware and improving the deployment efficiency in practical applications; the load balancing algorithm is encapsulated as a middleware of an industrial blockchain network, and the account location decision that optimizes the system performance is made by tracking the history of shard and account activity in each transaction cycle of the network, so as to meet the user's requirements for high data throughput and low transaction confirmation delay in industrial production scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the system model of the present invention;

[0050] Figure 2 A schematic diagram of the training process of the deep reinforcement learning model of the present invention;

[0051] Figure 3This is a schematic diagram of the variance of transaction processing volume of each shard under different numbers of shards;

[0052] Figure 4 Schematic diagram of the changing trend of the variance of shard transaction processing volume at different transaction initiation rates. DETAILED DESCRIPTION

[0053] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0054] like Figures 1 to 4 As shown, the present invention proposes a blockchain load balancing method based on deep reinforcement learning, which comprehensively learns the transaction trajectories of accounts and shards and the cross-shard transaction ratio in the blockchain network to determine the optimal shard position of the account in the next transaction cycle.

[0055] The working process of blockchain sharding load balancing middleware is as follows:

[0056] S11. The consortium blockchain transaction monitoring component continuously monitors blockchain network activities and saves them to the memory database;

[0057] S12, the shard load balancer obtains the account and shard workload data of the network in two consecutive transaction cycles from the database, and converts the account and shard workload data in the transaction cycle into states and actions of reinforcement learning;

[0058] S13. The load balancer calculates the action with the largest long-term reward based on the state and action through the deep reinforcement learning model as the final prediction result of the account belonging to the new shard;

[0059] S14. For each account that needs to be moved to the new shard, a migration transaction is initiated for the account of the new shard according to the final prediction result. The migration transaction will be routed to the shard to which the account currently belongs, waiting for the validator to reach a consensus;

[0060] S15. The validator node responsible for packaging the migration transaction will initiate a new cross-shard transaction to the new shard of the account. The new cross-shard transaction contains the account information waiting to be migrated and a list of pending transactions related to the account. The account information will then be deleted from the original shard, and the validator of the new shard will broadcast the new account information within the shard so that all validators can process transactions of the new account.

[0061] In this embodiment, a complete middleware operation process will last for two shard transaction cycles, that is, the account position of the network may be adjusted after every two cycles. For the convenience of description, two chronological cycles are selected here to illustrate the workflow of the load balancer. In the first cycle E, the monitoring center (MC) acts as a trusted third party to continuously monitor the blockchain network activities and continuously save them to the memory database. For the load-balancing middleware, the most important information includes the number of transactions sent by the block in this transaction cycle and the transaction initiation activities of each account. The workload data of the account is defined as a tuple and saved.

[0062] In transaction cycle E+1, the monitoring center (MC) continues its monitoring work, while the load balancer (LB) obtains the workload data recorded by the monitoring center (MC) in two consecutive transaction cycles E-1 and E from the database, and converts the activities of the shards and accounts in this transaction cycle into the initial state of reinforcement learning, and the fully trained model gives the shard position with the highest Q value for each account. Whenever the prediction of an account is completed, the shard manager (RM) will initiate a migration transaction for each account that needs to be migrated based on the prediction results. This transaction will be routed to the shard to which the account currently belongs, waiting for the validator to reach a consensus. If the message of this account migration is agreed upon within the shard, then the validator node responsible for packaging the transaction will initiate a special cross-shard transaction to the new partition of this account, which contains the account information to be migrated and the list of pending transactions related to the account. The account information will then be deleted from the original shard. Assuming that this transaction is successful, the validator of the destination shard will broadcast the new account information within the shard so that all validators can start processing transactions for the new account. All pending transactions of the migrated account will be added to the head of the destination shard transaction waiting queue for execution. During the E+1 cycle, RM will ensure that all account migration work is completed.

[0063] A. Markov decision process

[0064] Before using reinforcement learning to build a model for the blockchain sharding network environment, this embodiment must formulate the problem as a Markov decision process (MDP). t+1 Depends only on the current state s t and action a t , independent of the previous sequence of events.

[0065] (1) Status definition

[0066] At the load balancing layer, the monitoring center (MC) captures the necessary information such as the number of transactions and the number of cross-shard transactions (CST) of all shards and accounts as the state s at time t t.

[0067] (2) Impact of actions

[0068] Action a t (i.e., reassign the account to a new shard in the next cycle or keep it unchanged) decisively t Transformed into t+1 .

[0069] (3) Rewards

[0070] The performance gain of the blockchain after sharding depends only on the current state and action a t , and does not depend on past blockchain states or account migration actions.

[0071] In summary, the dynamic sharding workload balancing problem meets the requirements of MDP, and a reinforcement learning model can be built to predict the optimal solution.

[0072] B. Deep reinforcement learning model, specifically for blockchain simulation state

[0073] b1 Environment and Agent

[0074] The blockchain network including all shards constitutes the environment of reinforcement learning. The Monitor Center (MC) serves as a bridge for the agent to obtain environmental knowledge. It will capture the ongoing transaction activities in the network and calculate statistical information such as workload and cross-shard transaction ratio. The Load Balancer (LB) as an intelligent agent will query the knowledge of the environment from the MC, take actions, and obtain feedback from the environment through the MC.

[0075] b2 Action Space

[0076] The action of deep reinforcement learning is defined as the change in the shard position of a blockchain account after a transaction cycle. Assume that at transaction cycle t, an industrial blockchain network has shards S = {S1, S2, .. S i ,,…}. At the same time, a shard has A={A1,A2,..A j ,...} account set, the S i is the i-th shard, A j is the jth account. The sharding middleware will make the most appropriate sharding location based on the transaction characteristics of the account. That is, an account can keep the original shard unchanged, or be reallocated to a new shard.

[0077] b3 status

[0078] In the deep reinforcement learning model of this embodiment, the state of the current environment is defined as a tuple, in which the account state and shard state in the blockchain network are mainly focused.

[0079] 1) Status information related to the account, including the account ID, the number of transactions submitted in a certain transaction cycle, the number of transactions that have reached consensus, the proportion of cross-shard transactions in the number of submitted transactions, the most frequently appearing destination shard ID in cross-shard transactions, and the shard ID to which the account belongs.

[0080] 2) Status information related to the shard, including the shard ID, the total number of transactions processed by the shard in a transaction cycle, and the proportion of cross-shard transactions in the total number of transactions processed.

[0081] b4 Reward function

[0082] The goal of the reward function is to find the shard that is most suitable for hot accounts in the next cycle. However, in the load balancing scenario of this embodiment, there is a trade-off between workload uniformity and low CST ratio. Therefore, the reward function of the deep reinforcement learning model consists of two parts: cross-shard transaction (CST) reward and load uniformity reward.

[0083] CST reward R1: It is designed to punish the increase of CST ratio in all shards and reward the decrease of CST ratio. The calculation of CST reward is as follows: after the redistribution of account A, the workload data (the number of transactions in the target shard) will be updated according to formula (1); based on the updated workload data, the new cross-shard transaction ratio is calculated according to formula (2); the penalty method is applied, and the CST reward R1 is calculated according to formula (3);

[0084] M′(S i )=M(S i )+M(A) (1)

[0085] P′(S i )=(P(S i )×M(S i )+P(A,S i )×M(A)) / M′(S i ) (2)

[0086] R1=-(P′(S i )-P(S i )) (3)

[0087] Among them, M′(S i ) indicates shard S i The workload data updated in a certain period, M(S i ) indicates shard S iThe workload data in a certain period, M(A) is the workload data of account A in a certain period, P′(S i ) is the new cross-shard transaction ratio, P(S i ) is the fragment S i The proportion of cross-shard transactions processed in a certain period, M(S i ) is the fragment S i The workload data in a certain period is the total number of transactions processed, P(A, S i ) is the fragment S i The percentage of the total number of cross-shard transactions sent by account A in a certain period.

[0088] Load uniformity reward R2: It measures the balance reward to encourage load balancing among all shards, and uses standard deviation to measure this. Specifically, the standard deviation of the workload of the current cycle is calculated based on formula (4), and then the standard deviation of the new cycle is calculated in formula (5), and the load uniformity reward R2 is obtained in formula (6) through the penalty method;

[0089]

[0090] R2=-(D′-D); (6)

[0091] Where N is the total number of shards, M(S i ) is the fragment S i Workload data in the current cycle (i.e., the total number of transactions successfully packaged by the shard in the cycle), M'(S i ) is the fragment S i Workload data in the new cycle, is the mean value of workload data in the current cycle, is the mean of the workload data in the new cycle, D is the standard deviation of transactions in all shards of the blockchain in the current cycle, and D' is the standard deviation of transactions in all shards of the blockchain in the new cycle.

[0092] Finally, the reward function R of the deep reinforcement learning model is constructed based on the CST reward R1 and the load uniformity reward R2:

[0093] R=w1×R1+w2×R2; (7)

[0094] Among them, w1 and w2 are weight coefficients for balancing the cross-shard transaction ratio and workload balance, respectively.

[0095] C. Q function

[0096] In the above content, the load balancing problem is expressed as a Markov decision process (MDP). However, in the blockchain sharding scenario, the state in question is high-dimensional, and the traditional reinforcement learning model has limited efficiency in retrieving state-action pairs. Therefore, a deep neural network is used to approximate the Q function, and DQN is used instead of the traditional lookup table search. D3QN is an existing DQN variant based on Double DQN and Dueling DQN, which absorbs the advantages of each to improve the efficiency of DQN. Based on the principles of its ideas, the reinforcement Q function used in the deep reinforcement learning model in this embodiment is constructed.

[0097] First, we use the idea of ​​Double DQN to calculate two parallel streams in Dueling DQN: one is used to estimate the state value function Q v (s t ), and another one for estimating the action advantage function Q a (s t ,a t ). Q v (s t ) function can be written as shown in (8):

[0098]

[0099] Among them, s t is the state, a t For action. t Represents in state s t Take action a t The immediate reward is also the representative short-term reward of the action; γ is the long-term reward weight coefficient between 0 and 1, the higher the value, the more important the long-term reward; Represents first selecting the next state s t+1 The action a that gets the maximum Q value t+1 , and then calculate the highest long-term reward.

[0100] Q a (s t ,a t ) function can be written as shown in (9):

[0101]

[0102] Among them, s t is the state, a t For action. t Represents in state s t Take action a t The immediate reward is also the representative short-term reward of the action; γ is the long-term reward weight coefficient between 0 and 1, the higher the value, the more important the long-term reward; Represents first selecting the next state s t+1 The action a that gets the maximum advantage t+1 , and then calculate the highest long-term advantage value.

[0103] Finally, according to the Q calculated above v (s t ,a t ) and Q a (s t ,a t ), and then we can get the Q function used in this model by substituting it into the expression of Dueling DQN, as shown in (10):

[0104]

[0105] Among them, Q(s t ,a t ) represents the state s t Take action a t Q value; Q v (s t ) represents the state s t The value function under Q a (s t ,a t ) represents the state s t Take action a t Advantage functions over taking other actions; Represents in state s t The average value of the advantage function of taking all possible actions under state s; A represents the set of action spaces of the deep reinforcement model, |A| represents the size of the action space (that is, the number of all possible actions), and a'∈A represents the t Under each possible action, that is, A is a large set, a' is an element of the set, and |A| is the size of the set.

[0106] A, B, and C are used to construct a deep reinforcement learning model, and then the constructed model is trained. The specific training process is as follows:

[0107] The accuracy of the reinforcement learning model depends on the full exploration of the environment by the agent during model training, but the transaction data collected from real industrial equipment is limited, so direct application to the model training process will cause large errors. In order to solve this problem, this embodiment collects and cleans the blockchain transaction records of the data upload device in the real industrial blockchain network, and uses the long short-term memory network (LSTM) as a recurrent neural network to generate simulated accounts and their long-term account activities. These virtual data will operate in an independent virtual blockchain environment. The deep reinforcement learning model (DRL model) learns and makes decisions through the activities of these virtual accounts to determine the account migration strategy. First, the account transaction record data from the real environment is cleaned and standardized, and grouped according to the shards to determine the CST ratio standard. Next, the target shard of the transaction is defined according to all the shards involved in the transaction history of the account in the real blockchain environment. Finally, we input the shard and transaction initiation time of a specific account into the LSTM model, and the model will predict the behavior of the account in the next cycle, and continue this prediction process in subsequent cycles until sufficient account transaction data is obtained to complete the full training of the model.

[0108] E. Testing

[0109] In order to prove that this embodiment can evenly distribute the blockchain transactions sent to the network to each shard, so that the load is as balanced as possible, this embodiment performs two types of stress tests. In the first test, the consensus cycle is maintained at 10 blockchains, and the number of transactions sent by the accounts is equal and random to simulate the situation in most actual production scenarios where the data volume is large but the peak conditions are less. In the second test, a typical network with 32 shards and 10 epochs is maintained, and some specific accounts in each shard will initiate transactions at 1.25 times, 1.5 times, and 1.75 times the standard sending rate to simulate the peak pressure scenario, and record the load changes of the system, while other accounts trade at the normal rate. The results are shown in Figure 2 and Figure 3 In the first set of tests, although the load variance fluctuated under different numbers of shards, the workload case with model optimization had a lower workload variance than the random shard case. In the second set of tests, when the load of the entire network became higher, the workload of some shards increased significantly, making the overall load variance of the system larger. However, with the progress of reinforcement learning, accounts that frequently send transactions will adjust in time according to their transaction characteristics and the load of other shards, so that the final network load will gradually decrease and remain stable.

[0110] This embodiment models the workload balancing problem of industrial IoT blockchain shards as a Markov decision process (MDP) and designs a deep reinforcement learning load balancing algorithm based on the deep double Q network (D3QN), which not only considers the workload of accounts and shards, but also integrates the cross-shard transaction characteristics of accounts. This paper also provides LSTM to assist the model training process, thereby improving the accuracy of the load balancing middleware and improving the deployment efficiency in practical applications.

[0111] Secondly, the load balancing algorithm is encapsulated as a middleware of an industrial blockchain network. By tracking the sharding and account activity history in each transaction cycle of the network, account location decisions that optimize system performance can be made to meet the user requirements for high data throughput and low transaction confirmation latency in industrial production scenarios.

[0112] Finally, our solution was fully simulated in the constructed experimental environment using real data collected from an actual industrial IoT environment, and the results showed that this embodiment is effective.

[0113] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A blockchain load balancing middleware system based on deep reinforcement learning, characterized in that: By comprehensively learning the transaction tracks of accounts and shards in the blockchain network and the cross-shard transaction ratio, the optimal shard position of the account in the next transaction cycle is determined; The working process of the blockchain sharding load balancing middleware is as follows: The consortium blockchain transaction monitoring component continuously monitors blockchain network activities and saves them to the in-memory database; The shard load balancer obtains the network's account and shard workload data in two consecutive transaction cycles from the database, and converts the account and shard workload data in the transaction cycle into the state and action of reinforcement learning; Based on the state and action, the load balancer uses a deep reinforcement learning model to calculate the action with the largest long-term reward as the final prediction result of the account belonging to the new shard; For each account that needs to be moved to the new shard, a migration transaction is initiated for the account of the new shard based on the final prediction result. The migration transaction will be routed to the shard to which the account currently belongs, waiting for the validators to reach a consensus; The validator node responsible for packaging the migration transaction will initiate a new cross-shard transaction to the new shard of the account. The new cross-shard transaction contains the account information waiting to be migrated and a list of pending transactions related to the account. The account information will then be deleted from the original shard, and the validator of the new shard will broadcast the new account information within the shard so that all validators can process transactions for the new account.

2. The blockchain load balancing method based on deep reinforcement learning according to claim 1 is characterized in that: In the final prediction result of the action with the largest long-term reward calculated by the deep reinforcement learning model as the account belonging to the new shard, the basic parameters of the deep reinforcement learning model are as follows: Action space: The action of deep reinforcement learning is defined as the change in the shard position of a blockchain account after a transaction cycle. Blockchain network sets shard S = {S1, S2, ..S i ,,…}, each shard has A={A1,A2,..A j ,...}, the account set S i is the i-th shard, A j is the jth account; Status: 1) Status information related to the account, including the account ID, the number of transactions submitted in a certain cycle, the number of transactions that have reached consensus, the proportion of cross-shard transactions in the number of submitted transactions, the most frequently appearing destination shard ID in cross-shard transactions, and the shard ID to which the account belongs; 2) Status information related to the shard, including the shard ID, the total number of transactions processed by the shard in a certain period, and the proportion of cross-shard transactions in the total number of transactions processed.

3. The blockchain load balancing method based on deep reinforcement learning according to claim 2 is characterized in that: The basic parameters of the deep reinforcement learning model also include a reward function, which includes a reward R1 for the cross-shard transaction ratio and a reward R2 for the load uniformity of the workload data, where: The calculation of CST reward R1 is as follows: After account A is reallocated, the workload data will be updated according to formula (1); based on the updated workload data, the new cross-shard transaction ratio is calculated according to formula (2); the penalty method is applied and the CST reward R1 is calculated according to formula (3); M′(S i )=M(S i )+M(A) (1) P′(S i )=(P(S i )×M(S i )+P(A,S i )×M(A)) / M′(S i ) (2) R1=-(P′(S i )-P(S i )) (3) Among them, M′(S i ) indicates shard S i The workload data updated in a certain period, M(S i ) indicates shard S i The workload data in a certain period, M(A) is the workload data of account A in a certain period, P′(S i ) is the new cross-shard transaction ratio, P(S i ) is the fragment S i The proportion of cross-shard transactions processed in a certain period, M(S i ) is the fragment S i The workload data in a certain period is the total number of transactions processed, P(A, S i ) is the fragment S i The percentage of the total number of cross-shard transactions sent by account A in a certain period.

4. The blockchain load balancing method based on deep reinforcement learning according to claim 3 is characterized in that: The calculation of the load evenness bonus R2 is as follows: The workload standard deviation of the current cycle is calculated based on formula (4), and then the standard deviation of the new cycle is calculated in formula (5), and the load uniformity reward R2 is obtained in formula (6) through the penalty method; R2=-(D'-D); (6) Where N is the total number of shards, M(S i ) is the fragment S i Workload data in the current cycle (i.e., the total number of transactions successfully packaged by the shard in the cycle), M'(S i ) is the fragment S i Workload data in the new cycle, is the mean value of workload data in the current cycle, is the mean of the workload data in the new cycle, D is the standard deviation of transactions in all shards of the blockchain in the current cycle, and D' is the standard deviation of transactions in all shards of the blockchain in the new cycle.

5. The blockchain load balancing method based on deep reinforcement learning according to claim 4 is characterized in that: The reward function R of the deep reinforcement learning model is constructed based on the CST reward R1 and the load uniformity reward R2: R=w1×R1+w2×R2 Among them, w1 and w2 are weight coefficients for balancing the cross-shard transaction ratio and workload balance, respectively.

6. The blockchain load balancing method based on deep reinforcement learning according to claim 1 is characterized in that: The Q function of the deep reinforcement learning model is specifically: Among them, Q(s t ,a t ) represents the state s t Take action a t Q value; Q v (s t ) represents the state s t The value function under Q a (s t ,a t ) represents the state s t Take action a t Advantage functions over taking other actions; Represents in state s t The average value of the advantage function of taking all possible actions under state s, A represents the set of action spaces of the deep reinforcement model, |A| represents the size of the action space, and a'∈A represents the state s. t List every possible action you can take.

7. The blockchain load balancing method based on deep reinforcement learning according to claim 1 is characterized in that: The training process of the deep reinforcement learning model is as follows: The blockchain transaction records of data upload devices in the real industrial blockchain network were collected and cleaned for cleaning and standardization, and grouped according to shards to determine the cross-shard transaction ratio standard; Define the target shard set of the transaction based on all shards involved in the account's transaction history in the real blockchain environment; The shard and transaction initiation time of the set account are input into the long short-term memory model as raw data to predict the behavior of the set account in the next transaction cycle. The obtained prediction results are verified to ensure that they are consistent with the configuration of the account and shard in the deep reinforcement learning model. If passed, the raw data and the prediction results are combined into an extended data set and provided to the intelligent agent; Based on the expanded environmental data, the agent continuously explores the environment, and continuously optimizes the action strategy with the best Q value in different states by combining the environmental feedback brought by the strategy and the enhanced Q function until the deep reinforcement learning model training is completed; The deep reinforcement learning model training that has passed the verification test is applied to make decisions in the actual blockchain network.

8. A blockchain load balancing system based on deep reinforcement learning, characterized in that: By comprehensively learning the transaction tracks of accounts and shards in the blockchain network and the cross-shard transaction ratio, the optimal shard position of the account in the next transaction cycle is determined; The working process of the blockchain sharding load balancing middleware is as follows: The consortium blockchain transaction monitoring component continuously monitors blockchain network activities and saves them to the in-memory database; The shard load balancer obtains the network's account and shard workload data in two consecutive transaction cycles from the database, and converts the account and shard workload data in the transaction cycle into the state and action of reinforcement learning; Based on the state and action, the load balancer uses a deep reinforcement learning model to calculate the action with the largest long-term reward as the final prediction result of the account belonging to the new shard; For each account that needs to be moved to the new shard, a migration transaction is initiated for the account of the new shard based on the final prediction result. The migration transaction will be routed to the shard to which the account currently belongs, waiting for the validators to reach a consensus; The validator node responsible for packaging the migration transaction will initiate a new cross-shard transaction to the new shard of the account. The new cross-shard transaction contains the account information waiting to be migrated and a list of pending transactions related to the account. The account information will then be deleted from the original shard, and the validator of the new shard will broadcast the new account information within the shard so that all validators can process transactions for the new account.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the blockchain load balancing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the blockchain load balancing method as described in any one of claims 1 to 7.

Images