Blockchain performance optimization method based on multi-agent deep reinforcement learning

Abstract

The application discloses a kind of based on multi-agent deep reinforcement learning's block chain performance optimization method, steps include: the every action of influence block chain system is regarded as an agent, and Markov decision process model is established to block chain performance optimization problem;Neural network and environment network parameter are initialized;Based on Markov decision process model, MDRL is used to carry out multiple iteration training, and the optimal block generation strategy is generated;In the multiple iteration training using MDRL, the state information of each part of agent of block chain and the action information executed in last iteration are obtained to solve optimal Q value each time iteration, and the sample information of each iteration is stored in experience database, and loss function is calculated to update Q network from experience database selected sample.This application has the advantages of simple implementation method, high optimization efficiency and data security reliability, good optimization effect and scalable performance.

Description

Technical Field

[0001] This invention relates to the field of blockchain technology, and in particular to a blockchain performance optimization method based on multi-agent deep reinforcement learning. Background Technology

[0002] Blockchain technology can solve the problem of reaching agreements among untrusted parties. It features decentralization, persistence, anonymity, and auditability, and can be applied to various fields such as supply chains, the Internet of Things (IoT), and the Internet of Vehicles (IoV). Initially used as a peer-to-peer (P2P) ledger for Bitcoin economic transactions, blockchain ensures data security and efficiency by supporting anonymous and trusted transactions and eliminating intermediaries. Despite the significant benefits of blockchain technology, in practical applications, scalability is a pressing and unavoidable problem that blockchain must solve. Furthermore, the throughput of current blockchain platforms is relatively low, sometimes less than 1% of that of traditional databases, and traditional blockchain systems struggle to provide the scalability required to meet high transaction throughput demands.

[0003] Traditional performance optimization methods are difficult to apply directly to the performance issues of blockchain because: (1) the network is constantly changing, and various influencing factors interact and combine, resulting in a complex effect that makes it difficult to find a universal solution. (2) different open-source blockchain systems have different architectures, making it difficult to establish a unified optimization model. (3) the security of blockchain systems is not easy to quantify, and malicious nodes and malicious attacks may exist in the system. (4) it is difficult to make accurate real-time decisions on variable attributes, thus traditional optimization schemes are difficult to effectively improve the data security and efficiency performance of blockchain.

[0004] In reinforcement learning algorithms, when an agent performs a task, it first interacts with the environment, generating new states. Simultaneously, the environment provides rewards, and this process repeats, with the agent and environment continuously interacting to generate more new data. The core and training objective of reinforcement learning is to select a suitable policy that maximizes the sum of rewards at the end of each iteration. Deep reinforcement learning (DRL) combines deep learning and reinforcement learning to achieve end-to-end processing from perception to action. It stores the data obtained by the system exploring the environment and then randomly samples samples to update the parameters of the deep neural network.

[0005] To address the performance issues of data security and efficiency in blockchain, some practitioners have proposed combining deep reinforcement learning methods to optimize blockchain sharding systems. However, such solutions typically treat the blockchain sharding system as a whole, directly using deep reinforcement learning to continuously select the optimal behavior, ultimately forming a sharding strategy for the entire system. But blockchain involves numerous actions that influence the system, and changes in these factors can have combined effects. Deep reinforcement learning on the sharding system as a whole cannot learn about the individual actions affecting the system, nor can it consider the interrelationships between these actions. This results in decisions that are often not optimal, and thus, actual performance optimization still needs improvement. Summary of the Invention

[0006] The technical problem to be solved by this invention is: to address the technical problems existing in the prior art, this invention provides a blockchain performance optimization method based on multi-agent deep reinforcement learning that is simple to implement, highly efficient in optimization, and secure and reliable in data.

[0007] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0008] A blockchain performance optimization method based on multi-agent deep reinforcement learning, comprising the following steps:

[0009] Treating each action that affects the blockchain system as an intelligent agent, a Markov decision process model is established for the blockchain performance optimization problem.

[0010] Initialize the neural network and environmental network parameters;

[0011] Based on the Markov decision process model, MDRL is used for multiple iterative training to generate the optimal block generation strategy.

[0012] In the multi-agent deep reinforcement learning (MDRL) training process, the state information of each agent in the blockchain and the action information executed in the previous iteration are obtained in each iteration to solve for the optimal Q value in order to select the action of each agent. The sample information of each iteration is stored in the database, and the loss function is calculated from the sample in the database to update the Q network.

[0013] Furthermore, in establishing a Markov decision process model for the blockchain performance optimization problem, the state space S of the blockchain system is defined as follows: when the number of iterations is t... (t) This includes transaction size χ, share distribution γ, node geographical location x, and node computing power c = {c k} and the data transmission rate R, i.e., S, between each pair of nodes. ( w ) = [χ, γ, x, c, R] (t) Define the action space A when the number of iterations is t. (t) Includes block producer a, consensus algorithm δ, and block size S. B and block spacing T I A (t) =[a, δ, S] B T I ] (t) .

[0014] Furthermore, in the multiple iterations of training using MDRL, the action policy with the largest Q-value is selected and output to the environment for execution in each iteration. That is, each iteration must make a decision to solve the following problem:

[0015]

[0016] Where Q(S,A) is the action-state function, S is the state of the agent, and A is the action;

[0017] The constraint condition is set as follows: C1: G(γ)≤η s G(λ)≤η l

[0018] C2:T F，δ ≤ω×T I δ=0,1,2

[0019] C3: f≤F δ ,δ=0,1,2

[0020] Where G(γ) is the Gini coefficient of block producer equity, G(λ) is the Gini coefficient of block producer geographical location, ηs, η l The thresholds for decentralization are T for equity distribution and geographic location distribution, respectively. F,δ For TTF delay, T F,δ =T I +T C,δ ,T C, For consensus latency, i.e., the time cost for validators to verify generated blocks, δ represents the different consensus protocol identifiers, and T I ω represents the number of intervals; F 2 =0 indicates the maximum tolerable number of malicious validators; f is the number of malicious validators.

[0021] Furthermore, the formula for calculating the Gini coefficient G(Υ) of block producer equity is as follows:

[0022]

[0023] in, For block producer b i The equity ratio, where K is the number of block producers;

[0024] The formula for calculating the Gini coefficient G(λ) of the block producer's geographical location is as follows:

[0025]

[0026] Wherein, the density set λ={λ(x)}, x∈Ξ, and the block producers are scattered in the region Ξ.

[0027] Furthermore, in establishing a Markov decision process model for the blockchain performance optimization problem, the reward function is defined as:

[0028]

[0029] Furthermore, the step of performing multiple iterative training using MDRL includes:

[0030] In each iteration, the block producer node receives new computational tasks and stores them in the data cache, observing the state of each part of the blockchain system's intelligent agents. And integrate them into a global state S (t) t is the iteration number, and n is the agent index;

[0031] Input the state information of each agent Action information from the last iteration The Q-value of each agent is solved using the QMIX (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning) algorithm, and the global state S is used. (t) Estimate the Q-value of the combined action;

[0032] Choose the action strategy with the highest Q value (max). A Q(S,A) outputs to the environment to execute action A. (t) ,Right now

[0033] Select block producers and consensus algorithms, adjust block size and block interval, and receive corresponding reward values ​​from the environment.

[0034] The global state S is changed during each iteration. (t) Action Information A (t) ,award The global state S up to and the next iteration (t+1)The data is stored in the database. Samples are extracted from the experience database to calculate the Q-value and loss function. The Q-network is then updated using the loss function.

[0035] The process is repeated multiple times until the optimal block generation strategy is obtained after training is completed.

[0036] Furthermore, specifically, samples i are randomly drawn from the empirical database according to y. i =R ( w ) +γmax A′ Calculate the sample Q value according to L(θ)=[(y i -Q(S (i) ,A′;θ)) 2 Calculate the loss function, where θ is learned by sampling a batch of b transitions from the replay memory and minimizing the squared TD error, and b is the number of samples sampled from the empirical memory. Let A' be the parameters of the target network in the DQN, which are periodically copied from θ, and let S' be the state. (i+1) The actions taken State S (i+1) The maximum value obtained under certain circumstances.

[0037] Furthermore, the optimal block generation strategy includes the selection and / or adjustment of block producers, consensus algorithms, block size, and block interval.

[0038] Furthermore, step S01 also includes loading historical state transition curves and Q-value estimates into the experience memory D, and pre-training the DNN using input pairs (S,A) and the corresponding estimated Q(S,A), where S represents the state and A represents the action.

[0039] A computer device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0040] Compared with the prior art, the advantages of the present invention are as follows:

[0041] 1. This invention treats each action affecting the blockchain system as an agent, establishes a Markov decision process model for the blockchain performance optimization problem, initializes the neural network and environmental network parameters, and performs multiple iterations of training based on the multi-agent deep reinforcement learning model (MDRL) to generate the optimal block generation strategy. This approach can fully learn the numerous actions that affect the blockchain system, effectively improving the accuracy of the optimal block generation strategy and thus enhancing optimization performance. Furthermore, based on the MDRL model, multiple agents interact with the environment to establish different relationships, eliminating the need for data labeling and reducing reliance on manual work. This avoids the problem of manually defining a huge value space, effectively improving optimization efficiency.

[0042] 2. This invention further quantifies the performance of the blockchain system in terms of scalability, decentralization, latency, and security. During the multiple iterations of training using MDRL, by setting constraints on decentralization, latency, and security, the scalability of the underlying blockchain can be improved without affecting the system's decentralization, latency, and security. This effectively reduces security risks while improving the performance of the blockchain system.

[0043] 3. The present invention further defines a reward function to evaluate the security and reliability of block producers. When incoming data does not meet the conditions defined in the reward function, it is determined to be data with potential security risks and is discarded to prevent the blockchain system from being maliciously attacked. This can effectively avoid malicious attacks and further improve data security and reliability. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structural principle of a multi-agent deep reinforcement learning system.

[0045] Figure 2 This is a schematic diagram illustrating the implementation principle of the QMIX algorithm.

[0046] Figure 3 This is a detailed implementation flowchart of the blockchain performance optimization method based on multi-agent deep reinforcement learning in this embodiment. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0048] like Figure 1As shown, based on Deep Reinforcement Learning (DRL), when multiple agents interact with the environment simultaneously, the entire system becomes a multi-agent system. In this system, all agents simultaneously select and execute their respective actions based on the current environmental state (or observations). The combined action resulting from these individual actions influences the transition and update of the environmental state and determines the reward feedback received by each agent. Each agent still follows the goal of reinforcement learning, namely, maximizing the cumulative reward. In this case, the change in the global state of the environment is the combined action of all agents.

[0049] This invention applies multi-agent deep reinforcement learning to optimize blockchain performance, thereby addressing the performance issues of data security and efficiency in blockchain. The specific steps of the blockchain performance optimization method based on multi-agent deep reinforcement learning in this embodiment include:

[0050] Step S01. Treat each action affecting the blockchain system as an intelligent agent and establish a Markov decision process model for the blockchain performance optimization problem;

[0051] Step S02. Initialize the neural network and environmental network parameters;

[0052] Step S03. Based on the Markov decision process model, MDRL is used for multiple iterative training to generate the optimal block generation strategy;

[0053] In step S03, MDRL is used for multiple iterations of training. During each iteration, the state information of each part of the blockchain agent and the action information executed in the previous iteration are obtained to solve for the optimal Q value to select the action of each agent. The sample information of each iteration is stored in the database, and the loss function is calculated from the database to update the Q network.

[0054] This embodiment treats each action affecting the blockchain system as an agent, establishes a Markov decision process model for the blockchain performance optimization problem, initializes the neural network and environmental network parameters, and performs multiple iterations of training based on the multi-agent deep reinforcement learning model (MDRL) to generate the optimal block generation strategy. This approach can fully learn that there are a large number of actions in the blockchain that affect the blockchain system, effectively improving the accuracy of the optimal block generation strategy and thus improving optimization performance. Furthermore, based on the MDRL model, multiple agents interact with the environment to establish different relationships, eliminating the need for data labeling and reducing reliance on manual work. This avoids the problem of traditionally requiring the manual definition of a huge value space, effectively improving optimization efficiency.

[0055] In this embodiment, a Markov decision process model is established for the blockchain performance optimization problem. Specifically, the state space S of the blockchain system is defined when the number of iterations is t. (t) This includes transaction size χ, share distribution γ, node geographical location x, and node computing power c = {c k} and the data transmission rate R, i.e., S, between each pair of nodes. (t) = [χ, γ, x, c, R] (t) Define the action space A when the number of iterations is t. (t) Includes block producer a, consensus algorithm δ, and block size S. B and block spacing T I A (t) =[a, δ, S] B T I ] (t) .

[0056] This embodiment further considers the scalability of the blockchain system. Scalability can be evaluated through transaction throughput, which directly depends on two performance-related parameters: one is the block size, i.e., the number of bytes that can be contained in each block, which determines how many transactions can be included in a block; the other is the block interval, i.e., the average time required for a block producer to produce a new block, which captures the block release rate. Considering the above two factors, the transaction throughput Ω can be expressed as:

[0057]

[0058] Among them, S B T represents the block size (the number of bytes each block can contain). I χ represents the block interval (the average time required for a block producer to produce a new block), and χ represents the average size of a transaction. Block size and throughput are positively correlated, while block range and throughput are negatively correlated. Therefore, it is necessary to configure reasonable block size and block range to maximize throughput and minimize latency.

[0059] To achieve scalability in the blockchain system, the system is further evaluated in terms of decentralization, latency, and security:

[0060] (1) Decentralization

[0061] To measure the decentralization of a blockchain system, this embodiment introduces the Gini coefficient to measure the decentralization of block producers, and constructs the Gini coefficient G(γ) of block producer equity and the Gini coefficient G(λ) of block producer geographical location, taking into account two typical factors: equity distribution and geographical location.

[0062] To describe the decentralized equity distribution, the Gini coefficient G(γ) of block producer equity in this embodiment is specifically calculated as follows:

[0063]

[0064] in, For block producer b i The equity ratio, where K is the number of block producers.

[0065] The formula for calculating the Gini coefficient G(λ) of the block producer's geographical location is as follows:

[0066]

[0067] Wherein, the density set λ={λ(x)}, x∈Ξ, and the block producers are scattered in the region Ξ.

[0068] To ensure the dispersion of block producers in terms of equity allocation and geographical location, the following constraints need to be met:

[0069] G(γ)≤η s G(λ)≤η l (4)

[0070] Where η s η l ∈[0,1] represents the decentralized threshold-related equity distribution and geographic location distribution.

[0071] The decentralization of the blockchain system can be achieved by using the above constraints.

[0072] (2)TTF delay (The time for the transactions to be finalized)

[0073] The latency of a blockchain system is assessed using TTF (Transaction Time To Failure). The TTF of a transaction includes the block generation time (i.e., block interval) and the block verification time, as shown in the following formula:

[0074] T F,δ =T I +T C,δ (5)

[0075] Among them, T C,δ Consensus latency, i.e., the time cost for validators to verify generated blocks, depends on the consensus algorithm used. In this embodiment, δ = 0, 1, 2 represents different consensus protocols, specifically PBFT, Quorum, and Raft, respectively.

[0076] In a network, users typically expect to receive the final result of a transaction within a short period of time. To meet the network's latency requirements, assume that a block is published and validated over multiple consecutive block intervals, i.e., ω (ω>1) block intervals:

[0077] TF,δ ≤ω×T I (6)

[0078] In other words, by satisfying the above constraints, the blockchain system can meet the latency requirements.

[0079] (3) Security

[0080] To ensure the security of the consensus algorithm used in the blockchain system, the number of malicious validators f should be subject to the following constraints:

[0081] f≤F δ ,δ=0,1,2 (7)

[0082] F 2 =0 indicates the maximum number of malicious validators that can be tolerated.

[0083] This embodiment quantifies the performance of the blockchain system in terms of scalability, decentralization, latency, and security in the manner described above. During multiple iterative training using MDRL, by setting the aforementioned constraints on decentralization, latency, and security, the scalability of the underlying blockchain can be improved without affecting the system's decentralization, latency, and security. This effectively reduces security risks while improving the performance of the blockchain system.

[0084] The value function can be constructed using the above constraints. Specifically, in this embodiment, MDRL is used for multiple iterative training. In each iteration, the action policy with the largest Q value is selected and output to the environment for execution. That is, each iteration must make a decision to solve the following problem:

[0085]

[0086] Where Q(S,A) is the action-state function, S is the state of the agent, and A is the action;

[0087] The constraint condition is set as follows: C1: G(γ)≤η s G(λ)≤η l

[0088] C2:T F，δ ≤ω×T I δ=0,1,2

[0089] C3: f≤F δ ,δ=0,1,2

[0090] Where G(γ) is the Gini coefficient of block producer equity, G(λ) is the Gini coefficient of block producer geographical location, ηs, η l The thresholds for decentralization are T for equity distribution and geographic location distribution, respectively.F,δ For TTF delay, T F,δ =T I +T C,δ ,T C, For consensus latency, i.e., the time cost for validators to verify generated blocks, δ represents the different consensus protocol identifiers, and T I ω represents the number of intervals; F 2 =0 indicates the maximum tolerable number of malicious validators; f is the number of malicious validators.

[0091] Using the above constraints, in this embodiment, the reward function is specifically defined as follows in establishing a Markov decision process model for the blockchain performance optimization problem:

[0092]

[0093] This embodiment evaluates the security and reliability of block producers by using the reward function defined above. When incoming data does not meet the conditions defined in the reward function, it is judged as data with potential security risks and discarded to prevent the blockchain system from being maliciously attacked. This can effectively avoid malicious attacks and further improve data security and reliability.

[0094] In MDRL, the DRL model is applied to a multi-agent environment to find the optimal blockchain architecture by weighing trade-offs among multiple agents. A multi-agent system is formed when multiple agents interact with the environment. During the interaction, other agents besides the current agent are considered factors in the environment and interact with the environment during policy updates.

[0095] This embodiment establishes a Markov decision process model for the blockchain performance optimization problem, which consists of the following four parts:

[0096] (1) State (state space)

[0097] Let the state space of the blockchain system be defined as follows when the number of iterations is t (t = 1, 2, ...): transaction size χ, share distribution γ, node geographical location x, and node computing power c = {c k The data transmission rate R between each pair of nodes is R = {R i,j The union of} is denoted as:

[0098] S (t) = [χ, γ, x, c, R] (t)

[0099] (2) Action (Action Space)

[0100] To maximize throughput, several components of the blockchain system need to be adjusted to adapt to dynamic environments, including the block producer a, the consensus algorithm δ, and the block size S. B and block spacing T I The action space when the number of iterations is t is represented as:

[0101] A (t) =[a, δ, S] B T I ] (t)

[0102] (3) Reward (reward function)

[0103] Define the reward function as follows:

[0104]

[0105] If constraints C1-C3 cannot be met, it means that the adjusted blockchain system performs poorly in terms of decentralization, TTF, or security. Therefore, the reward in this case is set to 0 to avoid this ineffective situation.

[0106] (4) Value Function

[0107] While ensuring the decentralization, finality, and security of the blockchain system, the goal is to maximize transaction throughput, meaning that a decision must be made in each iteration to address the following issues:

[0108]

[0109] Constraints:

[0110] C1:G(γ)≤η s G(λ)≤η l

[0111] C2:T F，δ ≤ω×T I δ=0,1,2

[0112] C3: f≤F δ ,δ=0,1,2

[0113] Among them, the action state function The discount factor μ∈(0,1] reflects the trade-off between current rewards and future rewards.

[0114] Solving the above model involves calculating the optimal value function and selecting the optimal action in the decision-making process to maximize the cumulative reward. This embodiment specifically uses the QMIX algorithm from MDRL to solve the model. QMIX is an architecture composed of an agent network, a hybrid network, and a set of supernetworks. It employs a hybrid network module as the integrating QMIX function. n The function expression for generating global Q is introduced, and global information is added during training to assist in the process. A hybrid network is used to merge the local value functions of a single agent instead of simply adding them linearly.

[0115] Assuming global Q-value and local Q-value n The values ​​satisfy the following relationship: the action that maximizes the global Q-value is to maximize each local Q-value. n The combination of values ​​and corresponding actions, i.e.:

[0116]

[0117] QMIX network structure as follows Figure 2 As shown, each agent has a DRQN ​​(Deep Recurrent Q-Learning) network, which takes the individual's state as input, uses a recurrent neural network to retain and utilize historical information, and outputs the individual's local Q-value. n Value; local Q of all individuals n The input is a hybrid network module where the weights of each layer are generated using a hypernetwork and absolute value calculation. Absolute value calculation ensures that the weights are non-negative, thus maximizing local Q-factors. n The integration of values ​​satisfies the monotonicity constraint; by using the global state S through the hypernetwork to generate weights, global information can be used more fully and flexibly to estimate the Q-value of joint actions, which to some extent helps in the learning and convergence of the global Q-value.

[0118] This embodiment incorporates the idea of ​​DQN, randomly sampling a small batch of state transition samples from the experience pool, using the global Q-value as the target for iterative updates, and selecting the action of each agent based on the Q-value in each iteration. The loss function is specifically defined as follows:

[0119]

[0120] Where y = R + γmax A′ θ is learned by sampling a batch of b transitions from the replay memory and minimizing the squared TD error. The parameters of the target network in DQN are periodically copied from θ and kept constant over multiple iterations.

[0121] Based on the state space, action space, and reward function defined above, this embodiment initializes the parameters of the neural network and the environment network. Through multi-round interaction and training optimization between MDRL and the environment network, it finally outputs the optimal block size, block interval, and consensus algorithm to improve the scalability of the overall system.

[0122] like Figure 3 As shown, the detailed steps of blockchain performance optimization based on multi-agent deep reinforcement learning in this embodiment include:

[0123] Step 1: Treat each action that affects the blockchain system as an intelligent agent, define the state space, action space and reward function according to the above equations (1) to (9), and establish a Markov decision process model for the blockchain performance optimization problem.

[0124] Step 2: Initialize neural network and environment network parameters: Load historical state transition curves and Q-value estimates into the empirical memory D; pre-train the DNN (main Q-network) with input pairs (S,A) and corresponding estimated Q(S,A).

[0125] Step 2: Perform multiple iterative training using MDRL

[0126] Step 2.1. In each iteration, the block producer node receives new computation tasks and stores them in the data cache, observing the state of each part of the blockchain system's intelligent agents. And integrate them into a global state S (t) t is the iteration number, and n is the agent index;

[0127] Step 2.2. Input the state information of each agent. Action information from the last iteration Right now The Q-value of each agent is solved using the QMIX algorithm, and the global state S is utilized. (t) Estimate the Q-value of the joint action; select the action policy with the largest Q-value (max). A Q(S,A) outputs to the environment to execute action A. (t) ,Right now

[0128] Step 2.3. Select block producers and consensus algorithms, and adjust block size and block interval, with the environment providing corresponding reward values.

[0129] Step 2.4. Observation and S (t+1) In each iteration, the global state S is changed. (t) Action Information A (t) ,award The global state S up to and the next iteration (t+1)Store the data in the experience database, and extract samples from the experience database to calculate the Q value and loss function;

[0130] Step 2.5. Update the Q-network using the loss function;

[0131] Step 2.6. Iterate multiple times until training is complete, then select the A corresponding to the optimal Q(S,A). (t) Calculate the corresponding The optimal block generation strategy is obtained.

[0132] This embodiment treats each part of the blockchain system as an intelligent agent by introducing MDRL and combining it with the QMIX algorithm to solve for the optimal Q value. This enables the acquisition of global information to make dynamic decisions for each variable, allowing end-to-end learning of decentralized strategies under centralized settings and effectively utilizing additional state information to make decisions more accurate, thereby further improving the precision of blockchain performance optimization.

[0133] In step 2.4 above, sample i is randomly drawn from the empirical database according to y. i =R (i) +γmax A′ Calculate the sample Q value according to L(θ)=[(y i -Q(S (i) ,A′;θ)) 2 Calculate the loss function, where θ is learned by sampling a batch of b transitions from the replay memory and minimizing the squared TD error, and b is the number of samples sampled from the empirical memory. Let A' be the parameters of the target network in the DQN obtained by periodically copying from θ, and let S' be the state. (i+1) The actions taken State S (i+1) The maximum value obtained under certain circumstances.

[0134] The optimal block generation strategy in this embodiment specifically includes the selection and / or adjustment of block producers, consensus algorithms, block size, and block interval. Through modular blockchain, block producers, consensus algorithms, block size, and block interval can be selected / adjusted using DRL technology, which can effectively solve the scalability problem of blockchain systems while ensuring decentralization, latency, and security, thereby improving the applicability of blockchain.

[0135] The computer device of this embodiment includes a processor and a memory. The memory is used to store computer programs, and the processor is used to execute the computer programs to perform the methods described above.

[0136] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A blockchain performance optimization method based on multi-agent deep reinforcement learning, characterized by the following steps: include: Treating each action that affects the blockchain system as an intelligent agent, a Markov decision process model is established for the blockchain performance optimization problem. Initialize the neural network and environmental network parameters; Based on the Markov decision process model, MDRL is used for multiple iterative training to generate the optimal block generation strategy. In the multiple iterations of training using MDRL, the state information of each agent in the blockchain and the action information executed in the previous iteration are obtained in each iteration to solve for the optimal Q value to select the action of each agent. The sample information of each iteration is stored in the experience database, and the loss function is calculated from the sample in the experience database to update the Q network.

2. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In establishing a Markov decision process model for the blockchain performance optimization problem, the state space of the blockchain system is defined when the number of iterations is t. Including transaction size Shareholding distribution Geographical location of nodes Node computing power and the data transmission rate of the link between each pair of nodes. ,Right now When the number of iterations is defined as t, the action space includes block producer a and consensus algorithm. Block size and block spacing ,Right now .

3. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to claim 2, characterized in that, In MDRL-based iterative training, the action policy with the largest Q-value is selected and output to the environment for execution in each iteration. In other words, each iteration requires making a decision to solve the following problem: in, Let S be the action-state function, where S is the agent's state and A is the action. Set the constraints as follows: in, The Gini coefficient represents the equity of block producers. The Gini coefficient of the geographical location of block producers. The thresholds for decentralization are related to equity distribution and geographic location distribution, respectively. Consensus latency refers to the time cost for validators to verify the generated blocks. Different consensus protocol identifiers It is a block interval. It refers to the number of intervals; Indicates the maximum tolerable number of malicious validators; It represents the number of malicious validators.

4. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to claim 3, characterized in that, Gini coefficient of block producer equity The calculation expression is: in, For block producers The percentage of shares, The number of block producers; Gini coefficient of block producer geographic location The calculation expression is: Among them, density set , Block producers are scattered in area.

5. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to claim 3, characterized in that, In establishing a Markov decision process model for the blockchain performance optimization problem, the reward function is defined as follows: 。 6. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to any one of claims 1 to 5, characterized in that, The steps for multiple iterative training using MDRL include: In each iteration, the block producer node receives new computational tasks and stores them in the data cache, observing the state of each part of the blockchain system's intelligent agents. And integrate it into a global state. t is the iteration number, and n is the agent index; Input the state information of each agent Action information from the last iteration The Q-value of each agent is solved using the QMIX algorithm, and the global state is utilized. Estimate the Q-value of the combined action; Choose the action strategy with the highest Q value. Output to the environment to perform actions ,Right now ; Select block producers and consensus algorithms, adjust block size and block interval, and receive corresponding reward values ​​from the environment. ; The global state is changed during each iteration. Action information ,award up to and the global state of the next iteration The data is stored in the database. Samples are extracted from the experience database to calculate the Q-value and loss function. The Q-network is then updated using the loss function. The process is repeated multiple times until the optimal block generation strategy is obtained after training is completed.

7. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to claim 6, characterized in that, Specifically, samples i are randomly drawn from the empirical database according to... Calculate the sample Q value, according to Calculate the loss function, where It is learned by sampling a batch of b transitions from the replay memory and minimizing the squared TD error, where b is the number of samples sampled from the empirical memory. To periodically from Copy the parameters of the target network from the obtained DQN. For state The actions taken For state The maximum value obtained under certain circumstances.

8. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to any one of claims 1 to 5, characterized in that, The optimal block generation strategy includes the selection and / or adjustment of block producers, consensus algorithms, block size, and block interval.

9. The blockchain performance optimization method based on multi-agent deep reinforcement learning according to any one of claims 1 to 5, characterized in that, The method of treating each action affecting the blockchain system as an intelligent agent and establishing a Markov decision process model for the blockchain performance optimization problem also includes loading historical state transition curves and Q-value estimates into the experience memory D, and pre-training the DNN using input pairs (S,A) and the corresponding estimated Q(S,A), where S represents the state and A represents the action.

10. A computer device comprising a processor and a memory, the memory being used to store computer programs, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 9.

Images