A Blockchain Consensus Method Based on Multi-Agent Reinforcement Learning and Its Application

Abstract

This invention belongs to the fields of Internet of Things (IoT) and blockchain technology, specifically relating to the application of blockchain consensus methods in the IoT field. A blockchain consensus method based on multi-agent reinforcement learning includes: Step 1: Constructing a multi-agent reinforcement learning model, the basic elements required for multi-agent reinforcement learning include agents, actions, states, environment, and reward functions; Step 2: Constructing a problem scenario suitable for blockchain consensus algorithms as the environment for the multi-agent reinforcement learning model; Step 3: The multi-agent reinforcement learning model interacts with the environment for training; Step 4: Deploying the trained multi-agent reinforcement learning model to achieve blockchain consensus. This invention employs multi-agent reinforcement learning to implement an IoT blockchain consensus method, solving the communication problem between wireless nodes in the IoT, ensuring fair channel usage while improving channel utilization.

Description

Technical Field

[0001] This invention belongs to the fields of Internet of Things (IoT) and blockchain technology, specifically relating to the application of blockchain consensus methods in the IoT field. Background Technology

[0002] The Internet of Things (IoT) connects a vast number of devices to enable them to work collaboratively automatically without human intervention. IoT devices can generate different types of data to serve various purposes. For example, automated industrial systems, smart logistics systems, e-health systems, and other similar intelligent service systems are achieved through large-scale, complex IoT technologies to automate and optimize processes for better service quality and resource utilization. However, these advantages come at a cost. Complex, interconnected, and heterogeneous networks are vulnerable to cyberattacks. Handling unstructured data generated by smart devices, achieving higher standards of data acquisition, integrating heterogeneous data into a unified system through common protocols, and access control for industrial networks are some of the key challenges facing intelligent systems.

[0003] Traditional centralized networks, such as Bluetooth, cellular networks, and WiFi, rely on centralized nodes like base stations to centrally manage and process user requests. All information is aggregated at this central node, leading to congestion and high energy consumption. With the further development of IoT technology, traditional centralized networks can no longer guarantee efficient collaborative communication between things. Alleviating the communication pressure on centralized networks is a new challenge facing IoT technology.

[0004] Blockchain is a distributed ledger technology that provides a secure way to create and record transactions and contracts. It is consensus-driven, offering a highly secure, immutable, and encrypted record-keeping mechanism. The Internet of Things (IoT) can significantly benefit from distributed ledger technology for data exchange, access control, and management. Blockchain eliminates the need for a central server and provides high data availability, security, transparency, and immutability for the IoT. By providing data integrity, blockchain technology offers the possibility of solving traditional IoT problems.

[0005] Blockchain technology includes peer-to-peer communication, consensus algorithms, distributed storage technology, and encryption algorithms. Consensus algorithms are the core technology of blockchain, and previous research on this topic has often encountered two types of problems: The first problem is that most consensus protocols are designed based on the assumption of efficient and reliable transmission, such as Proof-of-Work (PoW) and Proof-of-Acativity (PoA), ignoring the potential instability, unpredictability, and even congestion in real-world wireless network applications. The second problem is that while real-world scenarios of self-organizing networks are considered, such as using a realistic SINR model to describe interference in the wireless channel and proposing a Proof-of-Communication (PoC) consensus protocol based on this, their channel utilization is often low. Summary of the Invention

[0006] To address the above challenges and improve the channel utilization of blockchain consensus algorithms, this invention proposes a blockchain consensus algorithm based on multi-agent reinforcement learning. After applying multi-agent reinforcement learning, a channel utilization index is proposed to measure fairness and design rewards accordingly, enabling each agent to learn the optimal strategy and achieve both fairness and maximum channel utilization.

[0007] The technical solution adopted in this invention is: a blockchain consensus method based on multi-agent reinforcement learning, comprising:

[0008] Step 1: Construct a multi-agent reinforcement learning model. The basic elements required for multi-agent reinforcement learning include agents, actions, states, environment, and reward functions.

[0009] Step 2: Construct problem scenarios suitable for blockchain consensus algorithms as an environment for multi-agent reinforcement learning models;

[0010] Step 3: The multi-agent reinforcement learning model interacts with the environment for training;

[0011] Step 4: Deploy the trained multi-agent reinforcement learning model to achieve blockchain consensus.

[0012] Furthermore, the construction of the multi-agent reinforcement learning model includes the construction of agents, actions, states, environment, and reward functions, specifically:

[0013] (1) Constructing agents for multi-agent reinforcement learning: Construct n agents, each acting as a validator in the blockchain, only able to observe the public channel state and its own state; each agent i contains two neural networks, namely:

[0014] Policy Network, π i (·|s i ;θ i The input is the state s of each agent. i The output is a vector, where each element corresponds to the probability of an action. When making a decision, action 'a' is obtained by randomly sampling from this vector. i That is, the actions to be taken at the initial stage of a certain time cycle;

[0015] Value network, q(s) i a i ;w i ), by inputting state s i And action a i The output is the corresponding evaluation value of the policy network's performance, which is used to guide the update of the policy network, while the update of the value network itself depends on the feedback of the reward function.

[0016] (2) Constructing the agent's actions: In each time round t, each agent i selects an action after sampling according to its own probability:

[0017]

[0018] Each agent i has p i The probability of taking the action of sending information. Or there is 1-p i The probability of taking action to listen to the channel. A i (t) represents the set of actions for sending information and actions for listening to the channel;

[0019] (3) Constructing the states required for multi-agent reinforcement learning:

[0020] If, in a given round, only one agent chooses to send a message while all other agents choose to listen to the channel, then the leader election for that round is successful, and the channel is successfully utilized. Therefore, this round is defined as a successful round t. s Update the channel utilization index s of the intelligent agent. i (t s ):

[0021]

[0022] s i (t s This describes the interval between successful rounds since agent i was last successfully elected as leader; where i represents the agent's ID, t... s The number of iterations t during the training process in which the leader was successfully elected.

[0023] Each agent i maintains a global channel utilization index table locally for calculation

[0024]

[0025] in, The average channel utilization is represented by n; n is the number of agents.

[0026] (4) Constructing a multi-agent reinforcement learning environment:

[0027] Within each time round t, each agent calculates the probability of taking actions such as sending information and listening to the channel, and then samples and takes actions based on the probabilities; after all agents have taken their actions, the state of the channel is denoted as C(t):

[0028]

[0029] (5) Construct the agent reward function:

[0030] If an agent i is in time wheel t at time A i If (t) = 1, then its s is simply written as s i (t s ) for s a Each agent learns about s from its own local global channel utilization index table update. a The reward function is:

[0031] For each agent i, in a certain time round t, if after executing an action, the channel state is detected as C(t) = 1: the channel is not empty and exactly only one agent is sending information, while other agents choose to listen to the channel, then this round can be considered a successful leader election; at this time, if the agent elected as the leader satisfies: Then all agents will receive a reward of 1; conversely, the agent chosen as the leader will satisfy: Then, a penalty of -1 is applied to the agent, and the condition that is met is also penalized. The intelligent agent.

[0032] A further preferred approach is to use a problem scenario suitable for blockchain consensus algorithms to calculate the channel usage status for training the multi-agent reinforcement learning model. The construction method is as follows: n wireless communication nodes are arbitrarily deployed in a two-dimensional Euclidean space, and the network where the nodes reside is a synchronous single-hop network transmitting in half-duplex mode; time in the network is divided into synchronous rounds; u and v represent the transmitter sending information and the receiver listening to the channel in the current round, respectively; B is defined as the set of all transmitters in the current round; then the transmission model from transmitter to receiver is as follows:

[0033]

[0034]

[0035] Where Signal(v) is the signal strength received by the receiver, SINR(u, v) is the signal-to-noise ratio of the transmitted signal, b is any transmitter in B; d(b, v) is the Euclidean distance between the transmitter and the receiver, and P b ρ is the transmitter's transmit power, ρ∈(2,6] is the path loss exponent, and N is the ambient noise; where ρ and N are determined by the environment; when SINR(u,v)≥M, it means that the receiver can decode the signal from the transmitter, indicating that the channel is successfully used; otherwise, when SINR(u,v)<M, it means that the channel transmission has failed; M is a threshold determined by hardware and is usually greater than 1.

[0036] Furthermore, the multi-agent reinforcement learning model interacts with the environment to train and update its network parameters, specifically: 1) randomly initializes the policy network π. i (·|s i ;θ i The parameter θ in ) i Value Network The parameter w in i Initialize channel utilization metrics: s i (t)=s i (t s =0, and the global Interval table, decay factors γ, α, β, and maximum training epochs T. max ;

[0037] (2) Perform T max In this training iteration, each agent trains its own two networks, and the training steps are as follows:

[0038] The policy network uses the channel utilization index of the current time round t. Making decisions: The agent is then instructed to sample and execute actions based on the probability distribution output by the current policy network.

[0039] Get new and rewards

[0040] Make decisions based on policy networks: But prevent the agent from performing actions.

[0041] Value network scoring: and

[0042] Calculate the TD target and TD error: and

[0043] Updating the value network:

[0044] Update policy network:

[0045] Furthermore, the parameter update methods for the policy network include:

[0046] The parameter update methods for policy networks include:

[0047] Use the value network q(s, a; w) to represent Q π Approximating (s, a) yields the approximate policy gradient:

[0048]

[0049] Q π (s, a) is the value of policy π when taking action a in specific state s; it is a function that measures the value of the action. π(a|s) is the probability of taking action a in specific state s.

[0050] Update the parameters of the policy network using gradient ascent:

[0051] Each agent i needs a value network q(s) to approximate its value. i a i ;w i The value network scores at time t. Then, for each agent's policy π, the parameter θ at time t... i The updated formula is

[0052]

[0053] Furthermore, the parameters of the value network are updated using the SARSA algorithm, specifically as follows:

[0054] Calculate the TD target:

[0055] in, It is based on the reward r of the previous round as actually observed in round t+1. t Action value function Q π (s t a t The estimate;

[0056] In round t, the value network output is The gradient of the loss function is:

[0057]

[0058] Perform a gradient descent update on w:

[0059]

[0060] This invention also provides an application of the blockchain consensus method based on multi-agent reinforcement learning, which can be applied to the Internet of Everything to realize distributed channel contention among wireless nodes.

[0061] This invention employs a multi-agent reinforcement learning approach to implement an IoT blockchain consensus method. Based on a pre-trained multi-agent reinforcement learning model, this method uses blockchain consensus to solve the communication problem between wireless nodes in the IoT. It considers real-world scenarios of self-organizing network applications, ensuring fair channel usage while also improving channel utilization. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the blockchain consensus method based on multi-agent reinforcement learning in an embodiment of the present invention.

[0063] Figure 2 This is a framework diagram of the interaction between agents and the environment in the multi-agent reinforcement learning model in this embodiment of the invention;

[0064] Figure 3 This is a comparison of experimental results between the blockchain consensus method based on multi-agent reinforcement learning and the blockchain consensus method using existing random algorithms in this embodiment of the invention; (a), (b), (c), and (d) are comparisons of four cases where n takes the values ​​3, 5, 8, and 10, respectively. Detailed Implementation

[0065] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the present invention.

[0066] In a blockchain system, the node devices used to reach consensus are called validators. Similar to the characteristics of classic consensus problems, this invention also defines the consensus characteristics in a blockchain system to ensure that all validators can quickly reach agreement on a common network transaction history. The three characteristics of consensus in a blockchain system are as follows:

[0067] a) Termination: Validators reach a consensus on a proposed block within a limited time.

[0068] b) Validity and Agreement: This invention considers a scenario in which all nodes in a blockchain are relatively honest and reliable. Once a validator is elected as the leader, proposes a new block and immediately broadcasts it, then all validators write the new block into the main chain because of the high level of trust among the nodes.

[0069] As can be seen from the above consensus characteristics, reaching a consensus first requires electing a leader in the blockchain system. This leader proposes a new block, broadcasts it to all other nodes for verification, and receives opinions from all nodes regarding the new block, ultimately reaching a consensus. It is evident that the leader election step is fundamental and crucial. Previous work typically used a distributed random algorithm for leader election: in each round, each validator sends a message with a probability of 1 / n and listens for messages with a probability of 1-1 / n. After several rounds of iteration, only one validator in the entire blockchain system will send a message, while the rest will listen, thus completing one leader election. This algorithm ensures that only one validator will be elected as the leader with a high probability during the leader election phase, ultimately guaranteeing that all validators in the blockchain system have an equal probability of being elected as the leader, thus ensuring expected fairness. This invention proposes a channel utilization index and, through its combination with reinforcement learning, designs a more equitable and efficient algorithm that ultimately guarantees that each validator can deterministically take turns being elected as the leader to propose the desired consensus.

[0070] To achieve consensus in blockchain systems, a four-phase consensus framework has been widely adopted in previous work. This invention also considers the problem within this framework. The following is a detailed description of the four-phase consensus framework:

[0071] a) Leader Election (LE) Phase: Select a validator from all validators in the blockchain system to serve as the leader.

[0072] b) Block Proposal (BP) Phase: During the LE phase, the elected leader proposes a new block containing transaction records and related information from previous blocks in the network. In the current BP phase, the new block is distributed to all other validators in the blockchain system.

[0073] c) Block Validation (BV) Phase: After receiving the block proposed and propagated by the leader, other validators will validate the content of the block and return their confirmation to the leader, that is, agree or disagree with the proposed block.

[0074] d) Chain Updation (CU) Phase: All validators reach a consensus on whether the proposed block should be written into the main chain.

[0075] The algorithm of this invention is also designed within a four-phase consensus framework. Specifically, by designing a reinforcement learning algorithm, this invention ensures that in the LE phase, only one validator is deterministically elected as the leader, and all other validators know who the elected leader is; in the BP phase, the block proposed by the leader will be received by all other validators; in the BV phase, this invention considers the case where all nodes in the blockchain are relatively honest and reliable, and all validators that receive the proposed block return their agreement information to the leader; in the CU phase, all validators will write blocks on the main chain.

[0076] As attached Figure 1 As shown, the blockchain consensus method based on multi-agent reinforcement learning provided by this invention includes the following specific steps:

[0077] Step 1: Constructing problem scenarios suitable for blockchain consensus algorithms

[0078] There are n wireless node devices that can be arbitrarily deployed in a two-dimensional Euclidean space. The simulation experiments of this invention used four cases where n is 3, 5, 8, and 10. The network in which the nodes reside is a synchronous single-hop network and transmits in half-duplex mode: specifically, time in the network is divided into synchronous rounds, in which all nodes are synchronously awakened. At the beginning of each round, each node in the network can choose to send a message or listen to the channel in the wireless channel, but since the node is half-duplex, it cannot send a message and listen to the channel simultaneously. Each node is within each other's transmission range, but whether it can receive a message in the current round depends on its signal-to-noise ratio (SINR), i.e., the ratio of signal to interference plus noise.

[0079] This invention uses transmitters and receivers to distinguish nodes that send messages or listen to the channel in the current round. Let B represent all transmitters in the current round. Then, the transmission model for transmitter (u) to receiver (v) is as follows:

[0080]

[0081]

[0082] Where Signal(v) is the signal strength received by the receiver, SINR(u, v) is the signal-to-noise ratio of the transmitted signal, b is any transmitter in B; d(b, v) is the Euclidean distance between the transmitter and the receiver, and P b ρ is the transmitter's transmit power, ρδ(2,6) is the path loss exponent, and N is the ambient noise; where ρ and N are determined by the environment; when SINR(u,v)≥M, it means that the receiver can decode the signal from the transmitter, indicating that the channel is successfully used; otherwise, when SINR(u,v)<M, it means that the channel transmission has failed; M is a threshold determined by the hardware and is usually greater than 1.

[0083] This invention considers the sufficient condition for the success of the SINR model: that is, there is one and only one node transmitting messages while all other nodes are receiving messages. Therefore, the channel is successfully used when only a single node sends messages in a certain time round while all other nodes are listening to the channel; and when no node sends messages or multiple nodes send messages in a certain time round, the channel transmission fails.

[0084] Step 2: Construct the basic elements and multi-agent reinforcement learning model required for multi-agent reinforcement learning, specifically including:

[0085] (a) Constructing agents for multi-agent reinforcement learning: Construct n agents, each acting as a validator in the blockchain, only able to observe the public channel state and its own state; each agent i contains two neural networks, namely:

[0086] Policy Network, π i (·|s i ;θ i The input is the state s of each agent. i The output is a vector, where each element corresponds to the probability of an action. When making a decision, action 'a' is obtained by randomly sampling from this vector. i That is, the actions to be taken at the initial stage of a certain time cycle;

[0087] Value network, q(s) i a i ;w i ), by inputting state s i And action a i The output is the corresponding evaluation value of the policy network's performance, which is used to guide the update of the policy network, while the update of the value network itself depends on the feedback of the reward function.

[0088] (b) Actions for constructing agents: In each round t, each agent i has an independent p i The probability of taking the action of sending information a i (t), or in terms of 1-p i The probability of taking action to listen to the channel is a i (t). And it is clear that each agent, after sampling according to its own probability, can only take one of two actions, denoted as A in this invention. i (t):

[0089]

[0090] (c) Constructing the states required for multi-agent reinforcement learning:

[0091] If only one agent chooses to send a message in this round of the channel, while all other agents choose to listen to the channel, then the leader election for this round is successful, and the channel for this round is successfully utilized. Therefore, it is defined as a successful round t. s To better leverage reinforcement learning to solve problems, this invention proposes channel utilization metrics (Intervals). i (t s Interval describes the number of successful rounds that have passed since agent i was last successfully elected as leader. Specifically, there are:

[0092]

[0093] It should be noted that this metric is only updated when an agent is successfully selected as the leader in the current round (successful round), where i represents the agent's ID, and t... sThe number of iterations t during training represents the number of times the leader is successfully elected. The update of the indicators is shown in the above formula. For example, if agent i is successfully elected as the leader in this successful round, its own Interval is reset to zero, while the Intervals of other agents are incremented by 1.

[0094] This leads to the concept of average channel utilization. For each agent i, a global Interval table is maintained locally to compute... For example, Table 1 shows the initial scenario where only the third agent was successfully elected as the leader.

[0095] Table 1 Local Global Interval Table

[0096]

[0097] (d) Constructing a Multi-Agent Reinforcement Learning Environment: A problem scenario suitable for blockchain consensus algorithms is constructed as the interactive environment for multi-agent reinforcement learning. Each node acts as an agent, participating in the leader election competition within the channel while simultaneously shaping the channel state. Each agent calculates the probability of performing one of two actions based on its own state, and then samples actions according to these probabilities. Clearly, after all agents have performed their actions, the channel state (environment) is determined, i.e., whether the leader election is successful or unsuccessful, or whether the channel is successfully utilized or not. Specifically, after all agents have performed their actions in each round, there are only two possible outcomes in the channel, denoted as C(t) in this invention:

[0098]

[0099] (e) Constructing the agent reward function:

[0100] If an agent has A(t) = 1 at time t, then its Interval s is simply denoted as s. i (t s ) for s a Each agent can know the status quo from its own local Interval table updates. a Therefore, the reward design of this invention is as follows:

[0101]

[0102] As shown in the above formula, for each agent i, if in a certain round t, after performing its action, it detects the channel state, i.e., C(t) = 1: at this time, the channel is not empty and exactly only one agent is sending information, while other agents choose to listen to the channel, then this round can be considered a successful leader election. At this point, if the Inteval of the agent elected as the leader is greater than or equal to the average Interval, i.e. This is the best-case scenario: all agents cooperated to complete a leader election while ensuring fairness in the Interval metric, so all agents will receive a reward of 1; however, if the Inteval of the agent elected as the leader is less than the average Interval, i.e. This invention argues that the number of times the agent is elected as the leader in this round is too frequent. Therefore, it is penalized with a penalty of -1, and also penalized for failing to run for leadership when it should have. The intelligent agent. This invention argues that the above two situations represent the essence of the problem as a hybrid task type of reinforcement learning involving competition and cooperation, while other general situations are neither good nor bad, and therefore neither rewarded nor punished.

[0103] Based on the basic elements constructed above, the multi-agent reinforcement learning model required by the algorithm is built; as shown in the appendix. Figure 2 As shown.

[0104] Step 3: Using the constructed problem scenario suitable for blockchain consensus algorithms as the environment, the multi-agent reinforcement learning model interacts with the environment. By continuously updating the network parameters of the agents, the model is trained until a relatively fair blockchain consensus is achieved.

[0105] The model training process specifically includes:

[0106] (1) Randomly initialize the policy network (Actor network) π of the agent. i (·|s i ;θ i The parameter θ in ) i Value Network (Critic Network) The parameter w in i Initialize channel utilization metrics: s i (t)=s i (t s =0, and the global Interval table. Attenuation factor γ is set to 0.0005, attenuation factor α to 0.001, attenuation factor β to 0.001, T max The adjustment is dynamic and varies depending on the number of nodes, as long as it can be guaranteed at T. max Within this scope, the algorithm only needs to converge; as the number of elements N increases, T needs to be increased accordingly. max The superscript i represents the i-th agent; the subscript t represents the t-th time round, or the t-th round.

[0107] (2) Based on the maximum training rounds T max Perform T maxIn each training round, each agent trains its own two networks. The training steps for each round are as follows:

[0108] a. The policy network uses the channel utilization index at the current round t. Making decisions: The agent is then instructed to sample and execute actions based on the probability distribution output by the current policy network.

[0109] b. Obtain new channel utilization indicators and rewards Make decisions based on policy networks: But prevent the agent from performing actions.

[0110] c. Let the value network score: and

[0111] d. Calculate the TD target and TD error: and

[0112] f. Update the value network: Update policy network:

[0113]

[0114] Specifically, the parameter update method for the policy network is as follows:

[0115] Theoretically, the loss function of the policy network is as follows:

[0116]

[0117] θ represents the parameters that the policy neural network itself needs to update; the uppercase letter S represents the state in the expectation formula; V in the loss function π (S) represents the state value of policy π under all states S. Taking its expected value is the theoretical loss function, which measures the state value of policy π and thus determines the quality of the policy. Correspondingly, policy π in a specific state s... t The theoretical state value V π (s) is as follows

[0118]

[0119] In this formula, the uppercase letter A represents the action, the lowercase letter a represents the specific action, and the lowercase letter s represents the specific state. Q is the set of all actions that strategy π can take, where π(a|s) is the probability of taking action a in a specific state s. π(s, a) is the value of policy π when taking specific action a in specific state s. It is a function that measures the value of an action.

[0120] By the policy gradient theorem

[0121]

[0122] The stochastic gradient can be calculated:

[0123]

[0124] Clearly, g(s, a; θ) is the policy gradient. Unbiased estimation:

[0125]

[0126] However, the action value function Q cannot be directly calculated. π (s, a), can only be achieved by using another neural network to Q. π Approximating (s, a), we then use the value network q(s, a; w) to approximate Q. π Approximating (s, a), we can use Q in the above stochastic gradient formula. π Replacing it with a value network yields an approximate policy gradient:

[0127]

[0128] Finally, the parameters of the policy network are updated using gradient ascent:

[0129] Each agent i needs a value network q(s) to approximate its value. i a i ;w i This value network scores at time t. Then, for each agent's policy π, the parameter θ at time t... i The updated formula is

[0130]

[0131] The specific method for updating the parameters of the value network is as follows:

[0132] The specific steps are as follows: A time-division (TD) algorithm called SARSA in reinforcement learning is used.

[0133] In round t, the value network output

[0134] q t =q(s t a t ;w)

[0135] w represents the parameters that the value neural network itself needs to update, and qt represents the action value function Q of the value network at round t. π (s t a t The estimation of policy π in state s t Take action a t Value estimation at time. In round t+1, r was actually observed. t s t+1 a t+1 Therefore, the TD target can be calculated:

[0136]

[0137] Here, γ is the decay factor, which needs to be selected appropriately in actual training to train the neural network well. It is also a function of action value Q π (s t a t The estimate is based on the fact that... Partially based on the actual observed reward r t The present invention believes that Comparison of q(s) t a t (w) is closer to the truth. Therefore, it is... Fix it, encourage q(s) t a t ;w) to approach The SARSA algorithm updates the value network parameters w as follows, and defines the loss function:

[0138]

[0139] set up The gradient of the loss function is:

[0140]

[0141] Perform a gradient descent update on w:

[0142]

[0143] α is a decay factor, and it needs to be set to an appropriate value in actual training to train the neural network well. This way, updating w can improve q(s). t a t ;w) closer

[0144] Step 4: Use the trained model for blockchain consensus to achieve fair and efficient use of wireless channels in the Internet of Things.

[0145] The consensus method of this invention is simulated and compared with the conventional random algorithm consensus method, as shown in the attached figure. Figure 3 As shown, (a), (b), (c), and (d) represent four cases where n is 3, 5, 8, and 10, respectively. The horizontal axis represents the ID number of each node, and the vertical axis represents the count of the number of times each node was successfully selected as the leader to propose consensus in 1000 time rounds. The vertical bar data represents the experimental data after the method of this invention was trained and deployed, and the diagonal bar data represents the experimental data of the random algorithm in the same scenario (when there are n nodes, each node has a probability of 1 / n to send a message and a probability of 1-1 / n to listen to the channel). It can be seen that, compared with the count of each agent's success in 1000 time rounds, the results show that the random algorithm... The channel utilization of the random algorithm is very low, and it only guarantees probabilistic fairness, which is always highly uncertain in actual situations. As shown in the figure, the sum of the number of diagonal bars is far less than 1000, and it can be seen that the number of times each node is selected as the leader is not absolutely equal, but varies to different degrees with different n. Unlike the random algorithm, the method of this invention, when trained well, can ultimately guarantee that the sum of the number of times each agent is selected as the leader and proposes consensus is almost equal to 1000, and the number of times each agent is selected is almost completely equal, as shown by the vertical bars in the figure. This shows that the method of this invention guarantees the maximization of channel utilization and actual fairness.

[0146] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A blockchain consensus method based on multi-agent reinforcement learning, characterized in that, include: Step 1: Construct a multi-agent reinforcement learning model. The basic elements required for multi-agent reinforcement learning include agents, actions, states, environment, and reward functions. The method for constructing the multi-agent reinforcement learning model is as follows: (1) Constructing agents for multi-agent reinforcement learning: Constructing n Each agent acts as a validator in the blockchain and can only observe the public channel state and its own state; each agent... It contains two neural networks, namely: Policy networks The input is the state of each agent. The output is a vector, where each element corresponds to the probability of an action. When making a decision, actions are randomly sampled based on this vector. That is, the actions to be taken at the initial stage of a certain time cycle; These are the parameters of the policy network; Value network By inputting the status and actions The output is the corresponding evaluation value of the policy network's performance, which is used to guide the updating of the policy network; These are the parameters of the value network; (2) Constructing the agent's actions: in each time round In the middle, each intelligent agent After sampling based on their calculated probabilities, select an action: ； Each intelligent agent have The probability of taking the action of sending information. Or there is The probability of taking action to listen to the channel. ; Select a result for the action; (3) Constructing the states required for multi-agent reinforcement learning: If, in a given round, only one agent chooses to send a message while all other agents choose to listen to the channel, then the leader election for that round is successful, and the channel is successfully utilized; therefore, this round is defined as a successful round. Update the channel utilization index of the intelligent agent. : ； It describes an intelligent agent. How many successes have passed since he was last successfully elected as a leader? ID representing the intelligent agent, Represents the number of iterations during the training process. The number of successful elections for Chinese leaders; Each intelligent agent Maintain a global Interval table locally for calculation. : ;in, This represents the average channel utilization. n The number of agents; (4) Constructing a multi-agent reinforcement learning environment: Each time wheel Within the system, each agent calculates its own probability of taking actions such as sending information and listening to the channel, and then samples and performs actions based on these probabilities; after all agents have performed their actions, the channel state is recorded as follows: : ； (5) Construct the agent reward function: If an intelligent agent In the Wheel of Time of Then simply remember it. for Each agent learns from its own local Interval table updates. The reward function is: ； For each intelligent agent In a certain time cycle ,if If so, all agents will receive a reward of 1; otherwise, if If the action fails, a penalty of -1 is applied to the agent, and the action that satisfies the condition is also penalized. and intelligent agents; Then neither reward nor punish; Step 2: Construct a problem scenario suitable for blockchain consensus algorithms to calculate the signal-to-noise ratio of the channel, serving as the environment for a multi-agent reinforcement learning model. The construction method is as follows: Will n A number of wireless communication nodes are arbitrarily deployed in a two-dimensional Euclidean space, and the network in which the nodes reside is a synchronous single-hop network, transmitting in half-duplex mode; the time in the network is divided into synchronization wheels; and each is represented by a different time group. and To represent the transmitter sending information and the receiver listening to the channel in the current round; define Let be the set of all transmitters in the current round; then the transmission model from transmitter to receiver is as follows: ； in, It is the signal strength received by the receiver. It is the signal-to-noise ratio of the signal transmitted by the transmitter. for Any transmitter in the array; It is the Euclidean distance between the transmitter and the receiver. It is a transmitter The transmission power, It is a transmitter The transmission power, It is the path loss index. It is environmental noise; among which and It's all determined by the environment; when When the signal is clear, it means the receiver can decode the signal from the transmitter, indicating that the channel has been successfully used; otherwise... When this happens, it means the channel transmission has failed; It is a threshold determined by the hardware; Step 3: The multi-agent reinforcement learning model interacts with the environment for training; Step 4: Deploy the trained multi-agent reinforcement learning model to achieve blockchain consensus.

2. The blockchain consensus method based on multi-agent reinforcement learning according to claim 1, characterized in that, The multi-agent reinforcement learning model interacts with the environment to train and update its network parameters, specifically as follows: (1) Randomly initialize the policy network Parameters in Value Network Parameters in Initialize channel utilization indicators Global Interval table, decay factor Maximum number of training rounds ; (2) Perform In this training iteration, each agent trains its own two networks, and the training steps are as follows: The policy network is based on the current time wheel. Channel utilization indicators Making decisions: The agent is then instructed to sample and execute actions based on the probability distribution output by the current policy network. ; Get new and rewards ; Make decisions based on policy networks: However, the agent is not allowed to perform the action. ; Value network scoring: And calculate the value of the next action: ; calculate Target: ,calculate error: ; Updating the value network: ; Update the policy network; 3. The blockchain consensus method based on multi-agent reinforcement learning according to claim 2, characterized in that, The parameter update methods for policy networks include: Use value networks right By approximation, we obtain the approximate policy gradient: ； It is a strategy In specific states The following specific actions will be taken. The value of time is a function that measures the value of an action; In a specific state The following specific actions will be taken. The probability of; Update the parameters of the policy network using gradient ascent: .

4. The blockchain consensus method based on multi-agent reinforcement learning according to claim 2, characterized in that, The parameters of the value network are updated using the SARSA algorithm in reinforcement learning, specifically as follows: Calculate the TD objective: ; in, Based on The actual observed reward from the previous round in each round. Action value function The estimate; exist In each round, the value network output is The gradient of the loss function is: ； Perform one round of gradient descent update : 。 5. An application of the blockchain consensus method based on multi-agent reinforcement learning as described in any one of claims 1-4, characterized in that: This method is applied in the Internet of Things (IoT) to realize distributed channel contention among wireless nodes.

Images