Long safety chain network parameter optimization method and device based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based long safety chain network parameter optimization method and device, and the method comprises the steps: respectively collecting interaction data when all agents execute a current strategy; calculating a discount return function value and a dominant function value according to the interaction data; updating a strategy parameter of the current strategy according to the discount return function value and the advantage function value to obtain an updated parameter; and updating the network parameters of the target intelligent agent in the Changsafety chain by adopting the updating parameters. Therefore, the throughput of the block chain system is improved, the transaction confirmation time delay is reduced, and better adaptability and efficiency are shown in the face of network environment change.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to a Changan Chain network parameter optimization method and device based on reinforcement learning. Background Art

[0002] The blockchain database is a special distributed database that stores data in a series of interconnected blocks. Each block contains a batch of transaction records or other data, and these blocks are linked in chronological order to form a chain-like data structure. Unlike traditional databases, blockchain databases use cryptography technology to ensure the security, integrity and immutability of data.

[0003] In the field of blockchain performance optimization, a variety of technical solutions have been proposed to solve the problems of transaction processing efficiency and network throughput. Specifically, they can be divided into expert-based methods, manual parameter adjustment-based methods, and traditional machine learning methods.

[0004] Expert-based methods are highly dependent on the personal experience and knowledge of experts, which limits the universality and replicability of optimization and is difficult to adapt to rapidly changing network environments. Manual parameter adjustment methods are inefficient in actual operations and are difficult to effectively deal with the complexity of parameter configuration issues, especially when faced with a large number of parameters that need to be adjusted. Manual parameter adjustment is not only time-consuming but also prone to errors, which in turn limits blockchain performance and reduces processing efficiency. Summary of the invention

[0005] The present invention provides a Chang'an Chain network parameter optimization method and device based on reinforcement learning, which solves the technical problem that when a large number of parameters need to be adjusted, manual parameter adjustment is not only time-consuming but also prone to errors, which in turn leads to limited blockchain performance and reduced processing efficiency.

[0006] The present invention provides a Changan Chain network parameter optimization method based on reinforcement learning, wherein the Changan Chain includes a plurality of intelligent agents that communicate with each other, and the method includes:

[0007] When each of the intelligent agents executes the current strategy, the interaction data is collected respectively;

[0008] Calculating a discounted reward function value and an advantage function value based on the interaction data;

[0009] Updating the policy parameters of the current policy according to the discounted reward function value and the advantage function value to obtain updated parameters;

[0010] The update parameters are used to update the network parameters of the target agent in the Changan chain.

[0011] Optionally, when each of the intelligent agents executes the current strategy, the interaction data is collected respectively, including:

[0012] When each of the agents selects and executes an action in the current state according to the current strategy, the corresponding action reward, update status and end mark are collected;

[0013] An experience tuple is constructed using the current state, the executed action, the action reward, the updated state and the end identifier as interaction data.

[0014] Optionally, calculating the discounted reward function value and the advantage function value according to the interaction data includes:

[0015] Substituting the update status, the end flag and the action reward into a preset discount reward function to generate a discount reward function value;

[0016] Using a value network to evaluate a current state value corresponding to the current state;

[0017] Calculate the difference between the discounted reward function value and the current state value to obtain the advantage function value;

[0018] The discounted reward function is:

[0019]

[0020] in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

[0021] Optionally, updating the policy parameters of the current policy according to the discounted reward function value and the advantage function value to obtain updated parameters includes:

[0022] Calculating the TD error corresponding to the interaction data;

[0023] Update network parameters of the value network according to the TD error;

[0024] Calculate the policy probability of the policy network under the policy parameters;

[0025] Determining a policy gradient of the policy network according to the policy probability and the advantage function value;

[0026] Clipping the policy gradient to obtain an update gradient;

[0027] Perform gradient ascent optimization on the policy parameters of the current policy according to the update gradient to obtain updated parameters.

[0028] Optionally, the clipping the policy gradient to obtain an updated gradient includes:

[0029] Calculate the multiplication value between the policy gradient and the preset clipping parameter to obtain the clipping gradient;

[0030] The minimum value between the clipping gradient and the policy gradient is selected as the update gradient.

[0031] Optionally, the method comprises:

[0032] After the network parameters are updated, periodically obtaining the performance information of the target agent;

[0033] If the performance information does not meet the preset adjustment conditions, the process jumps to the step of collecting interaction data respectively when each of the agents executes the current strategy.

[0034] The present invention also provides a Changan Chain network parameter optimization device based on reinforcement learning, wherein the Changan Chain includes a plurality of intelligent agents communicating with each other, and the device includes:

[0035] A data collection module, used to collect interaction data when each of the intelligent agents executes the current strategy;

[0036] A function value calculation module, used to calculate a discounted reward function value and an advantage function value according to the interaction data;

[0037] A strategy parameter updating module, used to update the strategy parameters of the current strategy according to the discounted reward function value and the advantage function value to obtain updated parameters;

[0038] A parameter updating module is used to update the network parameters of the target intelligent agent in the Changan chain using the updated parameters.

[0039] Optionally, the data acquisition module is specifically used for:

[0040] When each of the agents selects and executes an action in the current state according to the current strategy, the corresponding action reward, update status and end mark are collected;

[0041] An experience tuple is constructed using the current state, the executed action, the action reward, the updated state and the end identifier as interaction data.

[0042] Optionally, the function value calculation module is specifically used to:

[0043] Substituting the update status, the end flag and the action reward into a preset discount reward function to generate a discount reward function value;

[0044] Using a value network to evaluate a current state value corresponding to the current state;

[0045] Calculate the difference between the discounted reward function value and the current state value to obtain the advantage function value;

[0046] The discounted reward function is:

[0047]

[0048] in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

[0049] Optionally, the policy parameter updating module is specifically used for:

[0050] Calculating the TD error corresponding to the interaction data;

[0051] Update network parameters of the value network according to the TD error;

[0052] Calculate the policy probability of the policy network under the policy parameters;

[0053] Determining a policy gradient of the policy network according to the policy probability and the advantage function value;

[0054] Clipping the policy gradient to obtain an update gradient;

[0055] Perform gradient ascent optimization on the policy parameters of the current policy according to the update gradient to obtain updated parameters.

[0056] It can be seen from the above technical solutions that the present invention has the following advantages:

[0057] When each agent executes the current strategy, it collects interaction data respectively; calculates the discount return function value and advantage function value based on the interaction data; updates the policy parameters of the current strategy based on the discount return function value and advantage function value to obtain the updated parameters; and uses the updated parameters to update the network parameters of the target agent in the Changan Chain. This improves the throughput of the blockchain system and reduces the transaction confirmation delay, while showing better adaptability and efficiency when facing changes in the network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A flowchart of the steps of a Changan Chain network parameter optimization method based on reinforcement learning provided in an embodiment of the present invention;

[0060] Figure 2 A cyclic update flow chart of a Changan Chain network parameter optimization method based on reinforcement learning provided in an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of a process for implementing parameter optimization by an optimizer provided in an embodiment of the present invention;

[0062] Figure 4 A structural block diagram of a Changan Chain network parameter optimization device based on reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In the existing technology, in the field of blockchain performance optimization, a variety of technical solutions have been proposed to solve the problems of transaction processing efficiency and network throughput. These solutions are mainly divided into the following categories:

[0064] Expert-based methods: These methods rely on expert knowledge and experience to optimize the configuration of blockchain systems. Experts usually manually adjust key parameters to achieve performance improvement based on the specific performance and historical data of the system. However, this method is limited by the personal experience of experts and their grasp of the complexity of the system, and it is often difficult to fully consider the interactions and impacts between all parameters.

[0065] Manual parameter adjustment method: Similar to the expert-based method, manual parameter adjustment also adjusts the configuration parameters of the blockchain through human intervention. This method is inefficient in actual operation and is difficult to effectively deal with the complexity of parameter configuration issues, especially when faced with a large number of parameters that need to be adjusted. Manual parameter adjustment is not only time-consuming but also prone to errors.

[0066] Traditional machine learning methods: This type of method predicts the optimal configuration of the system by using historical data and statistical models. Commonly used techniques include linear regression, support vector machine (SVM), decision tree, etc. These models learn the historical performance data of the system and establish a mapping relationship between input parameters and system performance, thereby predicting and recommending the best configuration.

[0067] To solve the above defects, the embodiments of the present invention provide a Changan Chain network parameter optimization method and device based on reinforcement learning, which is used to solve the technical problem that manual parameter adjustment is not only time-consuming but also prone to errors when a large number of parameters need to be adjusted, which in turn leads to limited blockchain performance and reduced processing efficiency. With system throughput and transaction confirmation delay as optimization targets, the overall performance is improved by learning and adjusting various system parameters.

[0068] Blockchain: A shared database in which the data or information stored has the characteristics of "unforgeable", "leaving traces throughout the process", "traceable", "open and transparent", and "collectively maintained".

[0069] Permissioned Blockchain: A permissioned blockchain is a blockchain that restricts access and operating permissions, so that only authorized participants can join the network, verify transactions, and access data.

[0070] Changan Chain: An open source blockchain framework for enterprise-level applications, with modular design, flexible consensus mechanism and privacy protection as its core features. It supports the creation of private channels to protect data privacy, and adopts a division of labor mode between peer nodes and sorting nodes to improve transaction processing efficiency. With its high compatibility and customizability, Changan Chain meets the needs of secure and efficient transactions and data sharing between enterprises.

[0071] Gossip protocol: A decentralized communication protocol that allows nodes to randomly select partners to exchange information, so that data in the network is gradually spread and synchronized among all nodes, similar to the way a virus spreads in a group.

[0072] Proximal Policy Optimization (PPO) algorithm: A reinforcement learning algorithm that aims to improve the performance of an agent in an environment by optimizing policies. PPO uses a truncated policy gradient method that allows multiple small updates in each update to balance exploration and exploitation. The algorithm ensures the stability and efficiency of the training process by limiting the magnitude of each update, and is widely used in various reinforcement learning tasks. The core of the PPO algorithm is to stabilize the training process by limiting the difference between the new and old policies, which is mainly achieved through the following key steps: Collect trajectories: Use the current policy to interact with the environment and collect data such as state, action, and reward. Calculate advantage: Estimate the advantage value of each state-action pair. Policy update: Update the policy network multiple times, but limit the update amplitude. Value function update: Optimize the value network to better estimate the state value.

[0073] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] See also Figure 1 , Figure 1 A flowchart of the steps of a Changan Chain network parameter optimization method based on reinforcement learning provided in an embodiment of the present invention.

[0075] The present invention provides a Changan Chain network parameter optimization method based on reinforcement learning, wherein the Changan Chain includes multiple intelligent agents that communicate with each other, and the method includes:

[0076] Step 101, when each agent executes the current strategy, the interaction data is collected respectively;

[0077] In an example of the present invention, step 101 may include the following sub-steps:

[0078] When each agent selects and executes an action in the current state according to the current strategy, it collects the corresponding action reward, updates the state, and ends the flag;

[0079] The current state, executed action, action reward, updated state and end flag are used to construct the experience tuple as interaction data.

[0080] In an embodiment of the present invention, Changan Chain includes multiple intelligent entities that communicate with each other. The intelligent entities can realize functions through data interaction. Each intelligent entity has a blockchain network layer equipped with an action execution network. When the action execution network detects the current state, it selects actions for the intelligent entity to execute according to the current strategy.

[0081] When each agent selects and executes actions according to the current strategy in the Changan Chain environment, the corresponding action reward, update status and end mark of the agent are collected. Then these data are combined with the current status and execution action to construct an experience tuple as interaction data. The experience tuple is as follows:

[0082]

[0083] The interaction data can include multiple groups. To facilitate subsequent processing, the interaction data corresponding to all agents at the current time step can be added to the same list and updated at the same time.

[0084] It should be noted that the policy network and the value network are two key components. The policy network is responsible for outputting the probability of taking each possible action in a given state, while the value network predicts the expected return of a given state. This method can be implemented through an optimizer, using an optimizer (such as Adam) to update the parameters of the policy network and the value network. The parameters of the optimizer, such as the learning rate, are adjustable to adapt to different learning speeds and convergence requirements.

[0085] In the specific implementation, the strategy network and value network of the intelligent agent as well as the strategy parameters and value parameters can also be initialized, and both the strategy network and the value network have corresponding target networks, where the strategy parameters of the target network are At the same time, you can also initialize the random process N for action exploration, reset the environment, and obtain the initial state , initialize an empty list to store experience tuples .

[0086] Step 102, calculating the discounted reward function value and the advantage function value according to the interaction data;

[0087] In one example of the present invention, step 102 may include the following sub-steps:

[0088] Substitute the updated state, end flag and action reward into the preset discounted reward function to generate the discounted reward function value;

[0089] Use the value network to evaluate the current state value corresponding to the current state;

[0090] Calculate the difference between the discounted reward function value and the current state value to get the advantage function value;

[0091] The discounted reward function is:

[0092]

[0093] in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

[0094] In an embodiment of the present invention, after obtaining the updated state, the end flag and the action reward, they are imported into the discounted reward function, and the discounted reward function value at each time step is calculated. Furthermore, in order to evaluate the quality of an action relative to the average level, it is necessary to calculate the advantage function. This is usually done through some form of time difference (TD) estimation or generalized advantage estimation (GAE). The difference between the discounted reward function value and the current state value is calculated to obtain the advantage function value.

[0095] Among them, the current state value can be evaluated through the value network corresponding to the strategy network.

[0096] Step 103, updating the policy parameters of the current policy according to the discounted reward function value and the advantage function value to obtain updated parameters;

[0097] In an example of the present invention, step 103 may include the following sub-steps S11-S16:

[0098] S11, calculating the TD error corresponding to the interaction data;

[0099] In this embodiment, TD error refers to Temporal - Difference Error, which is used to measure the difference between the value function based on the current estimate and the actual reward obtained. This difference reflects the accuracy of the value function estimation. For example, the agent will receive a reward after performing an action, and the TD error is the difference between the agent's expectation of future rewards (estimated by the value function) and the actual reward.

[0100] TD Error The specific calculation process can be as follows:

[0101]

[0102] S12, updating the network parameters of the value network according to the TD error;

[0103] After obtaining the TD error, the network parameters of the value network are To update, the specific method is as follows:

[0104]

[0105] in, is the learning rate of the optimizer, for The direction and rate of gradient change can be calculated through the automatic differentiation function of the deep learning framework to determine the value network parameters in the current state. The gradient direction and rate of change.

[0106] With the current parameters Subtract the product of the learning rate and the gradient to get the updated parameter value, completing an update operation on the value network parameters. With multiple such iterative updates, the value network's estimate of the state value will become more and more accurate and more in line with the actual situation.

[0107] S13, calculating the strategy probability of the strategy network under the strategy parameters;

[0108] S14. Determine the policy gradient of the policy network according to the policy probability and the advantage function value;

[0109] Specifically, the policy gradient The expression is as follows:

[0110]

[0111] in, is the advantage function value at time t, is the current state at time t , in the strategy parameters Next action The probability value of . For policy gradient calculation, for example, based on the parameters The position and role in the formula, the policy function is derived to obtain its These partial derivatives form the gradient vector.

[0112] S15. Clip the policy gradient to obtain the updated gradient;

[0113] Furthermore, S15 may include the following sub-steps:

[0114] Calculate the multiplication between the policy gradient and the preset clipping parameter to obtain the clipping gradient;

[0115] The minimum value between the clipping gradient and the policy gradient is selected as the update gradient.

[0116] In this embodiment, the gradient is updated The calculation can be expressed by the following formula:

[0117]

[0118] in, The preset cropping parameters.

[0119] In this embodiment, a clipped probability ratio is provided, which is the ratio of the probability of an action generated by the old policy to the new policy. This ratio is limited to a range to prevent the policy from making too large a change during the update.

[0120] S16. Perform gradient ascent optimization on the policy parameters of the current policy according to the update gradient to obtain updated parameters.

[0121] After obtaining the updated gradient, the policy parameters of the current policy can be optimized by gradient ascent according to the updated gradient. The specific process is as follows:

[0122]

[0123] in, is the learning rate of the optimizer, is the policy parameter of the policy network in the current policy.

[0124] In this embodiment, the gradient ascent method is used to update the policy parameters, that is, the policy is adjusted by optimizing the objective function so that the policy can better adapt to the environment and improve performance. Multiple updates can be safely performed on a data batch to strike a balance between learning stability and sufficient exploration to avoid local optimality and improve policy performance. By simultaneously optimizing the policy network and the value network, the decision-making ability of the agent in the Changan Chain environment is improved. According to the real-time performance feedback of the Changan Chain, the exploration strategy is dynamically adjusted to improve the exploration efficiency. Through the PPO algorithm, the agent learns how to adjust the network layer parameters to maximize transaction throughput and minimize confirmation time.

[0125] Step 104, using the updated parameters to update the network parameters of the target agent in the Changan chain.

[0126] In one example of the present invention, after the determination of the update parameters is completed, the network parameters of the target agent in the Changan chain are updated using the update parameters, as shown below:

[0127]

[0128]

[0129] in, is the update rate, which can be between 0.001 and 0.1. When it is close to 1, the target network parameters will be updated to the current network parameters more quickly; when When it is close to 0, the target network parameters are updated slowly and more stable.

[0130] In one example of the present invention, the method further comprises the following steps:

[0131] When the network parameters are updated, the performance information of the target agent is periodically obtained;

[0132] If the performance information does not meet the preset adjustment conditions, the process jumps to the step of collecting interaction data when each agent executes the current strategy.

[0133] In the embodiment of the present invention, Figure 2 As shown, the performance information of the target intelligent body can also be periodically obtained through a continuous monitoring and feedback mechanism through a specific performance feedback module after the network parameters are updated, so as to start the environment Env framework according to the performance information, and provide the real-time parameters of the blockchain to the reinforcement learning model, that is, the optimizer in this embodiment, through the parameter modification module and the parameter monitoring and calculation module, and obtain the new parameter state through the optimizer to make decisions and predict actions. Determine the update parameters according to the predicted actions. Finally, adjust the network layer parameters of the target intelligent body in the blockchain, and use smart contracts to update the parameters, thereby realizing the automation and intelligent optimization of Changan Chain performance, improving transaction processing efficiency, reducing energy consumption, and enhancing the dynamic adaptability of the system.

[0134] See also Figure 3 , Figure 3 The figure is a schematic diagram of the process of implementing parameter optimization by the optimizer in the embodiment of the present invention.

[0135] In this embodiment, information such as runtime performance parameters, block interval, transmission time and block size are obtained and input into the policy network. These parameters reflect the current operating status of the blockchain network; action a, i.e., adjustable parameters of the blockchain network layer, is selected through the policy network and acts on the network layer environment and the value network; the value network outputs the value function q according to the state s and action a to evaluate the current state and action value; the value network calculates the reward r of action a to the network layer environment to reflect the impact of action a on the performance of the blockchain network; according to the action a output by the policy network Actor, the parameters of the blockchain network layer are adjusted. This process is based on a reinforcement learning strategy, which aims to optimize the performance of the blockchain network. By continuously adjusting the parameters of the blockchain network layer, the system can adapt to the changing network environment and improve the overall performance.

[0136] In the embodiment of the present invention, when each agent executes the current strategy, the interaction data is collected respectively; the discount reward function value and the advantage function value are calculated according to the interaction data; the policy parameters of the current strategy are updated according to the discount reward function value and the advantage function value to obtain the updated parameters; and the network parameters of the target agent in the Changan chain are updated using the updated parameters. This improves the throughput of the blockchain system and reduces the transaction confirmation delay, while showing better adaptability and efficiency when facing changes in the network environment.

[0137] See also Figure 4 , Figure 4 A structural block diagram of a Changan Chain network parameter optimization device based on reinforcement learning in an embodiment of the present invention is shown.

[0138] The present invention also provides a Changan Chain network parameter optimization device based on reinforcement learning, wherein the Changan Chain includes a plurality of intelligent agents communicating with each other, and the device includes:

[0139] The data collection module 401 is used to collect interaction data when each agent executes the current strategy;

[0140] Function value calculation module 402, used to calculate the discounted reward function value and the advantage function value according to the interaction data;

[0141] A strategy parameter updating module 403 is used to update the strategy parameters of the current strategy according to the discounted reward function value and the advantage function value to obtain updated parameters;

[0142] The parameter updating module 404 is used to update the network parameters of the target agent in the Changan chain using the updated parameters.

[0143] Optionally, the data acquisition module 401 is specifically used for:

[0144] When each agent selects and executes an action in the current state according to the current strategy, it collects the corresponding action reward, updates the state, and ends the flag;

[0145] The current state, executed action, action reward, updated state and end flag are used to construct the experience tuple as interaction data.

[0146] Optionally, the function value calculation module 402 is specifically used for:

[0147] Substitute the updated state, end flag and action reward into the preset discounted reward function to generate the discounted reward function value;

[0148] Use the value network to evaluate the current state value corresponding to the current state;

[0149] Calculate the difference between the discounted reward function value and the current state value to get the advantage function value;

[0150] The discounted reward function is:

[0151]

[0152] in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

[0153] Optionally, the policy parameter updating module 403 is specifically used for:

[0154] Calculate the TD error corresponding to the interaction data;

[0155] Update the network parameters of the value network according to the TD error;

[0156] Calculate the policy probability of the policy network under the policy parameters;

[0157] Determine the policy gradient of the policy network according to the policy probability and advantage function value;

[0158] Clip the policy gradient to get the updated gradient;

[0159] According to the update gradient, the policy parameters of the current policy are optimized by gradient ascent to obtain the updated parameters.

[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0161] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0162] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Changan Chain network parameter optimization method based on reinforcement learning, characterized in that: The Changan Chain includes a plurality of intelligent agents communicating with each other, and the method includes: When each of the intelligent agents executes the current strategy, the interaction data is collected respectively; Calculating a discounted reward function value and an advantage function value based on the interaction data; Updating the policy parameters of the current policy according to the discounted reward function value and the advantage function value to obtain updated parameters; The update parameters are used to update the network parameters of the target agent in the Changan chain.

2. The method according to claim 1, characterized in that: When each of the intelligent agents executes the current strategy, the interaction data is collected respectively, including: When each of the agents selects and executes an action in the current state according to the current strategy, the corresponding action reward, update status and end mark are collected; An experience tuple is constructed using the current state, the executed action, the action reward, the updated state and the end identifier as interaction data.

3. The method according to claim 2, characterized in that The calculating the discounted reward function value and the advantage function value according to the interaction data includes: Substituting the update status, the end flag and the action reward into a preset discount reward function to generate a discount reward function value; Using a value network to evaluate a current state value corresponding to the current state; Calculate the difference between the discounted reward function value and the current state value to obtain the advantage function value; The discounted reward function is: in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

4. The method according to claim 2, characterized in that: The updating of the policy parameters of the current policy according to the discounted reward function value and the advantage function value to obtain updated parameters includes: Calculating the TD error corresponding to the interaction data; Update network parameters of the value network according to the TD error; Calculate the policy probability of the policy network under the policy parameters; Determining a policy gradient of the policy network according to the policy probability and the advantage function value; Clipping the policy gradient to obtain an updated gradient; Perform gradient ascent optimization on the policy parameters of the current policy according to the update gradient to obtain updated parameters.

5. The method according to claim 4, characterized in that The step of clipping the policy gradient to obtain an updated gradient includes: Calculate the multiplication value between the policy gradient and the preset clipping parameter to obtain the clipping gradient; The minimum value between the clipping gradient and the policy gradient is selected as the update gradient.

6. The method according to claim 1, characterized in that The method comprises: After the network parameters are updated, periodically obtaining the performance information of the target agent; If the performance information does not meet the preset adjustment conditions, the process jumps to the step of collecting interaction data respectively when each of the agents executes the current strategy.

7. A Changan Chain network parameter optimization device based on reinforcement learning, characterized in that: The Changan Chain includes a plurality of intelligent entities that communicate with each other, and the device includes: A data collection module, used to collect interaction data when each of the intelligent agents executes the current strategy; A function value calculation module, used to calculate a discounted reward function value and an advantage function value according to the interaction data; A strategy parameter updating module, used to update the strategy parameters of the current strategy according to the discounted reward function value and the advantage function value to obtain updated parameters; A parameter updating module is used to update the network parameters of the target intelligent agent in the Changan chain using the updated parameters.

8. The device according to claim 7, characterized in that The data acquisition module is specifically used for: When each of the agents selects and executes an action in the current state according to the current strategy, the corresponding action reward, update status and end mark are collected; An experience tuple is constructed using the current state, the executed action, the action reward, the updated state and the end identifier as interaction data.

9. The device according to claim 8, characterized in that The function value calculation module is specifically used for: Substituting the update status, the end flag and the action reward into a preset discount reward function to generate a discount reward function value; Using a value network to evaluate a current state value corresponding to the current state; Calculate the difference between the discounted reward function value and the current state value to obtain the advantage function value; The discounted reward function is: in, is the discounted return function value, For action rewards, is the preset discount factor, Update the state for the value network pair The assessed value of End mark.

10. The device according to claim 8, characterized in that The strategy parameter updating module is specifically used for: Calculating the TD error corresponding to the interaction data; Update network parameters of the value network according to the TD error; Calculate the policy probability of the policy network under the policy parameters; Determining a policy gradient of the policy network according to the policy probability and the advantage function value; Clipping the policy gradient to obtain an updated gradient; Perform gradient ascent optimization on the policy parameters of the current policy according to the update gradient to obtain updated parameters.

Images