A mapping operation method based on MADDPG algorithm

Abstract

The present invention belongs to the field of deep reinforcement learning and discloses a mapping operation method based on the MADDPG algorithm. In order to achieve peak regulation and over-response limitation for differential nodes in a decentralized control process and improve system safety and economy, the MADDPG (Multi-Agent Deep Deterministic Policy Gradient) deep reinforcement learning training method is introduced to train auxiliary parameters. A multi-agent algorithm is selected to simulate the interaction process between multiple participants and the environment. Then, an Actor-Critic architecture is constructed, and a value function is established for each agent to form a reasonable agent training and application mechanism. Subsequently, the experience replay pool, deep connection layer, and target network are further configured and the idea of ​​virtual regret minimization is integrated to guide agent training under sparse rewards. Finally, a cost function is used to estimate the strategies of other agents, and the strategy set is optimized to ensure that multiple agents can each give the next optimal action strategy in the application state after the training phase.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning, and specifically relates to a mapping operation method based on the MADDPG algorithm. Background Art

[0002] In recent years, with the development of smart grids, traditional power systems have undergone significant changes in many aspects, including structure, operation, dispatch, and control. On the power generation side, the emergence of renewable energy sources such as large-scale wind farms and centralized photovoltaic power plants has significantly increased the uncertainty of power output. On the distribution side, new power supply models such as distributed generation and microgrids have emerged, increasing the flexibility of grid operation while also increasing operational complexity. On the user side, loads are becoming more proactive, and the increasing popularity of electric vehicles, smart homes, and buildings has led to a diversification of the decision-makers involved in leading power system operations.

[0003] The increasingly complex operational characteristics of the new generation of power systems, characterized by smart grids, make determining the optimal strategy for each decision-maker to balance and optimize the interests of all parties involved in the power system a challenging task. Traditional optimization theories, characterized by single-agent decision-making, struggle to overcome this difficulty. Therefore, game theory, which focuses on multi-objective optimization for complex agents, is poised to become a powerful tool for addressing many key challenges in smart grids. However, with the increasing sophistication and complexity of problem models, the contradiction between operational computability and theoretical computability in solving game equilibrium points, or algorithmic solutions, continues to intensify. Consequently, a growing number of researchers are turning to machine learning and artificial intelligence methods, leveraging their data-driven and self-learning characteristics to assist in the decision-making process within existing game architectures. These researchers have also made numerous contributions to the underlying proofs and mathematical foundations of these algorithms, earning widespread recognition within the academic community.

[0004] Multi-party Bayesian games, also known as multi-party incomplete information games, are an extension of incomplete information games with multiple players. Since a game is considered fully informed only when all players have common knowledge of the set of players, their action sequences, their action space, and their payoff functions, the entire game is considered an incomplete information game if at least one player has incomplete knowledge of any of these. This means that fully informed games are only a specific type of incomplete information game. Therefore, Bayesian games are far more widely used in engineering modeling. However, due to their structural characteristics, these games are more difficult to solve. Traditional methods for finding Bayesian equilibrium solutions typically use the Harsanyi transformation proposed by John C. Harsanyi. This method states that any game with incomplete information under the original definition can be remodeled as a game with complete but imperfect information without changing its refinement. This requires only the addition of an initial action where nature chooses from a different set of rules. This method pioneered a new approach by disproving the classical assertion in game theory that "games with incomplete information are unanalyzable," achieving a breakthrough from scratch. However, because this method requires traversing all information sets of all participants in the game tree, it is computationally inefficient and virtually unsolvable for complex games with numerous decisions and participants.

[0005] Machine game theory, also known as computer game theory, is one of the most challenging and important research areas in the field of artificial intelligence. It forms a crucial foundation for AI research in intelligent decision-making. Machine game theory typically employs deep reinforcement learning methods within the AI ​​field. In theory, multiple participants in a multi-party game are modeled as multiple agents. Using centralized learning and distributed applications, each agent generates a real-time guess about the decisions of other agents based on historical experience. Based on this guess, each agent generates the optimal response during the game, thereby determining the game equilibrium and strategy. However, questions remain at the theoretical research stage regarding how to properly guide the learning direction of agents in real-world multi-party Bayesian games using sparse rewards, how to build and configure the architecture, and how to maintain a training environment with minimal communication requirements. Therefore, research on improving and using machine learning methods to solve multi-party Bayesian game strategies is of great significance.

[0006] With the continuous development of new power systems, more and more distributed energy resources (DERs) are being connected to the grid and participating in demand response processes. In price-based demand response, there is a time lag between load response and real-time price adjustments. This can cause a large amount of power to shift to the lowest price period, generating new load peaks that can even exceed the original peaks. This is known as over-response. To avoid this, it is necessary to introduce a certain number of load-assisted parameters into the load response process to impose constraints on the load response, thereby promoting safe system operation and reducing system operating costs.

[0007] The introduction of load auxiliary parameters can be summarized into the following three types according to the control method: 1) upper and lower limits of control response, such as stipulating upper and lower limits of responsive power or transferable time period, directly limiting over-response of controllable load; 2) setting penalty item coefficients for power increase or decrease, reducing the possibility of large-scale investment of controllable load during low electricity price period; 3) sending different price signals for different types of controllable loads to make their response time period different.

[0008] However, in practical applications, existing methods have the following shortcomings:

[0009] ① The availability of information required for node-specific optimization in microgrid decentralized control is limited. Due to user privacy protection, detailed load operating models and parameters are unavailable, making it difficult to solve the two-level optimization problem and calculate the optimal values ​​of auxiliary parameters using traditional analytical methods. ② Existing methods for limiting over-response do not optimize the values ​​of auxiliary parameters, fail to consider node differences caused by network constraints, and are difficult to optimize using analytical methods. ③ When dealing with complex environments and continuous actions, Q-learning (QL) and deep Q networks (DQN) generate excessively large training data volumes and often lead to suboptimal policy solutions. Therefore, studying how to establish a deep reinforcement learning (MADDPG) method for training auxiliary parameters based on actual power system requirements is of great significance for improving system safety and economic efficiency by addressing peak shaving and limiting over-response at differentiated nodes during decentralized microgrid control. Summary of the Invention

[0010] On the basis of the above, the present invention provides a mapping operation method based on the MADDPG algorithm. In order to achieve peak regulation and over-response limitation for differential nodes in the decentralized control process, improve system safety and economy, and achieve equilibrium solution for multi-party Bayesian game, based on the solution ideas of traditional methods, the idea of ​​virtual regret minimization is integrated, and a mapping operation method based on the MADDPG algorithm is proposed. The main examples are the auxiliary parameters of microgrid controllable loads and the multi-party Bayesian game solution training method.

[0011] The technical solution of the present invention is as follows: a mapping operation method based on the MADDPG algorithm, comprising the following steps:

[0012] S1 maps the multi-network microsystem model into the MADDPG multi-agent environment;

[0013] S2 builds an Actor-Critic framework: combining Value-Based single-step updates and Policy-Based continuous actions;

[0014] S3 builds the Actor update method: It extends the gradient formula to a deterministic strategy to build a main network for each agent and constructs an experience pool to store past experience for subsequent training of the agent.

[0015] S4 constructs the critic update method: for N agents, a strategy network, a target strategy network, a value network, and a target value network are established, totaling 4N agents.

[0016] S5 constructs estimates of the strategies of other agents: for each agent, N-1 strategy approximation functions are maintained, whose approximation cost is a logarithmic cost function, and the entropy of the strategy is added.

[0017] Furthermore, in step S1, the multi-grid microsystem model is: microgrid controllable load nodes or multi-party Bayesian game.

[0018] Furthermore, in step S1, the MADDPG algorithm includes N agents, each of which can only observe the local state information of the environment and cannot know the global state, and contains a neural network with network parameters θ={θ1,…,θ N}, the set of its strategies is π={π1,…,π N};

[0019] The main network of the agent consists of two networks: one is the policy network μ, which is used to make decisions that are consistent with the current environment and state; the other is the value network Q, which is used to judge the quality of the output actions of the policy network;

[0020] The policy network is based on the local state information o i Generate action a i After that, the environment will return the reward value r i and the observation value o at the next moment i ', all the agent's information {x,a1,…,a N ,r1,…,r N ,x'} are stored in the experience pool D and wait for the training phase as training samples for the neural network to use. x is the current state of the environment, a1,…,a Nare the actions of agents 1,…,N, r1,…,r N is the immediate reward value returned by the environment based on the agent's 1,…,N actions, and x' is the new environment state at the next moment.

[0021] Furthermore, in step S4, the target network is divided into a target policy network μ' and a target value network Q', which are initialized using the policy network parameters and the value network parameters in the main network, respectively. Specifically,

[0022] The S201 policy network is based on the agent's cumulative expected reward J(θ i ) to update the network function, and the gradient strategy update formula 1 and formula 2:

[0023]

[0024] Among them, J(θ i ) is the cumulative expected reward of the agent, For state s and action a i The joint expectation of , that is, when the state s changes from the steady-state state distribution ρ under the policy π π Medium sampling, action a i The expectation generated by the policy πθ of agent i, The cumulative discounted reward of agent i, γ t is the t-th power of the discount factor, used to decay future rewards, r i,t is the immediate reward obtained by the i-th agent at time t.

[0025] In formula 2: The network parameters are θ i The expected return gradient of the policy network is used to update the policy network parameters; For the current state x and action set {a1,…,a N The action value function output by the main value network is about action a i gradient; The action given by the policy network is about θ i The gradient of E x,a～D To find the expectation of the state x and action a sampled from the experience replay pool D, a i =μ i (o i ) is the policy network μ i According to the observed quantity o i Generated actions.

[0026] The S202 value network updates the network parameters by minimizing the TD error, updating formula 3:

[0027]

[0028] In formula 3: L(θ i ) is the loss function of the value network, which is used to evaluate the error between the value function estimated by the value network and the actual value function; E x,a,r,x' To find the expectation of the transfer sample (x, a, r, x') sampled in the experience replay pool, y is the actual action value function, which can be expressed as Formula 4:

[0029]

[0030] In formula 4: r i is the current reward, a j '=μ j '(o j ) is the target strategy network μ of agent j j 'According to the observation quantity o j The generated action, γ is the discount factor; γ∈[0,1]; is the action value function, which is used to evaluate the quality of the subsequent steps and is output by the target value network;

[0031] The training goal of the S203 main network is to maximize the expected return J(μ i ), while minimizing the value network loss function L(θ i ), the policy network and value network in the main network are updated with the learning rate α as the update step, and the update is obtained as formula 5:

[0032]

[0033] Among them, μ i is the policy network parameter of agent i, θ i is the value network parameter of agent i, α is the learning rate, is the strategy objective function J(μ i ) about the parameter θ i The gradient of is used to guide the direction of parameter update.

[0034] In step S204, the target network is updated using a soft update method. The main network parameters are used to update the target network. The update amplitude of the target network is determined by the target network update rate τ, 0≤τ<1. The updated formula 6 is:

[0035]

[0036] Among them, θ' is the parameter of the target value network, μ' is the parameter of the target policy network, θ is the parameter of the value network in the main network, μ is the parameter of the policy network in the main network, and τ is the target network update rate.

[0037] Furthermore, the formula 4 also includes the strategy of the other agent, which is obtained through communication, and the agent uses an approximation related to the agent's true strategy To approximate, the approximation cost is the logarithmic cost function, plus the entropy of the strategy, the cost function is formula 7:

[0038]

[0039] in, is the cost function of the strategy approximation of agent i, is the observation o of other agents j and action a j Find the expected joint distribution of is the approximation of agent i’s strategy to other agents, i.e., given observation o j When agent i predicts that other agents will take action a j The probability distribution of Approximate strategy The entropy is used to encourage the randomness of the strategy; λ is the weight coefficient of entropy regularization, which is used to balance the accuracy and exploration of the strategy, minimize the above cost function, and obtain the approximation of other agent strategies, which is used to replace y in Formula 4 to obtain Formula 8:

[0040]

[0041] in, is the corrected target value, r i is the immediate reward obtained by agent i at the current moment, is the target value function of agent i, is the approximation of the strategy of agent i to other agents (agent 1), o1,…,o N is the observation quantity of agent 1,…,N, is the approximation of the strategy of agent i to other agents (agent N).

[0042] Furthermore, during the training phase, the microgrid dispatching center first conducts behavioral exploration based on the historical data of the nodes, learns the regulation situation in the real environment, and uses the loss function to evaluate the regulation. The information is stored in the experience pool as training material, and participates in the centralized training of the neural network through sampling.

[0043] Furthermore, the intelligent agents do not need to mobilize their own value networks to evaluate the current state and strategy, but rely on the optimized strategy network in the training phase to achieve multi-party Nash equilibrium, thereby solving the optimal values ​​of auxiliary parameters.

[0044] The beneficial effects of the present invention are:

[0045] 1. It is convenient to realize peak regulation and over-response limitation of differential nodes in the distributed control process, which improves the safety and economy of the system.

[0046] 2. Deep neural networks are used to train historical data. The training results can fit the actual situation well. At the same time, the idea of ​​MADDPG algorithm can efficiently and dynamically update the strategies of each player in the multi-party Bayesian game, and achieve Nash equilibrium under incomplete information games.

[0047] 3. The integration of virtual regret minimization improves the low training efficiency in MADDPG, which can effectively save time costs and improve the flexibility of control. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is the flow chart of the MADDPG algorithm of the present invention.

[0049] Figure 2 This is the model framework diagram of the present invention based on MADDPG.

[0050] Figure 3 This is a diagram of the 17-node microgrid test system of the present invention.

[0051] Figure 4 This is a diagram of auxiliary parameter values ​​for EV control in a 17-node microgrid of the present invention.

[0052] Figure 5 This is a diagram of auxiliary parameter values ​​for the 17-node microgrid SA control of the present invention. DETAILED DESCRIPTION

[0053] Example 1

[0054] According to the actual power system, the problem of microgrid demand over-response is caused by the time difference between the load response action and the real-time electricity price adjustment during the price-based demand response process, which usually leads to over-response. To address the over-response phenomenon, a certain amount of load auxiliary parameters are added to the microgrid demand response model, and a Markov decision model is established. Finally, the MADDPG algorithm is introduced to train the auxiliary parameters to obtain the optimal value of the auxiliary parameters. The auxiliary parameter training method includes the following steps:

[0055] 101: Using the MADDPG algorithm as a blueprint,

[0056] Map the controllable load nodes of the microgrid in the multi-agent environment.

[0057] The algorithm flow is as follows Figure 1 As shown in Figure 2, each microgrid controllable load node is regarded as an independent intelligent agent;

[0058] 102: To achieve the core idea of ​​centralized learning and distributed applications, we built an Actor-Critic architecture. This architecture combines the advantages of Value-Based single-step updates and Policy-Based continuous action support, enhancing the versatility of the method.

[0059] 103: Constructing an Actor Update Method: Based on the improvement of the random strategy algorithm, the gradient formula is extended to the deterministic strategy to establish a value function for each agent, and an experience pool is constructed to store past experience and provide it to the agent for subsequent training;

[0060] 104: Constructing the Critic update method: For N agents, establish a strategy network, a target strategy network, a value network, and a target value network, totaling 4N agents;

[0061] 105: Construct an estimate of the strategies of other agents: Maintain N-1 strategy approximation functions for each agent, whose approximation cost is a logarithmic cost function and adds the entropy of the strategy.

[0062] Example 2

[0063] This embodiment takes a microgrid in a certain area as an example. The solution in Example 1 is further introduced below in combination with a specific calculation formula. See the following description for details:

[0064] 201: Taking each load node in the microgrid as an intelligent agent and the entire microgrid decentralized control process as the environment, a Markov decision model for the microgrid decentralized control demand response is established, and MADDPG is used for training and solving. The workflow is as follows: Figure 2 As shown, the specific process is as follows:

[0065] 1) There are N agents in the environment. Each agent can only observe the local state information of the environment and cannot know the global state. It also contains a neural network with network parameters θ={θ1,…,θ N}, the set of its strategies is π={π1,…,π N}.

[0066] 2) Each agent’s main network consists of two types of networks: one is the policy network μ, which is used to make decisions that are consistent with the current environment and status; the other is the value network Q, which is used to judge the quality of the policy network’s output actions.

[0067] 3) Each time the strategy network is based on the local state information o i Generate action a i After that, the environment will return the reward value r i and the observation value o at the next moment i ', all the agent's information {x,a1,…,aN ,r1,…,r N ,x'} are stored in the experience pool D and wait for the training phase to serve as training samples for the neural network to use.

[0068] 202: To ensure training stability, a target network with a similar structure to the main network is established in the model. The target network consists of a target policy network μ' and a target value network Q'. The target network is initialized using the policy network parameters and value network parameters from the main network, respectively. Its role is to replace the main network in making decisions and judgments at the next moment, and to update the policy network and value network.

[0069] Herein, step 202 includes:

[0070] The policy network is based on the agent’s cumulative expected reward J(θ i ) to update the network function, and the gradient strategy update formula is:

[0071]

[0072]

[0073] In the formula: The network parameters are θ i The expected return gradient of the policy network is used to update the policy network parameters; For the current state x and action set {a1,…,a N The action value function output by the main value network is about action a i gradient; The action given by the policy network is about θ i gradient.

[0074] 2) The value network updates the network parameters by minimizing the TD error. The update formula is:

[0075]

[0076] In the formula: L(θ i ) is the loss function of the value network, which is used to evaluate the error between the value function estimated by the value network and the actual value function; y is the actual action value function, which can be expressed as:

[0077]

[0078] In the formula: γ is the discount factor; γ∈[0,1]; It is the action value function, which is used to evaluate the quality of the subsequent steps and is output by the target value network.

[0079] 3) The training goal of the main network is to maximize the expected return J(μi ), while minimizing the value network loss function L(θ i ). The policy network and value network in the main network are updated with the learning rate α as the update step size. The update formulas are:

[0080]

[0081] 4) The target network adopts the soft update method. The main network parameters are used to update the target network. The update amplitude of the target network is determined by the target network update rate τ, 0≤τ<1. The update formula is

[0082]

[0083] 203: In formula (4), the strategies of other agents are used, which require continuous communication to obtain. In order to optimize the algorithm, the agent can use an approximation related to the agent's true strategy. The approximation cost is a logarithmic cost function, and with the entropy of the strategy, the cost function can be written as:

[0084]

[0085] As long as the above cost function is minimized, we can get an approximation of the strategies of other agents. Therefore, we can replace y in formula (4).

[0086]

[0087] 204: During the training phase, the microgrid dispatch center first explores the behavior of nodes based on their historical data, learning the regulation conditions in the real environment and using a loss function to evaluate the effectiveness of regulation. This information is ultimately stored in an experience pool as training material and sampled for centralized neural network training, accumulating a large amount of prior knowledge.

[0088] 205: During the execution phase, each agent does not need to mobilize its own value network to evaluate the current state and strategy. Instead, it relies on the optimized strategy network in the training phase to achieve multi-party Nash equilibrium, thereby solving the optimal value of the auxiliary parameters.

[0089] Example 3

[0090] This embodiment applies the proposed MADDPG-based node difference parameter value optimization method to Figure 3 A 17-node microgrid test system was established, and the cost reduction effect of node-differentiated parameter values ​​was compared with that of unified parameter values.

[0091] According to the results of the 17-node scenario analysis, due to the strong volatility of new energy, high penetration rate of flexible loads and severe network congestion in scenario 8, the node difference values ​​​​are the most different from the unified values, and the final total cost reduction effect is the best.

[0092] Table 1 Microgrid network parameters

[0093]

[0094]

[0095] Table 2 Distributed generation parameters

[0096]

[0097] In the table, They are the lower limit of active power output, the upper limit of active power output, the lower limit of reactive power output and the upper limit of reactive power output of DG respectively. They are respectively the quadratic term of active cost function, the linear term of active cost function, the quadratic term of reactive cost function and the linear term of reactive cost function.

[0098] Table 3 24-hour non-flexible load of each node

[0099]

[0100]

[0101] Example 4

[0102] Bayesian game solving methods can be broadly divided into two categories. While traditional methods have a sound theoretical foundation and broad applicability, they are demanding of the target game architecture, are highly idealized, are highly conditional, and suffer from poor computability. Machine learning can train intelligent agents using large amounts of historical data. This data-driven algorithm enables the agent to continuously refine the probabilities of the strategies it adopts within its action set based on historical experience, enabling more efficient and intelligent calculation of the optimal decision that represents the best response during the game.

[0103] The following steps are involved:

[0104] 101: Using the MADDPG algorithm as a blueprint, the basic architecture of multi-party Bayesian game is mapped to a multi-agent environment.

[0105] The algorithm flow is as follows Figure 1 As shown, each game participant is regarded as an independent intelligent agent;

[0106] 102: To achieve the core idea of ​​centralized learning and distributed applications, we built an Actor-Critic architecture. This architecture combines the advantages of Value-Based single-step updates and Policy-Based continuous action support, enhancing the versatility of the method.

[0107] 103: Constructing an Actor Update Method: Based on the improvement of the random strategy algorithm, the gradient formula is extended to the deterministic strategy to establish a value function for each agent, and an experience pool is constructed to store past experience and provide it to the agent for subsequent training;

[0108] 104: Constructing the Critic Update Method: For each agent, establish a strategy network, a target strategy network, a value network, and a target value network, which is 4 times the number of agents in total, and introduce the idea of ​​virtual regret minimization. The idea is as follows Figure 2 As shown in Figure 2, the Critic's guidance of the Actor's learning direction is strengthened by minimizing the average regret;

[0109] 105: Construct an estimate of the strategies of other agents. For each agent, maintain n-1 strategy approximation functions, whose approximation cost is a logarithmic cost function and adds the entropy of the strategy.

[0110] Example 5

[0111] This example uses the participation of load aggregators in demand response within a southern industrial park in 2020 as an example to verify the effectiveness of this method. Table 4 shows the average annual declared response capacity and average annual declared response information for all 28 aggregators participating in demand response within the park that year.

[0112] Table 4 Annual average declaration form of a park load aggregator

[0113]

[0114]

[0115] In this example, the park has a total of 28 game participants (aggregators). Each participant in the multi-party Bayesian game is an aggregator within the park. This means that all aggregators within the park compete for demand response revenue in a multi-party Bayesian game. To facilitate the visualization of the effects achieved in this example, other details of the game not relevant to this example have been simplified.

[0116] The data used in this example is the clearing results of all 28 aggregators participating in demand response within the park in 2020, totaling 1,677 demand responses. For verification, 98% of the clearing results from all 1,677 responses were used for training, and the arithmetic mean of the remaining 2% was used as the comparison result.

[0117] (Here, 1677*2%=33.54≈34 times, 1677-34=1643, that is, the results of the 1st to 1643rd times of this year are used for training, and the arithmetic mean of the data of the 1644th to 1677th times is used for verification)

[0118] Table 5 Comparison of the aggregator strategy utility of the mapping operation method using the MADDPG algorithm

[0119]

[0120]

[0121] The foregoing description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by any person skilled in the art within the technical scope disclosed herein and within the spirit and principles of the present invention shall be covered by the scope of protection of the present invention. Furthermore, any matters not described in detail in this specification constitute prior art known to those skilled in the art.

[0122] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

Claims

1. A mapping operation method based on the MADDPG algorithm, characterized in that: The following steps are involved: S1 maps the multi-network microsystem model into the MADDPG multi-agent environment; S2 builds an Actor-Critic framework: combining Value-Based single-step updates and Policy-Based continuous actions; S3 builds the Actor update method: It extends the gradient formula to a deterministic strategy to build a main network for each agent and constructs an experience pool to store past experience for subsequent training of the agent. S4 constructs the critic update method: for N agents, a strategy network, a target strategy network, a value network, and a target value network are established, totaling 4N agents. S5 builds estimates of other agent strategies: for each agent, N-1 strategy approximation functions are maintained, whose approximation cost is a logarithmic cost function, and the entropy of the strategy is added; In step S4, the target network is divided into a target policy network μ' and a target value network Q', which are initialized using the policy network parameters and value network parameters in the main network respectively. Specifically, The S201 policy network is based on the agent's cumulative expected reward J(θ i ) to update the network function, and the gradient strategy update formula 1 and formula 2: Among them, J(θ i ) is the cumulative expected reward of the agent, For state s and action a i The joint expectation of , that is, when the state s changes from the steady-state state distribution ρ under the policy π π Medium sampling, action a i The expectation generated by the policy πθ of agent i, is the cumulative discounted reward of agent i, γ t is the t-th power of the discount factor, used to decay future rewards, r i,t is the immediate reward obtained by the i-th agent at time t, The network parameters are θ i The expected return gradient of the policy network is used to update the policy network parameters; For the current state x and action set {a1,…,a N The action value function output by the main value network is about action a i gradient; The action given by the policy network is about θ i The gradient of E x,a～D To find the expectation of the state x and action a sampled from the experience replay pool D, a i =μ i (o i ) is the policy network μ i According to the observed quantity o i generated actions; The S202 value network updates the network parameters by minimizing the TD error, updating formula 3: Among them, L(θ i ) is the loss function of the value network, which is used to evaluate the error between the value function estimated by the value network and the actual value function; E x,a,r,x' To find the expectation of the transfer sample (x, a, r, x') sampled in the experience replay pool, y is the actual action value function, which can be expressed as Formula 4: In formula 4: r i is the current reward, a j '=μ j '(o j ) is the target strategy network μ of agent j j 'According to the observation quantity o j The generated action, γ is the discount factor; γ∈[0,1]; is the action value function, which is used to evaluate the quality of the subsequent steps and is output by the target value network; The training goal of the S203 main network is to maximize the expected return J(μ i ), while minimizing the value network loss function L(θ i ), the policy network and value network in the main network are updated with the learning rate α as the update step, and the update is obtained as formula 5: Among them, μ i is the policy network parameter of agent i, θ i is the value network parameter of agent i, α is the learning rate, is the strategy objective function J(μ i ) about the parameter θ i The gradient of is used to guide the direction of parameter update; In step S204, the target network is updated using a soft update method. The main network parameters are used to update the target network. The update amplitude of the target network is determined by the target network update rate τ, 0≤τ<1. The updated formula 6 is: Among them, θ' is the parameter of the target value network, μ' is the parameter of the target policy network, θ is the parameter of the value network in the main network, μ is the parameter of the policy network in the main network, and τ is the target network update rate.

2. A mapping operation method based on the MADDPG algorithm according to claim 1, characterized in that: In step S1, the multi-grid microsystem model is: microgrid controllable load nodes or multi-party Bayesian game.

3. A mapping operation method based on the MADDPG algorithm according to claim 1, characterized in that: In step S1, the MADDPG algorithm includes N agents, each of which can only observe the local state information of the environment and cannot know the global state. It also contains a neural network with network parameters θ={θ1,…,θ N }, the set of its strategies is π={π1,…,π N }; The main network of the agent consists of two networks: one is the policy network μ, which is used to make decisions that are consistent with the current environment and status; the other is the value network Q, which is used to judge the quality of the output actions of the policy network; The policy network is based on the local state information o i Generate action a i After that, the environment will return the reward value r i and the observation value o at the next moment i ', all the agent's information {x,a1,…,a N ,r1,…,r N ,x'} are stored in the experience pool D and wait for the training phase as training samples for the neural network to use. x is the current state of the environment, a1,…,a N are the actions of agents 1,…,N, r1,…,r N is the immediate reward value returned by the environment based on the agent's 1,…,N actions, and x' is the new environment state at the next moment.

4. A mapping operation method based on the MADDPG algorithm according to claim 1, characterized in that: Equation 4 also includes the strategies of other agents, which are obtained through communication, and the agent uses an approximation related to the agent's true strategy To approximate, the approximation cost is the logarithmic cost function, plus the entropy of the strategy, the cost function is formula 7: in, is the cost function of the strategy approximation of agent i, is the observation o of other agents j and action a j Find the expected joint distribution of is the approximation of agent i’s strategy to other agents, i.e., given observation o j When agent i predicts that other agents will take action a j The probability distribution of Approximate strategy The entropy is used to encourage the randomness of the strategy; λ is the weight coefficient of entropy regularization, which is used to balance the accuracy and exploration of the strategy, minimize the above cost function, and obtain the approximation of other agent strategies, which is used to replace y in Formula 4 to obtain Formula 8: in, is the corrected target value, r i is the immediate reward obtained by agent i at the current moment, is the target value function of agent i, is the approximation of agent i’s strategy to agent 1, o1,…,o N is the observation quantity of agent 1,…,N, is the approximation of agent i’s strategy to agent N.

5. A mapping operation method based on the MADDPG algorithm as claimed in claim 3, characterized in that: During the training phase, the microgrid dispatching center first conducts behavioral exploration based on the historical data of the nodes, learns the regulation situation in the real environment, and uses the loss function to evaluate the regulation. The information is stored in the experience pool as training material, and participates in the centralized training of the neural network through sampling.

6. A mapping operation method based on the MADDPG algorithm according to claim 1, characterized in that: The intelligent agents do not need to mobilize their own value networks to evaluate the current state and strategy. Instead, they rely on the optimized strategy network in the training phase to achieve multi-party Nash equilibrium, thereby solving the optimal values ​​of auxiliary parameters.

Images