Multi-agent strategic function fitting method and system in power dispatching task

Abstract

The invention belongs to the field of power system dispatching, and particularly relates to a multi-agent strategic function fitting method and system in a power dispatching task. A multi-agent environment is constructed, and an agent strategy is initialized; through multi-agent deep reinforcement learning, the cooperation and game relationship between the agents is considered, and effective coordination between the agents is realized; multi-source heterogeneous data are fused, and the data utilization rate and decision reliability are improved. According to the multi-agent strategic function fitting method in the power dispatching task, the defects of a traditional method in the aspects of agent collaboration and data fusion are overcome, and a more intelligent and efficient solution is provided for power dispatching.

Description

Technical Field

[0001] The invention belongs to the field of power system dispatching, and specifically relates to a multi-agent strategic function fitting method and system in a power dispatching task. Background Art

[0002] Power dispatching is the key to the stable operation of the power system, involving the effective configuration and optimization of power equipment and resources. Traditional methods mainly rely on manual experience and preset rules, which are difficult to cope with complex and changeable power demand and system status. With the development of artificial intelligence and machine learning technology, multi-agent deep reinforcement learning has gradually become an important means of optimizing power dispatching. However, existing methods mostly focus on the optimization of a single agent, and fail to fully consider the coordination and game between agents, resulting in poor dispatching results. In a multi-agent system, each agent makes autonomous decisions, the strategy coordination is complex, and there is a lack of overall planning methods. In addition, the data sources of the power system are diverse and heterogeneous. How to effectively integrate and utilize these data to improve the accuracy and reliability of dispatching decisions is still a problem that needs to be solved.

[0003] In the invention patent with the announcement number CN118378866A and the invention name "Method and system for constructing an intelligent agent and optimization method and system for energy storage system" announced on July 23, 2024, a method for constructing an intelligent agent is disclosed. The patent provides a method and system for constructing an intelligent agent for optimizing the scheduling of a new energy storage system. It adopts a two-layer deep reinforcement learning model framework, including a main strategy network, a main Q network, a target strategy network, a target Q network and an experience replay pool. The intelligent agent is trained by obtaining historical data and prediction data at multiple time scales, and finally the target intelligent agent is obtained. The intelligent agent constructed by this method can balance the daytime load and intraday power fluctuations of abundant and dry days during the week to a certain extent, and improve the renewable energy absorption capacity. However, the patent mainly focuses on constructing a single two-layer deep reinforcement learning intelligent agent, without considering the coordination and optimization issues between multiple intelligent agents. The flexibility and adaptability of the overall system need to be improved.

[0004] In the invention patent with announcement number CN118278495A and invention name "Generation method of power grid operation mode based on reinforcement learning" announced on July 2, 2024, a method for generating power grid operation mode based on reinforcement learning is disclosed. The patent uses Markov decision process to describe the interaction between the intelligent agent and the environment, and uses the SARSA algorithm to generate the power grid operation mode. The typical operation type is identified through z-score standardization, principal component analysis and DBSCAN clustering, and finally the intelligent algorithm is input into the artificial intelligence accelerator for complex calculations. This method improves the computational efficiency and accuracy of the power grid operation mode, and to a certain extent realizes the optimal action selection of the intelligent agent in an uncertain environment. However, this method relies on the SARSA algorithm to generate the power grid operation mode, but SARSA is a table-based reinforcement learning algorithm that may perform poorly in high-dimensional and continuous state spaces. Moreover, for complex power grid systems, this algorithm may not be able to fully capture the complexity and dynamics of the system, and thus cannot achieve optimal scheduling. Summary of the invention

[0005] In view of the above-mentioned problems existing in the prior art, the present invention is proposed to improve the intelligence level and overall operating efficiency of power dispatching.

[0006] The purpose of the present invention is to propose a power dispatch optimization method based on multi-agent strategic function fitting in view of the shortcomings of existing power system dispatch methods in terms of agent collaboration and data fusion. The present invention introduces multi-agent deep reinforcement learning and game theory to construct and optimize the strategic function of the agent, so that it can achieve optimal decision-making and collaborative control in a complex power dispatch environment. Through this method, the synergy between agents and the accuracy and efficiency of dispatch decisions are effectively improved, while the efficient fusion of multi-source heterogeneous data is achieved, thereby significantly improving the operating stability and overall efficiency of the power system.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a multi-agent strategic function fitting method in a power dispatching task, comprising: constructing a multi-agent environment and initializing the agent strategy;

[0008] Through multi-agent deep reinforcement learning, the coordination and game relationship between agents is considered to achieve effective coordination between agents;

[0009] Integrate multi-source heterogeneous data to improve data utilization and decision reliability.

[0010] As a preferred solution of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, wherein: the construction of the multi-agent environment includes state space definition, action space definition, and experience replay buffer definition.

[0011] As a preferred solution of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, wherein: the state space definition includes power equipment state, power grid state, environmental state, and time information;

[0012] The action space definition includes power generation adjustment, load adjustment, energy storage system operation, and demand response;

[0013] The experience replay buffer is defined to store experience data and randomly sample small batches of data.

[0014] As a preferred scheme of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, wherein: the initialization of the agent strategy includes: the state space defines all environmental states perceived by the agent, and the action space defines the operations performed by the agent in the state. In the strategy initialization stage, the agent uses the information in the state space and the action space to select appropriate actions in each state.

[0015] As a preferred solution of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, the multi-agent deep reinforcement learning includes the agents evaluating the value of different actions according to the state and gradually optimizing the strategy.

[0016] As a preferred solution of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, the strategies of each agent are integrated through strategic function fitting on the basis of multi-agent deep reinforcement learning, and multiple independent strategy functions are uniformly fitted into a global strategic function to achieve collaborative optimization among agents.

[0017] As a preferred scheme of the multi-agent strategic function fitting method in a power dispatching task described in the present invention, the coordination and game relationship includes: game analysis uses the results of strategic function fitting to study the gains and losses of the agents, find the balance point between the agents under different strategy combinations, and optimize the overall benefits of the system.

[0018] As a preferred solution of the multi-agent strategic function fitting system in a power dispatching task described in the present invention, it includes: an environment construction module, a collaborative learning module, and a data fusion module.

[0019] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of any one of the methods for fitting a multi-agent strategic function in a power dispatching task are implemented.

[0020] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any one of the methods for fitting a multi-agent strategic function in a power dispatching task are implemented.

[0021] Beneficial effects of the present invention: The present invention proposes a multi-agent strategic function fitting method for power dispatching tasks, which greatly overcomes the shortcomings of traditional methods in agent collaboration and data fusion, and provides a more intelligent and efficient solution for power dispatching. First, through multi-agent deep reinforcement learning, the collaborative and game relationship between agents is fully considered, effective coordination between agents is achieved, and the overall dispatching effect is improved. Secondly, through strategic function fitting, the decision-making process of each agent is optimized, making the dispatching decision more accurate and efficient. In addition, this method effectively integrates multi-source heterogeneous data, improves data utilization and decision reliability, provides more intelligent support for power dispatching, and significantly improves the operating stability and efficiency of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 A flowchart of a multi-agent strategic function fitting method in a power dispatching task provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0025] Example 1

[0026] Reference Figure 1 , which is the first embodiment of the present invention, and provides a multi-agent strategic function fitting method in a power dispatching task, such as Figure 1 As shown, including:

[0027] The present invention proposes a power grid dispatch optimization method based on multi-agent strategic function fitting. By constructing a multi-agent environment, the state space of power grid equipment status, load distribution and external environment is defined, and the agent performs dispatch operations by adjusting the power generation power, load distribution and energy storage operation. Using multi-agent deep reinforcement learning, the agent gradually optimizes its strategy by interacting with the environment, and stores relevant experience data for training. The strategies of multiple agents are fitted to construct a unified strategic function to ensure coordination and optimization among agents. The Nash equilibrium strategy of each agent is solved by game theory method to achieve the global optimal dispatch effect.

[0028] 1. Multi-agent environment modeling

[0029] (1) State space definition

[0030] The state space is the set of all possible states that the agent can observe in the environment. In the power dispatch task, the state space design needs to cover multiple factors of the power system. The state space can be defined as a combination of multiple states in order to capture all key factors in the dispatch process. The composition of the state space is as follows:

[0031] Power equipment status: generator set output power, load status, fuel usage, etc.

[0032] Grid status: voltage, current, frequency, load distribution, etc.

[0033] Environmental conditions: temperature, wind speed, humidity, demand forecast, etc.

[0034] Time information: current time, time interval, etc.

[0035] (2) Action Space Definition

[0036] The action space is the set of all possible actions that the agent can take in each state. The action space is defined as follows:

[0037] Power generation adjustment: Increase or decrease the output power of the generator set.

[0038] Load adjustment: Adjust the distribution or priority of loads.

[0039] Energy storage system operation: charging or discharging operation.

[0040] Demand response: Interacting with customers to reduce or increase demand for electricity.

[0041] Each agent i’s action a i can be represented as a vector, where M is the vector dimension,

[0042] a i =(a i1,a i2 ,La iM )

[0043] (3) Experience replay buffer definition

[0044] The experience replay buffer is used to store the agent's experience data during the interaction with the environment, including state, action, reward, and next state. i Indicates the current state, a i Indicates the current action, r i represents the current return, s i+1 Indicates the next state. In order to break the correlation of samples, an experience replay buffer is used to store the experience data of the agent's interaction with the environment. During each training, a small batch of data is randomly sampled from the buffer for update.

[0045] 2. Agent strategy initialization

[0046] Initialize the policy network and Q-value network for each agent, randomly generate the initial strategy, and provide an initial strategy and value evaluation mechanism for each agent, so as to gradually optimize these strategies and value evaluations in the subsequent training process.

[0047] Random strategy initialization

[0048] A random strategy is used to allow the agent to randomly select actions in the initial stage. The random strategy of each agent i can be expressed as:

[0049] a i =random(A i )

[0050] Among them, A i is the action space of agent i, random(A i ) represents randomly sampling an action from the action space.

[0051] (2) Greedy strategy initialization

[0052] The greedy strategy is used to let the agent select random actions with probability ∈ in the initial stage and select the current optimal action with probability 1-∈. The ∈-greedy strategy can be expressed as:

[0053]

[0054] Among them, π(s i ∣θ π ) is the policy network in state s i The optimal action to output is ∈, where ∈ is the exploration rate.

[0055] 3. Multi-agent Deep Reinforcement Learning

[0056] (1) Strategy network and Q-value network design

[0057] Each agent corresponds to a policy network and a Q-value network. The policy network is used to generate the agent's actions, and the Q-value network is used to evaluate the value of the state-action pair. Next, the two networks need to be initialized.

[0058] 1) Initialization of the policy network:

[0059] Input layer: receives the state vector s i .

[0060] Hidden layer: Several fully connected layers, each layer uses ReLU activation function.

[0061] Output layer: Generate action vector a i , using the Tanh activation function.

[0062] 2) Initialization of Q value network:

[0063] Input layer: receives the state vector s i and the action vector a i splicing.

[0064] Hidden layer: Several fully connected layers, each layer uses ReLU activation function.

[0065] Output layer: Generate Q value Q(s i ,a i ), using a linear activation function.

[0066] (2) Policy network and Q-value network training and sampling update

[0067] Each agent updates its strategy based on the Q-value network and the policy network. First, run the initial strategy in the simulation environment to collect the agent's state, action, reward, and next state. Store these data in the experience replay buffer, and then randomly sample a small batch of experience data (s i ,a i ,r i ,s i+1 ).

[0068] Next, the Q value network calculation is performed, and the target Q value can be written as follows:

[0069] y i =r i +γQ′(s i+1 ,π′(s i+1 )|)

[0070] Among them, γ is the discount factor, Q′ is the target Q value network, and π′ is the target policy network.

[0071] The Q-value network loss function is:

[0072]

[0073] Among them, θ Q is the parameter of the Q value network, N is the number of mini-batch samples, and the gradient descent method is used to minimize the loss function and update the parameter θ of the Q value network. Q .

[0074] After the Q value network is trained, the policy network is trained. The calculation formula of the policy gradient is:

[0075]

[0076] Use the gradient ascent method to maximize the objective function and update the policy network parameters.

[0077] (3) Target network parameter soft update

[0078] Use soft update method to periodically update the target Q value network

[0079] θ Q′ ←τθ Q +(1-τ)θ Q′

[0080] Among them, θ Q is the parameter of the current Q value network; θ Q′ is the parameter of the target Q value network; τ is the soft update coefficient.

[0081] Use soft update method to periodically update the target policy value network

[0082] θ π′ ←τθ π +(1-τ)θ π′

[0083] Among them, θ π is the parameter of the current policy network; θ π′ are the parameters of the target policy network; τ is the soft update coefficient.

[0084] 4. Strategy function fitting

[0085] 1) Collect the strategy function data of each agent

[0086] In this step, it is necessary to integrate the strategies of each agent and fit the strategy functions of multiple agents into a unified strategic function. In the process of multi-agent deep reinforcement learning, each agent will learn its own strategy function, that is, the mapping relationship from state to action π i(s i ). Let π i represents the strategy function of the ith agent, s i represents the state of the ith agent, a i Represents the action of the ith agent. The collected data includes state and action pairs:

[0087]

[0088] Among them, N i Represents the number of data samples of the i-th agent.

[0089] 2) Training and fitting model

[0090] Select Multilayer Perceptron (MLP) as the fitting model and use π global represents the strategy function, whose input is the concatenation of the state vectors of all agents s = (s i ,s2,L,s n ), the output is the concatenation of the action vectors of all agents a=(a i ,a2,L,a n ).

[0091] During training, the mean square error (MSE) is used as the loss function:

[0092]

[0093] Among them, θ represents the parameters of the fitted model and N represents the number of training data.

[0094] Update the model fitting parameters, where η represents the learning rate.

[0095]

[0096] 3) Verify and optimize the fitting model

[0097] After training is completed, the fitted model needs to be verified and optimized using an independent validation set. Evaluate the performance of the model and calculate the loss function value L on the validation set val , and adjust the model parameters according to the verification results. After verification, appropriate optimization can be performed to achieve the best model effect.

[0098] The policy functions of each agent generated in the deep reinforcement learning stage provide the data and model basis for strategic function fitting. Through multi-agent deep reinforcement learning, each agent gradually learns and optimizes its policy function in order to select the best action under a specific state. On this basis, "strategic function fitting" integrates the strategies of each agent and unifies multiple independent policy functions into a global strategic function to achieve collaborative optimization among agents.

[0099] In the deep reinforcement learning stage, each agent learns and generates an independent policy function, which records its optimal action selection under different states. The strategic function fitting stage uses the data of these policy functions as training input to build a model for the unified collaborative strategy of multiple agents. Through strategic function fitting, the strategy of each agent is unified in a global function to ensure that the behavior of each agent is coordinated and consistent. The independent strategy generated by deep reinforcement learning is further optimized in the global strategic function, so that the system can achieve the overall optimal scheduling effect in a collaborative way.

[0100] 5. Coordination and game analysis

[0101] In order to achieve the optimal scheduling of the overall system, in addition to strategic function fitting, coordination optimization and game analysis are also required to ensure that each intelligent agent can coordinate actions and jointly achieve the global optimal goal.

[0102] (1) Game model construction

[0103] In a multi-agent system, the interaction between agents can be regarded as a game process. A game model can be established to describe the gains and losses of each agent under different strategies. By solving the game model, a strategy combination that maximizes the overall benefits of the system can be found.

[0104] The strategic function fitting stage integrates the independent strategies of multiple agents into a global strategic function, so that the decisions of each agent can cooperate with each other under a unified framework. Coordination and game analysis analyzes the strategic interactions and possible conflicts between agents on the basis of this global strategic function, ensuring that the strategies of each agent are not only independently optimized but also coordinated with each other. After having a global strategic function, coordination and game analysis introduces game theory models to analyze the distribution of interests, conflicts and cooperative relationships among multiple agents. Game analysis uses the results of strategic function fitting to study the gains and losses of agents, find the balance point between agents under different strategy combinations (such as Nash equilibrium), and thus optimize the overall benefits of the system.

[0105] For this multi-agent system, each agent i has its strategy set Π i and the profit function U i . The game model can be expressed as:

[0106]

[0107] Among them, i represents the strategy space of agent i, U i represents the reward function of agent i, which is defined as,

[0108] U i(π1,π2,...,π n )

[0109] The above formula indicates that in each agent strategy combination (π1,π2,...,π n ), the benefit of agent i under .

[0110] (2) Nash equilibrium solution

[0111] remember is a strategy combination if all agents i satisfy:

[0112]

[0113] That is, when the strategies of other agents remain unchanged, agent i changes its strategy π i No higher benefits can be obtained, and the strategy combination at this time is the Nash equilibrium. There are many algorithms for solving Nash equilibrium. Choose one to solve Nash equilibrium to obtain the strategy combination that maximizes the overall benefits of the system, thereby making the entire power grid dispatching system more intelligent, efficient and accurate.

[0114] The present application aims to realize the fusion of multi-source heterogeneous data and achieve collaborative optimization among intelligent agents by fitting the strategic functions of multiple intelligent agents.

[0115] This application introduces strategic function fitting technology in multi-agent power dispatching to solve the complex problem of strategy coordination between agents and significantly improve the accuracy and reliability of dispatching decisions.

[0116] Example 2

[0117] The second embodiment of the present invention is different from the previous embodiment in that:

[0118] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0120] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0121] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] Example 3

[0123] The third embodiment of the present invention provides a multi-agent strategic function fitting system in power dispatching tasks, which is characterized by including an environment construction module, a collaborative learning module, and a data fusion module.

[0124] The environment construction module builds a multi-agent environment and initializes the agent strategy;

[0125] The collaborative learning module uses multi-agent deep reinforcement learning to consider the collaborative and game relationships between agents and achieve effective coordination between agents.

[0126] The data fusion module integrates multi-source heterogeneous data to improve data utilization and decision reliability.

[0127] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-agent strategic function fitting method in power dispatching tasks, characterized by: include, Build a multi-agent environment and initialize the agent strategy; Through multi-agent deep reinforcement learning, the coordination and game relationship between agents is considered to achieve effective coordination between agents; Integrate multi-source heterogeneous data to improve data utilization and decision reliability.

2. The multi-agent strategic function fitting method in a power dispatching task according to claim 1, characterized in that: The construction of the multi-agent environment includes the definition of state space, action space, and experience replay buffer.

3. The multi-agent strategic function fitting method in a power dispatching task as claimed in claim 2, characterized in that: The state space definition includes power equipment state, power grid state, environmental state, and time information; The action space definition includes power generation adjustment, load adjustment, energy storage system operation, and demand response; The experience replay buffer is defined to store experience data and randomly sample small batches of data.

4. The multi-agent strategic function fitting method in a power dispatching task as claimed in claim 3, characterized in that: The initialization of the agent strategy includes that the state space defines all environmental states perceived by the agent, and the action space defines the operations performed by the agent in the state. In the strategy initialization stage, the agent uses the information in the state space and the action space to select appropriate actions in each state.

5. The multi-agent strategic function fitting method in a power dispatching task according to claim 4, characterized in that: The multi-agent deep reinforcement learning includes the agents evaluating the value of different actions according to the states and gradually optimizing the strategies.

6. The multi-agent strategic function fitting method in a power dispatching task as claimed in claim 5, characterized in that: Based on multi-agent deep reinforcement learning, the strategies of each agent are integrated through strategic function fitting, and multiple independent strategy functions are unified into a global strategic function to achieve collaborative optimization among agents.

7. The multi-agent strategic function fitting method in a power dispatching task according to claim 6, characterized in that: The synergy and game relationship includes that the game analysis uses the results of strategic function fitting to study the gains and losses of the intelligent agents, find the balance point between the various intelligent agents under different strategy combinations, and optimize the overall benefits of the system.

8. A system based on the multi-agent strategic function fitting method in a power dispatching task according to any one of claims 1 to 7, characterized in that: Including environment construction module, collaborative learning module and data fusion module.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images