Progress balance-based agent multi-task parallel training method and system

Abstract

The invention discloses an agent multi-task parallel training method and system based on progress balance, and relates to the technical field of agent training. According to the invention, the intelligent agents are divided into the performer intelligent agent and the learner intelligent agent, so that the training process can be adjusted based on the performer intelligent agent and the learner intelligent agent in combination with the sampling weight of the trajectory in the training process, and the progress-balanced intelligent agent multi-task training is realized; the performance and training speed of the intelligent agent in a multi-task environment is effectively improved, and the control efficiency of a power system is improved. By constructing the multi-task parallel training framework, progress balance is achieved, it is ensured that the intelligent agent corresponding to each system task can have reasonable attention in the training process, and the stability and efficiency of the electric power system are improved. By dynamically adjusting the sampling weight of the intelligent agent and combining a flexible action evaluation algorithm, the balance of the intelligent agent in a multi-task environment is realized, and the control efficiency of a power system is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent agent training, and in particular relates to an intelligent agent multi-task parallel training method and system based on progress balance. Background Art

[0002] With the increasing development and growth of power systems, the complexity and dynamics of power systems are also increasing, which puts higher requirements on the system tasks of power systems, such as load forecasting, power generation scheduling, voltage stability control and fault detection. The intelligent management of these system tasks has become the key to improving the efficiency and stability of power systems.

[0003] At present, the intelligent management of system tasks is usually implemented based on reinforcement learning. However, the current reinforcement learning mainly focuses on a single system task, and a specific intelligent agent is trained for each task, which brings a huge training time burden and model storage burden to the power system. Therefore, multi-task reinforcement learning is gradually emerging, which can autonomously learn and optimize decisions in a dynamic environment, and thus gradually apply to power systems. However, the current multi-task reinforcement learning has different training difficulties for different tasks, and the learning speed is also different, which leads to low control efficiency of the power system and cannot meet the control needs of the power system. Therefore, there is an urgent need for a multi-task parallel training method and system of intelligent agents based on progress balance to solve the defects of the existing technology. Summary of the invention

[0004] The present invention aims to provide a method and system for multi-task parallel training of intelligent agents based on progress balance to solve the above-mentioned technical problems. By constructing intelligent agents for power system tasks and dividing the intelligent agents, progress balance of the multi-task parallel training process of the intelligent agents is achieved, thereby improving the control efficiency of the power system.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a method for parallel training of multi-task agents based on progress balance, comprising:

[0006] Acquire system operation data and system tasks of the power system, construct an intelligent agent corresponding to each system task, and construct a Markov process model of the power system based on the intelligent agent, the system operation data and the system tasks;

[0007] Dividing the agent into a performer agent and a learner agent;

[0008] Acquire the trajectory information generated by the performer agent, store the trajectory information in the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to build a multi-task parallel training framework;

[0009] According to the multi-task parallel training framework, the sampling weight of each performer agent is determined, and the training progress of the performer agent and the learner agent is regulated based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task;

[0010] The power system is regulated according to the first intelligent agent and the system task.

[0011] It can be understood that, compared with the prior art, the present invention divides the agents into performer agents and learner agents by constructing an agent and a Markov process model of the power system, and then can adjust the training process based on the sampling weights of the performer agents and learner agents in combination with the trajectory during the training process, thereby achieving multi-task training of agents with balanced progress, effectively improving the performance and training speed of the agents in a multi-task environment, and improving the control efficiency of the power system. By constructing a multi-task parallel training framework, progress balance is achieved, ensuring that the agents corresponding to each system task can have reasonable attention during the training process, and improving the stability and efficiency of the power system. By dynamically adjusting the sampling weights of the agents and combining the flexible action evaluation algorithm, the balance of the agents in the multi-task environment is achieved, ensuring the control efficiency of the power system.

[0012] As a preferred solution, the intelligent agent corresponding to each system task is constructed, and a Markov process model of the power system is constructed according to the intelligent agent, system operation data and system tasks, specifically including:

[0013] Constructing a simulation model of the power system according to the system operation data of the power system, and constructing an intelligent agent corresponding to each system task in combination with the system tasks;

[0014] Constructing an environment state set, an action set, a reward function and a state transition probability function according to the intelligent agent, system operation data and system tasks;

[0015] Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function;

[0016] According to the training objectives, the set of environmental states, the set of actions, the reward function and the state transition probability function, a Markov process model of the power system is constructed in combination with a preset attenuation factor.

[0017] As a preferred solution, the environment state set, action set, reward function and state transition probability function are constructed according to the intelligent agent, system operation data and system tasks, specifically including:

[0018] The system operation data includes: attribute status, topology status, historical status information, historical action information, power adjustment information and voltage information;

[0019] Constructing an environment state set according to the attribute state, topological state, historical state information and historical action information;

[0020] constructing an action set according to the power adjustment information;

[0021] Obtaining a stability index value of the power system, and constructing a reward function in combination with the voltage information;

[0022] Construct a state transition probability function based on the environment state set and action set.

[0023] As a preferred solution, the state value function of the intelligent agent is constructed according to the environment state set, action set, reward function and state transition probability function, and the training objective of the Markov process model is constructed based on the state value function, which specifically includes:

[0024] Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function;

[0025] The state value function is specifically:

[0026]

[0027] Among them, V(S t ) is the state value, s t represents the set of n state observations of the power system at time t; a t represents the action selected by the power system at time t; π is the strategy, p is the state transition probability, R t+k+1 represents the immediate incentive value of the reward function; τ represents the trajectory set;

[0028] The training objectives of the Markov process model are specifically:

[0029]

[0030] Among them, a is the action of the power system, s and s′ are the states of the power system, p is the state transition probability, r is the voltage reward value of the power system, γ is the attenuation factor, and V(s′) is the state value of state s′.

[0031] As a preferred solution, the step of obtaining the trajectory information generated by the performer agent, storing the trajectory information in the experience pool of the Markov process model, and updating the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework specifically includes:

[0032] Acquire trajectory information generated by the performer agent, and store the trajectory information into an experience pool of a Markov process model;

[0033] Extracting trajectory information from the experience pool according to a preset period to construct an experience sample;

[0034] According to the learner agent, the loss function of the trajectory information is calculated in combination with the experience samples to update the network parameters of the Markov process model to construct a multi-task parallel training framework.

[0035] As a preferred solution, the step of obtaining the trajectory information generated by the performer agent and storing the trajectory information in the experience pool of the Markov process model specifically includes:

[0036] Obtain the trajectory information generated by the performer agent, and store the trajectory information in the experience pool of the Markov process model, wherein the trajectory information is specifically:

[0037]

[0038] Where Γ is the trajectory, s t is the set of n state observations of the power system at time t, a t represents the action selected by the power system at time t, r t Indicates that in state s t The reward after taking the action, s t+1 is the next state, and T is the trajectory length.

[0039] As a preferred solution, the step of extracting trajectory information from the experience pool according to a preset period to construct an experience sample specifically includes:

[0040] Extract trajectory information from the experience pool according to a preset period to construct an experience sample, wherein the experience sample is specifically:

[0041]

[0042] Among them, θ is the empirical sample, s i and i+1 is the state of the power system, a i is the action of the power system, r i For state s i The reward obtained after taking the action, N is the number of extracted trajectories.

[0043] As a preferred solution, the loss function of the trajectory information specifically includes:

[0044]

[0045] Among them, θ and θ′ are network parameters, N is the number of extracted trajectories, and r i For state s i The reward obtained after taking the action, γ is the decay factor, s i and i+1 is the state of the power system, and V is the state value.

[0046] As a preferred solution, the sampling weight of each performer agent is determined according to the multi-task parallel training framework, and the training progress of the performer agent and the learner agent is regulated based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task, specifically including:

[0047] Obtaining a preset round of reward values ​​according to the multi-task parallel training framework, and determining a sampling weight of a performer agent according to the preset round of reward values;

[0048] Determine the number of trajectory extractions of the performer agent according to the sampling weights, and adjust the training progress of the performer agent and the learner agent in combination with a preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task;

[0049] Among them, the hyper parameters of the preset flexible action evaluation algorithm are:

[0050]

[0051] Among them, α t is the hyperparameter at time t, π t is the strategy at time t, s t Represents a set of n state observations of the power system at time t.

[0052] Accordingly, an embodiment of the present invention provides an agent multi-task parallel training system based on progress balance, comprising: a Markov process model building module, an agent partitioning module, a multi-task parallel training framework building module, a parallel training module and a power system control module;

[0053] The Markov process model building module is used to obtain system operation data and system tasks of the power system, build an intelligent agent corresponding to each system task, and build a Markov process model of the power system based on the intelligent agent, system operation data and system tasks;

[0054] The agent division module is used to divide the agent into performer agents and learner agents;

[0055] The multi-task parallel training framework construction module is used to obtain the trajectory information generated by the performer agent, store the trajectory information in the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework;

[0056] The parallel training module is used to determine the sampling weight of each performer agent according to the multi-task parallel training framework, and to regulate the training progress of the performer agent and the learner agent based on the sampling weight and the preset flexible action evaluation algorithm to complete the training of the performer agent and the learner agent and obtain the first agent corresponding to each system task;

[0057] The power system control module is used to control the power system according to the first intelligent agent and system tasks.

[0058] It can be understood that, compared with the existing technology, this system divides the agents into performer agents and learner agents by constructing the agent and Markov process model of the power system, and then can adjust the training process based on the sampling weights of the performer agent and the learner agent in combination with the trajectory during the training process, thereby realizing multi-task training of agents with balanced progress, effectively improving the performance and training speed of the agents in a multi-task environment, and improving the control efficiency of the power system. By constructing a multi-task parallel training framework, progress balance is achieved, ensuring that the agent corresponding to each system task can have reasonable attention during the training process, and improving the stability and efficiency of the power system. By dynamically adjusting the sampling weights of the agents and combining the flexible action evaluation algorithm, the balance of the agents in the multi-task environment is achieved, ensuring the control efficiency of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 : A flowchart of a method for parallel training of multi-task agents based on progress balance provided by an embodiment of the present invention;

[0060] Figure 2 : A flow chart of a multi-task parallel reinforcement learning algorithm for an intelligent agent with balanced progress provided by an embodiment of the present invention;

[0061] Figure 3 : A structural diagram of an intelligent agent multi-task parallel training system based on progress balance provided by an embodiment of the present invention;

[0062] Among them, 201: Markov process model construction module; 202: intelligent agent division module; 203: multi-task parallel training framework construction module; 204: parallel training module; 205: power system control module. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The complexity and dynamics of power systems are constantly increasing, and traditional control and optimization methods can no longer meet the needs of modern power networks. Intelligent management of tasks such as load forecasting, power generation scheduling, voltage stability control, and fault detection in power systems has become the key to improving system efficiency and stability. Current breakthroughs in reinforcement learning research are mainly focused on single tasks. In practical applications, a specific agent needs to be trained for each task, which brings a huge burden of training time and model storage. Therefore, it is urgent to explore ways to improve the ability of reinforcement learning methods to handle multiple tasks. Multi-task reinforcement learning has attracted widespread attention due to its ability to autonomously learn and optimize decisions in a dynamic environment. In multi-task reinforcement learning, traditional Markov decision making is extended to multiple tasks. Through the mutual influence between tasks, the agent can handle multiple different tasks at the same time, collect trajectory data through interaction with the simulation environment, and use this data to update strategies.

[0065] However, there are differences in the difficulty of training multiple tasks in a multi-task environment, and the corresponding learning speeds are also different. It is very easy for a task to make great progress while other tasks are still in the initial exploration stage. When the learning progress of different agents is inconsistent, it may lead to a decline in the overall system performance, and the difficulty of adjusting strategies and resource allocation will increase. In some tasks, there may be mutual dependence or association between tasks. If the progress of some agents is slow, the slower agents will drag down the performance of the entire system, which may affect the learning of other agents, causing problems in the entire power system when performing related tasks, which will extend the overall task completion time and affect efficiency. There are differences in the difficulty of training multiple tasks in a multi-task environment. If they are not controlled during the training process, they cannot meet the requirements of large-scale multi-task agent reinforcement learning training in new power systems.

[0066] Embodiment 1

[0067] Please refer to Figure 1 , which is a step flow chart of a method for parallel training of intelligent multi-tasks based on progress balance provided by an embodiment of the present invention, including steps S101 to S105.

[0068] Step S101: acquiring system operation data and system tasks of the power system, constructing an intelligent agent corresponding to each system task, and constructing a Markov process model of the power system according to the intelligent agent, system operation data and system tasks.

[0069] In this embodiment, the intelligent agent corresponding to each system task is constructed, and a Markov process model of the power system is constructed according to the intelligent agent, system operation data and system tasks, specifically including:

[0070] Constructing a simulation model of the power system according to the system operation data of the power system, and constructing an intelligent agent corresponding to each system task in combination with the system tasks;

[0071] Constructing an environment state set, an action set, a reward function and a state transition probability function according to the intelligent agent, system operation data and system tasks;

[0072] Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function;

[0073] According to the training objectives, the set of environmental states, the set of actions, the reward function and the state transition probability function, a Markov process model of the power system is constructed in combination with a preset attenuation factor.

[0074] In an optional embodiment, constructing a simulation model of the power system according to the system operation data of the power system specifically includes:

[0075] According to the system operation data of the power system, use power system simulation software such as MATLAB / Simulink, PowerWorld, DSP, BPA, etc. to determine the network topology of the power system, establish dynamic models of different types of power generation equipment, and define dynamic models of various types of loads; and then build a simulation model of the power system;

[0076] Among them, the network topology of the power system includes: the location and connection relationship of power plants, substations, transmission lines, distribution lines and load nodes; different types of power generation equipment include: photovoltaics, wind turbines and synchronous generators, etc.; the dynamic models of different types of power generation equipment are used to describe the charging and discharging characteristics, efficiency and state parameters of energy storage equipment (such as batteries, flywheels, etc.); the dynamic models of various types of loads (residential, industrial, commercial loads) include: load characteristics, volatility and response modes.

[0077] In this embodiment, the environment state set, action set, reward function and state transition probability function are constructed according to the agent, system operation data and system tasks, specifically including:

[0078] The system operation data includes: attribute status, topology status, historical status information, historical action information, power adjustment information and voltage information;

[0079] Constructing an environment state set according to the attribute state, topological state, historical state information and historical action information;

[0080] constructing an action set according to the power adjustment information;

[0081] Obtaining a stability index value of the power system, and constructing a reward function in combination with the voltage information;

[0082] Construct a state transition probability function based on the environment state set and action set.

[0083] In an optional embodiment, the environmental state set is S, s t represents the set of n state observations of the power system at time t; specifically, in the online dispatch control target requirements of the power grid, the power grid environment state is the core basis for the power grid operation and dispatch decision. The decision basis for power grid operation and dispatch is summarized into three categories: attribute state, topological structure, and state context (i.e., historical state information). Among them, X represents the attribute state, which characterizes the various attributes of each component in the power system (such as generators, loads, energy storage, and lines, etc.), including active power P t , reactive power Q t , voltage V t and other information such as line damage;

[0084] The expression of the environmental state set is:

[0085] S = [X, G, C],

[0086] X=[P t ,Q t ,other],

[0087] G=[N t ,Et ],

[0088] Among them, G is the topological state of the power system, that is, the topological structure of the power system at time t, including the node characteristic matrix N t and the adjacency matrix E t ; C represents the state context, which includes a sequence of historical states and actions.

[0089] In an optional embodiment, the action set A can be divided into discrete type and continuous type according to the attributes of the action. t represents the action selected by the power system at time t; specifically, in view of the coordinated interaction of heterogeneous devices in the power system under the development environment, the action set A includes the active power adjustment value ΔP of the generator gen , Active power adjustment value ΔP of adjustable load load And the energy storage device charging and discharging power adjustment value ΔP store ;

[0090] The expression of the action set is specifically:

[0091] A=[ΔP gen ,ΔP load ,ΔP store ],

[0092] Among them, ΔP gen is the active power adjustment value, ΔP load is the active power adjustment value of the adjustable load, ΔP store It is the charging and discharging power adjustment value of the energy storage device.

[0093] In an optional embodiment, the reward function is defined as the instantaneous reward value that can be obtained by the power system in a specific state; the expression of the reward function is:

[0094] R t =k f S qt +r t ,

[0095]

[0096] Among them, R t is the reward function, k f and α are positive weight coefficients, S qt is the system stability index, n sub Indicates the number of grid nodes, v i represents the voltage value of node i, represents the voltage upper limit of node i, Represents the lower voltage limit of node i.

[0097] In an optional embodiment, the state transition probability function P:S×A→P(S) is used to represent the state transition probability of the agent, which is a matrix composed of the probability of transferring to any subsequent state after taking a certain action in a specific state of the power system. The expression of the state transition probability function is:

[0098]

[0099] in, is the transition probability.

[0100] In an optional embodiment, the attenuation factor γ∈[0,1] is used to measure the impact of subsequent rewards obtained by the power system on the current state value. Intelligent agents with different goals have different state definitions, action definitions, transfer mechanisms, and reward mechanisms.

[0101] It should be noted that since the instability of the power grid system state is affected by both system load fluctuations and environmental random factors, such as random line disconnection faults and branch load over-limit faults, the state transition probability function takes these factors into account: S t+1 =f(S t ,A t ,v t ), where v t is the system disturbance and environmental random factors, S t+1 Indicates the state at the next moment.

[0102] In this embodiment, the state value function of the agent is constructed according to the environment state set, action set, reward function and state transition probability function, and the training objective of the Markov process model is constructed based on the state value function, specifically including:

[0103] Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function;

[0104] The state value function is specifically:

[0105]

[0106] Among them, V(S t ) is the state value, s t represents the set of n state observations of the power system at time t; a t represents the action selected by the power system at time t; π is the strategy, p is the state transition probability, R t+k+1 represents the immediate incentive value of the reward function; τ represents the trajectory set;

[0107] The training objectives of the Markov process model are specifically:

[0108]

[0109] Among them, a is the action of the power system, s and s′ are the states of the power system, p is the state transition probability, r is the voltage reward value of the power system, γ is the attenuation factor, and V(s′) is the state value of state s′.

[0110] In an optional embodiment, according to the Markov process model, the intelligent agent first randomly explores and collects a certain number of data sets for pre-training, and at the same time initializes the experience pool with the experience data set.

[0111] Step S102: Divide the agent into performer agents and learner agents.

[0112] Step S103: Acquire the trajectory information generated by the performer agent, store the trajectory information into the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework.

[0113] In this embodiment, the step of obtaining the trajectory information generated by the performer agent, storing the trajectory information in the experience pool of the Markov process model, and updating the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework specifically includes:

[0114] Acquire trajectory information generated by the performer agent, and store the trajectory information into an experience pool of a Markov process model;

[0115] Extracting trajectory information from the experience pool according to a preset period to construct an experience sample;

[0116] According to the learner agent, the loss function of the trajectory information is calculated in combination with the experience samples to update the network parameters of the Markov process model to construct a multi-task parallel training framework.

[0117] In this embodiment, the step of obtaining the trajectory information generated by the performer agent and storing the trajectory information in the experience pool of the Markov process model specifically includes:

[0118] Obtain the trajectory information generated by the performer agent, and store the trajectory information in the experience pool of the Markov process model, wherein the trajectory information is specifically:

[0119]

[0120] Where Γ is the trajectory, s tis the set of n state observations of the power system at time t, a t represents the action selected by the power system at time t, r t Indicates that in state s t The reward after taking the action, s t+1 is the next state, and T is the trajectory length.

[0121] In this embodiment, extracting trajectory information from the experience pool according to a preset period to construct an experience sample specifically includes:

[0122] Extract trajectory information from the experience pool according to a preset period to construct an experience sample, wherein the experience sample is specifically:

[0123]

[0124] Among them, θ is the empirical sample, s i and i+1 is the state of the power system, a i is the action of the power system, r i For state s i The reward obtained after taking the action, N is the number of extracted trajectories.

[0125] In this embodiment, the loss function of the trajectory information specifically includes:

[0126]

[0127] Among them, θ and θ′ are network parameters, N is the number of extracted trajectories, and r i For state s i The reward obtained after taking the action, γ is the decay factor, s i and i+1 is the state of the power system, and V is the state value.

[0128] It should be noted that in the performer agent, each task corresponds to an agent that interacts with it, and multiple performer agents can collect data in parallel; specifically, each performer agent corresponds to a power system task, such as load forecasting, generation scheduling, frequency control or voltage stability management. Each performer agent produces trajectory information by interacting with its environment, and these trajectories are aggregated into the experience pool for subsequent learning.

[0129] It should be noted that the learner agent periodically randomly extracts a certain number of trajectories from the experience pool to construct mini-batches (experience samples); the reward r iThe design can be based on a variety of factors such as the operating efficiency of the power system, the accuracy of load forecasting or voltage stability. This allows the agent to optimize its strategy in a dynamic environment to adapt to different operating conditions.

[0130] It should be noted that the multi-task parallel training framework includes: Ray framework. The management and resource allocation of learner agents and performer agents are implemented through the Ray framework. As an open source distributed computing framework, Ray allows users to easily parallelize and distribute the execution of Python functions, providing flexible resource management and task scheduling functions for multi-agent systems. In the application of power systems, the Ray framework can effectively coordinate and allocate computing resources to ensure that each agent obtains sufficient support during operation, thereby improving the performance of the overall system. For example, when an agent needs to perform a large amount of calculations, Ray can dynamically allocate additional computing nodes; when the computing load is reduced, resources are automatically recovered. This flexibility is particularly important for responding to dynamic changes in power systems.

[0131] It should be noted that in the training process of the performer agent and the learner agent, there is no order relationship between the performer agent and the learner agent. The learner agent randomly explores and collects a certain number of trajectories before starting training to ensure that the agent can fully adapt to changes and optimize decisions in the dynamic environment of the power system. At the same time, the parameters updated by the learner agent will also be sent to the performer agent through the experience pool to keep the trajectory data fresh; the update process can be expressed as: Through this parallel operation mechanism, the agents can quickly adjust based on the latest environmental information, thereby improving the overall stability and efficiency of the power system. For example, the load forecasting agent can adjust the power generation strategy in time when the power demand changes sharply, while the voltage control agent can quickly respond to load changes to ensure that the voltage level is within a safe range.

[0132] It should be noted that for multi-task reinforcement learning, most of the existing methods are uniform sampling, extracting the same number of trajectories, and making targeted adjustments during specific updates; however, for those tasks that lack exploration, the relative number of samples is the same as other tasks, and simply changing the learning rate or gradient will not lead to significant improvements. Assume that agent A is responsible for load forecasting, the task is relatively simple, and the data is stable; agent B is responsible for voltage stability control, the task is complex, and it is necessary to process multiple dynamic variables in real time. During the training process, agent A learns quickly and accurately predicts the load, while agent B has a slower learning progress due to the complexity of the task and environmental fluctuations. Due to the lag of agent B, the overall system cannot be effectively coordinated and controlled, thereby affecting the stability of the power system. Therefore, based on this defect, the embodiment of the present invention can dynamically adjust the sampling rate of the number of trajectories from different tasks according to the training process through the step S104, so that the tasks with slower training receive more attention during the update.

[0133] Step S104: According to the multi-task parallel training framework, the sampling weight of each performer agent is determined, and the training progress of the performer agent and the learner agent is regulated based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task.

[0134] In this embodiment, the sampling weight of each performer agent is determined according to the multi-task parallel training framework, and the training progress of the performer agent and the learner agent is regulated based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task, specifically including:

[0135] Obtaining a preset round of reward values ​​according to the multi-task parallel training framework, and determining a sampling weight of a performer agent according to the preset round of reward values;

[0136] Determine the number of trajectory extractions of the performer agent according to the sampling weights, and adjust the training progress of the performer agent and the learner agent in combination with a preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task;

[0137] Among them, the hyper parameters of the preset flexible action evaluation algorithm are:

[0138]

[0139] Among them, α t is the hyperparameter at time t, π t is the strategy at time t, s tRepresents a set of n state observations of the power system at time t.

[0140] It should be noted that, through the flexible action evaluation algorithm, after the loss of each system task is obtained, it is multiplied by the α of the corresponding system task to perform a weighted sum operation to balance the training progress of the system task again.

[0141] It should be noted that whether the training of the performer agent and the learner agent is completed is determined based on whether the performer agent is still collecting trajectories.

[0142] In an optional embodiment, the reward value of a preset round is obtained according to the multi-task parallel training framework, and the sampling weight of the performer agent is determined according to the reward value of the preset round, which can be determined by a weight formula, and the weight formula is specifically:

[0143]

[0144] Among them, T i is the weight, μ i represents the average reward of task i in the past preset rounds, t i is the target reward for task i, which can be obtained by training a single-task model or setting task expectations.

[0145] It should be noted that the weight formula reflects the training progress of the current task i through the ratio of the average reward to the expected reward. The obtained weight coefficient is truncated to calculate the ratio of the current reward to the expected reward so that the defined weight value T i It can represent the training progress of the corresponding task.

[0146] In an optional embodiment, the training steps of the performer agent specifically include: two layers of loops, namely an outer loop and an inner loop. The outer loop defines the number of trainings and determines the execution time of the training, and the inner loop is the loop of the system tasks in the environment.

[0147] For each system task in a multi-task environment, the performer agent first synchronizes with the learner agent to ensure the real-time nature of the actor network parameters, then initializes the corresponding system task, uses the actor network to interact with the system task, and sends experience samples to the experience replay pool at each step of the interaction process. After collecting a certain number of trajectories, the weight formula is used to calculate the proportion of each system task in the total number of trajectories in this update process. In order to ensure the stability of the agent training, the weight value obtained by the weight formula is updated in fixed rounds rather than adjusted in real time.

[0148] It should be noted that by weighting the number of sampled trajectories, the agent is able to focus more on tasks that are slower to train, such as voltage control or fault detection, when updating. This approach enables larger parameter updates on these critical tasks, thereby improving the overall performance of the agent in a multi-task power system environment.

[0149] In an optional embodiment, the process of sampling trajectory information and updating Markov model parameters by the learner agent is controlled by a layer of loop. The learner agent samples trajectories from the experience replay pool according to the weights of the weight formula. When calculating the loss function of the trajectory, the entropy of the output corresponding to each system task is combined to increase the proportion of the loss function of tasks with slower training speed in the total loss and reduce the proportion of tasks with faster speed.

[0150] Step S105: regulating the power system according to the first intelligent agent and the system task.

[0151] In an optional embodiment, please refer to the figure, which is a flow chart of a progress-balanced agent multi-task parallel reinforcement learning algorithm provided by an embodiment of the present invention. First, a simulation model of a new power system is constructed, and a power grid online dispatching agent model is constructed; each task, the trajectory corresponding to each task and the progress balance weight coefficient are determined according to the agent model, and the trajectory is merged into the experience pool. Through iterative training, the loss of each task is calculated, the loss of each task is multiplied by the weighted sum of the hyperparameters of the corresponding task, the overall loss function is calculated, and the performer and learner network parameters are adjusted according to the loss function.

[0152] This embodiment constructs an intelligent agent and Markov process model of the power system, divides the intelligent agent into a performer agent and a learner agent, and can adjust the training process based on the sampling weights of the performer agent and the learner agent in combination with the trajectory during the training process, thereby achieving multi-task training of intelligent agents with balanced progress, effectively improving the performance and training speed of the intelligent agents in a multi-task environment, and improving the control efficiency of the power system. By constructing a multi-task parallel training framework, progress balance is achieved, ensuring that the intelligent agent corresponding to each system task can have reasonable attention during the training process, thereby improving the stability and efficiency of the power system. By dynamically adjusting the sampling weights of the intelligent agent and combining the flexible action evaluation algorithm, the balance of the intelligent agent in the multi-task environment is achieved, ensuring the control efficiency of the power system.

[0153] Embodiment 2

[0154] Please refer to Figure 3, which is a structural diagram of an intelligent agent multi-task parallel training system based on progress balance provided in an embodiment of the present invention, including: a Markov process model construction module 201, an intelligent agent partitioning module 202, a multi-task parallel training framework construction module 203, a parallel training module 204 and a power system control module 205.

[0155] The Markov process model building module 201 is used to obtain system operation data and system tasks of the power system, build an intelligent agent corresponding to each system task, and build a Markov process model of the power system based on the intelligent agent, system operation data and system tasks.

[0156] In this embodiment, the Markov process model construction module 201 includes: a Markov process model construction unit;

[0157] The Markov process model building unit is used to build a simulation model of the power system according to the system operation data of the power system, and to build an intelligent agent corresponding to each system task in combination with the system task;

[0158] Constructing an environment state set, an action set, a reward function and a state transition probability function according to the intelligent agent, system operation data and system tasks;

[0159] Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function;

[0160] According to the training objectives, the set of environmental states, the set of actions, the reward function and the state transition probability function, a Markov process model of the power system is constructed in combination with a preset attenuation factor.

[0161] In this embodiment, the Markov process model construction unit includes: a Markov process model construction subunit;

[0162] The Markov process model building subunit is used for the system operation data including: attribute state, topology state, historical state information, historical action information, power adjustment information and voltage information;

[0163] Constructing an environment state set according to the attribute state, topological state, historical state information and historical action information;

[0164] constructing an action set according to the power adjustment information;

[0165] Obtaining a stability index value of the power system, and constructing a reward function in combination with the voltage information;

[0166] Construct a state transition probability function based on the environment state set and action set.

[0167] In this embodiment, the Markov process model construction unit includes: a state value function and training target construction subunit;

[0168] The state value function and training target construction subunit is used to construct the state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and to construct the training target of the Markov process model based on the state value function;

[0169] The state value function is specifically:

[0170]

[0171] Among them, V(S t ) is the state value, s t represents the set of n state observations of the power system at time t; a t represents the action selected by the power system at time t; π is the strategy, p is the state transition probability, R t+k+1 represents the immediate incentive value of the reward function; τ represents the trajectory set;

[0172] The training objectives of the Markov process model are specifically:

[0173]

[0174] Among them, a is the action of the power system, s and s′ are the states of the power system, p is the state transition probability, r is the voltage reward value of the power system, γ is the attenuation factor, and V(s′) is the state value of state s′.

[0175] The agent division module 202 is used to divide the agent into performer agents and learner agents.

[0176] The multi-task parallel training framework construction module 203 is used to obtain the trajectory information generated by the performer agent, store the trajectory information into the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework.

[0177] In this embodiment, the multi-task parallel training framework construction module 203 includes: a multi-task parallel training framework construction unit;

[0178] The multi-task parallel training framework construction unit is used to obtain the trajectory information generated by the performer agent and store the trajectory information into the experience pool of the Markov process model;

[0179] Extracting trajectory information from the experience pool according to a preset period to construct an experience sample;

[0180] According to the learner agent, the loss function of the trajectory information is calculated in combination with the experience samples to update the network parameters of the Markov process model to construct a multi-task parallel training framework.

[0181] In this embodiment, the multi-task parallel training framework construction unit includes: a trajectory information acquisition subunit;

[0182] The trajectory information acquisition subunit is used to acquire the trajectory information generated by the performer agent and store the trajectory information into the experience pool of the Markov process model, wherein the trajectory information is specifically:

[0183]

[0184] Where Γ is the trajectory, s t is the set of n state observations of the power system at time t, a t represents the action selected by the power system at time t, r t Indicates that in state s t The reward after taking the action, s t+1 is the next state, and T is the trajectory length.

[0185] In this embodiment, the multi-task parallel training framework construction unit includes: an experience sample construction subunit;

[0186] The experience sample construction subunit is used to extract trajectory information from the experience pool according to a preset period to construct an experience sample, wherein the experience sample is specifically:

[0187]

[0188] Among them, θ is the empirical sample, s i and i+1 is the state of the power system, a i is the action of the power system, r i For state s i The reward obtained after taking the action, N is the number of extracted trajectories.

[0189] In this embodiment, the multi-task parallel training framework construction unit includes: a loss function construction subunit;

[0190] The loss function construction subunit is used to

[0191] Among them, θ and θ′ are network parameters, N is the number of extracted trajectories, and r i For state s iThe reward obtained after taking the action, γ is the decay factor, s i and i+1 is the state of the power system, and V is the state value.

[0192] The parallel training module 204 is used to determine the sampling weight of each performer agent according to the multi-task parallel training framework, and to regulate the training progress of the performer agent and the learner agent based on the sampling weight and the preset flexible action evaluation algorithm to complete the training of the performer agent and the learner agent and obtain the first agent corresponding to each system task.

[0193] In this embodiment, the parallel training module 204 includes: a parallel training unit;

[0194] The parallel training unit is used to obtain a preset round of reward values ​​according to the multi-task parallel training framework, and determine the sampling weight of the performer agent according to the preset round of reward values;

[0195] Determine the number of trajectory extractions of the performer agent according to the sampling weights, and adjust the training progress of the performer agent and the learner agent in combination with a preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task;

[0196] Among them, the hyper parameters of the preset flexible action evaluation algorithm are:

[0197]

[0198] Among them, α t is the hyperparameter at time t, π t is the strategy at time t, s t Represents a set of n state observations of the power system at time t.

[0199] The power system control module 205 is used to control the power system according to the first intelligent agent and system tasks.

[0200] This embodiment constructs an intelligent agent and Markov process model of the power system, divides the intelligent agent into a performer agent and a learner agent, and can adjust the training process based on the sampling weights of the performer agent and the learner agent in combination with the trajectory during the training process, thereby achieving multi-task training of intelligent agents with balanced progress, effectively improving the performance and training speed of the intelligent agents in a multi-task environment, and improving the control efficiency of the power system. By constructing a multi-task parallel training framework, progress balance is achieved, ensuring that the intelligent agent corresponding to each system task can have reasonable attention during the training process, thereby improving the stability and efficiency of the power system. By dynamically adjusting the sampling weights of the intelligent agent and combining the flexible action evaluation algorithm, the balance of the intelligent agent in the multi-task environment is achieved, ensuring the control efficiency of the power system.

[0201] In summary, the embodiment of the present invention divides the agents into performer agents and learner agents by constructing an agent and a Markov process model of the power system, and then can adjust the training process based on the sampling weights of the performer agents and learner agents in combination with the trajectory during the training process, thereby achieving multi-task training of agents with balanced progress, effectively improving the performance and training speed of the agents in a multi-task environment, and improving the control efficiency of the power system. By constructing a multi-task parallel training framework, progress balance is achieved, ensuring that the agents corresponding to each system task can have reasonable attention during the training process, thereby improving the stability and efficiency of the power system. By dynamically adjusting the sampling weights of the agents and combining the flexible action evaluation algorithm, the balance of the agents in the multi-task environment is achieved, ensuring the control efficiency of the power system.

[0202] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-task parallel training method for intelligent agents based on progress balance, characterized in that: include: Acquire system operation data and system tasks of the power system, construct an intelligent agent corresponding to each system task, and construct a Markov process model of the power system based on the intelligent agent, the system operation data and the system tasks; Dividing the agent into a performer agent and a learner agent; Acquire the trajectory information generated by the performer agent, store the trajectory information in the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to build a multi-task parallel training framework; According to the multi-task parallel training framework, the sampling weight of each performer agent is determined, and the training progress of the performer agent and the learner agent is regulated based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task; The power system is regulated according to the first intelligent agent and the system task.

2. The method for parallel training of multi-task agents based on progress balance according to claim 1, characterized in that: The intelligent agent corresponding to each system task is constructed, and a Markov process model of the power system is constructed according to the intelligent agent, system operation data and system tasks, specifically including: Constructing a simulation model of the power system according to the system operation data of the power system, and constructing an intelligent agent corresponding to each system task in combination with the system tasks; Constructing an environment state set, an action set, a reward function and a state transition probability function according to the intelligent agent, system operation data and system tasks; Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function; According to the training objectives, the set of environmental states, the set of actions, the reward function and the state transition probability function, a Markov process model of the power system is constructed in combination with a preset attenuation factor.

3. The method for parallel training of multi-task agents based on progress balance according to claim 2, characterized in that: The step of constructing an environment state set, an action set, a reward function and a state transition probability function according to the intelligent agent, system operation data and system tasks specifically includes: The system operation data includes: attribute status, topology status, historical status information, historical action information, power adjustment information and voltage information; Constructing an environment state set according to the attribute state, topological state, historical state information and historical action information; Constructing an action set according to the power adjustment information; Obtaining a stability index value of the power system, and constructing a reward function in combination with the voltage information; Construct a state transition probability function based on the environment state set and action set.

4. The method for parallel training of multi-task agents based on progress balance according to claim 2 or 3, characterized in that: The step of constructing a state value function of an intelligent agent according to the set of environmental states, the set of actions, the reward function and the state transition probability function, and constructing a training target of a Markov process model based on the state value function, specifically includes: Constructing a state value function of the intelligent agent according to the environment state set, action set, reward function and state transition probability function, and constructing a training target of the Markov process model based on the state value function; The state value function is specifically: Among them, V(S t ) is the state value, s t represents the set of n state observations of the power system at time t; a t represents the action selected by the power system at time t; π is the strategy, p is the state transition probability, R t+k+1 represents the immediate incentive value of the reward function; τ represents the trajectory set; The training objectives of the Markov process model are specifically: Where a is the action of the power system, s and s ′ is the state of the power system, p is the state transition probability, r is the voltage reward value of the power system, γ is the attenuation factor, V(s ′ ) is state s ′ The status value.

5. The method for parallel training of multi-task agents based on progress balance according to claim 1, characterized in that: The step of obtaining the trajectory information generated by the performer agent, storing the trajectory information in the experience pool of the Markov process model, and updating the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework specifically includes: Acquire trajectory information generated by the performer agent, and store the trajectory information into an experience pool of a Markov process model; Extracting trajectory information from the experience pool according to a preset period to construct an experience sample; According to the learner agent, the loss function of the trajectory information is calculated in combination with the experience samples to update the network parameters of the Markov process model to construct a multi-task parallel training framework.

6. The method for parallel training of multi-task agents based on progress balance according to claim 5, characterized in that: The obtaining of the trajectory information generated by the performer agent and storing the trajectory information in the experience pool of the Markov process model specifically includes: Obtain the trajectory information generated by the performer agent, and store the trajectory information in the experience pool of the Markov process model, wherein the trajectory information is specifically: Where Γ is the trajectory, s t is the set of n state observations of the power system at time t, a t represents the action selected by the power system at time t, r t Indicates that in state s t The reward after taking the action, s t+1 is the next state, and T is the trajectory length.

7. The method for parallel training of multi-task agents based on progress balance according to claim 5, characterized in that: The extracting trajectory information from the experience pool according to a preset period to construct an experience sample specifically includes: Extract trajectory information from the experience pool according to a preset period to construct an experience sample, wherein the experience sample is specifically: in, is the experience sample, s i and i+1 is the state of the power system, a i is the action of the power system, r i For state s i The reward obtained after taking the action, N is the number of extracted trajectories.

8. The method for parallel training of multi-task agents based on progress balance according to claim 5, characterized in that: The loss function of the trajectory information specifically includes: Among them, θ and θ ′ is the network parameter, N is the number of extracted trajectories, r i For state s i The reward obtained after taking the action, γ is the decay factor, s i and i+1 is the state of the power system, and V is the state value.

9. The method for parallel training of multi-task agents based on progress balance according to claim 5, characterized in that: The method of determining the sampling weight of each performer agent according to the multi-task parallel training framework, and regulating the training progress of the performer agent and the learner agent based on the sampling weight and the preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task, specifically includes: Obtaining a preset round of reward values ​​according to the multi-task parallel training framework, and determining a sampling weight of a performer agent according to the preset round of reward values; Determine the number of trajectory extractions of the performer agent according to the sampling weights, and adjust the training progress of the performer agent and the learner agent in combination with a preset flexible action evaluation algorithm to complete the parallel training of the performer agent and the learner agent, and obtain the first agent corresponding to each system task; Among them, the hyper parameters of the preset flexible action evaluation algorithm are: Among them, α t is the hyperparameter at time t, π t is the strategy at time t, s t Represents a set of n state observations of the power system at time t.

10. An agent multi-task parallel training system based on progress balance, characterized in that: include: Markov process model building module, intelligent agent partitioning module, multi-task parallel training framework building module, parallel training module and power system control module; The Markov process model building module is used to obtain system operation data and system tasks of the power system, build an intelligent agent corresponding to each system task, and build a Markov process model of the power system based on the intelligent agent, system operation data and system tasks; The agent division module is used to divide the agent into performer agents and learner agents; The multi-task parallel training framework construction module is used to obtain the trajectory information generated by the performer agent, store the trajectory information in the experience pool of the Markov process model, and update the network parameters of the Markov process model according to the trajectory information and the learner agent to construct a multi-task parallel training framework; The parallel training module is used to determine the sampling weight of each performer agent according to the multi-task parallel training framework, and to regulate the training progress of the performer agent and the learner agent based on the sampling weight and the preset flexible action evaluation algorithm to complete the training of the performer agent and the learner agent and obtain the first agent corresponding to each system task; The power system control module is used to control the power system according to the first intelligent agent and system tasks.

Images