Multi-scene non-direct-control resource scheduling method based on multi-agent reinforcement learning

Abstract

The invention discloses a multi-agent reinforcement learning-based multi-scene non-direct-control resource scheduling method, the method is applied to a resource management system associated with a power distribution network, the resource management system comprises a control device and at least one management device loaded with an agent, and the method is executed by the control device. Comprising the following steps: acquiring current operation data and historical operation data generated by non-direct-control resources in a power distribution network under the scheduling of corresponding management equipment; determining a current scene reward weight coefficient according to the current operation data and the historical operation data; based on the current scene reward weight coefficient, determining a corresponding total reward function of the power distribution network in the current scene; through the management equipment, the intelligent agent is updated based on the total reward function, the resource scheduling strategy is generated by adopting the updated intelligent agent, and the non-direct-control resources are scheduled based on the resource scheduling strategy, so that the adaptability and the flexibility of corresponding non-direct-control resource management can be improved.

Description

Technical Field

[0001] This invention relates to the field of power distribution network resource management technology, and in particular to a multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning. Background Technology

[0002] In real-world applications, current power distribution networks are extensively integrated with distributed non-directly controlled resources such as distributed photovoltaic systems, energy storage systems, and electric vehicles. Against this backdrop, utilizing smart agent technology to manage these non-directly controlled resources in the power distribution network has become a crucial means of improving its operational efficiency.

[0003] The existing technical solution is to build corresponding intelligent agents for the non-directly controlled resources of different types of entities in the distribution network (such as self-optimizing resource groups, third-party independent entities, and direct control by large users), and then use multiple intelligent agents to manage the non-directly controlled resources of each type of entity in the distribution network, thereby realizing the scheduling of non-directly controlled resources in the distribution network.

[0004] However, as a power distribution hub, the distribution network undertakes multiple tasks, including renewable energy consumption, grid peak shaving, and congestion management. Under different task scenarios, the scheduling requirements for non-directly controlled resources vary. Existing technical solutions lack scenario adaptability, making it difficult to dynamically meet the scheduling needs of different task scenarios and maintain optimal scheduling performance under changing operating environments. Summary of the Invention

[0005] This invention provides a multi-scenario non-directly controlled resource scheduling method based on multi-agent reinforcement learning. It can solve the problem of lack of scenario adaptability in existing technical solutions by determining the reward weight coefficient of the current scenario in real time and constructing the total reward function of the distribution network, thereby improving the adaptability and flexibility of corresponding non-directly controlled resource management.

[0006] According to a first aspect of the present invention, a multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning is provided. The method is applied to a resource management system associated with a power distribution network. The resource management system includes a control device and at least one management device loaded with agents. The method is executed by the control device and includes:

[0007] Acquire current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of corresponding management equipment;

[0008] Based on the current running data and the historical running data, determine the reward weight coefficient for the current scenario;

[0009] Based on the current scenario reward weight coefficient, determine the total reward function of the distribution network in the current scenario;

[0010] The management device updates the agent based on the total reward function, and uses the updated agent to generate a resource scheduling strategy, which is then used to schedule the non-directly controlled resources.

[0011] According to a second aspect of the present invention, a power distribution network resource management system is provided, characterized in that it comprises: a control device and at least one management device loaded with an intelligent agent;

[0012] The control device includes:

[0013] The acquisition module is used to acquire the current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of the corresponding management equipment;

[0014] The first determining module is used to determine the reward weight coefficient for the current scenario based on the current running data and the historical running data.

[0015] The second determining module is used to determine the total reward function of the distribution network in the current scenario based on the reward weight coefficient of the current scenario;

[0016] The scheduling module is used to update the agent based on the total reward function through the management device, generate a resource scheduling strategy using the updated agent, and schedule the non-directly controlled resources based on the resource scheduling strategy.

[0017] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning according to any embodiment of the present invention.

[0019] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning as described in any embodiment of the present invention.

[0020] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning as described in any embodiment of the present invention.

[0021] The technical solution of this invention can obtain the current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of corresponding management equipment. Then, based on the current and historical operating data, the reward weight coefficient for the current scenario is determined, and based on the reward weight coefficient for the current scenario, the total reward function of the distribution network under the current scenario is determined. Finally, the management equipment updates the agent based on the total reward function, and the updated agent generates a resource scheduling strategy. The non-directly controlled resources are scheduled based on the resource scheduling strategy. By determining the reward weight coefficient for the current scenario in real time and constructing the total reward function of the distribution network, the resource scheduling strategy can be dynamically adjusted according to the current operating scenario. This solves the problem of lack of scenario adaptability in existing technical solutions and improves the adaptability and flexibility of the corresponding non-directly controlled resource management.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a multi-scenario indirect resource scheduling method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is an architecture diagram of a power distribution network resource management system provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the multi-agent reinforcement learning-based multi-scenario non-direct control resource scheduling method according to embodiments of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 This is a flowchart of a multi-scenario non-directly controlled resource scheduling method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where non-directly controlled resources in a distribution network need to be flexibly scheduled according to the current scenario requirements. This method can be executed by a control device, which can be implemented in hardware and / or software and can be configured in a resource management system. Figure 1 As shown, the method includes:

[0031] S101. Obtain the current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of the corresponding management equipment.

[0032] In this context, the distribution network can be defined as a power network system that distributes electrical energy from the transmission network to end users. Indirectly controlled resources are distributed resources within the distribution network that cannot be directly controlled by the distribution network's resource dispatch center but can be adjusted indirectly, such as distributed photovoltaic power generation systems, electric vehicle charging clusters, energy storage systems, and flexible loads. These distributed resources can typically be locally dispatched and managed by intelligent agents deployed in locally located management devices (such as smart terminals, gateways, or edge computing nodes).

[0033] An intelligent agent can generate corresponding scheduling instructions based on the state of non-directly controlled resources. Optionally, the intelligent agent may include a state space, an action space, a reward function, a policy network, and a value network. The state space can be a set of distribution network operation data observed by the intelligent agent, including but not limited to variables reflecting the real-time operation of the system such as node electrical parameters, equipment operating status, load demand, and renewable energy output. The action space can be a set of control operations that the intelligent agent can execute, including control instructions for various non-directly controlled resources, such as charging and discharging power setpoints and output adjustment instructions. The reward function can be used to evaluate the effectiveness of the corresponding actions taken by the intelligent agent under the distribution network operating state at various times. The policy network can be a decision model that maps states to actions, used to generate resource scheduling strategies. The value network can be used to provide guidance for policy updates.

[0034] Understandably, the state space allows the agent to perceive its environment. The action space allows the agent to perform relevant actions within that perceived environment. The reward function guides the agent's actions. The policy network is responsible for making decisions based on the current situation. And the value network is used to evaluate the long-term value of those decisions.

[0035] In addition, it should be noted that based on the subject type of distributed resources, various distributed resources can be divided into different types of non-directly controlled resources. Non-directly controlled resources of the same type can be managed or scheduled by pre-built corresponding intelligent agents. For example, distributed resources such as photovoltaic power generation, electric vehicle clusters, energy storage systems, and self-optimizing local loads can be classified as non-directly controlled resources of the self-optimizing resource group type, while virtual power plants can be classified as non-directly controlled resources of the third-party independent subject aggregation type, and industrial loads can be classified as non-directly controlled resources of the large user direct control type.

[0036] Furthermore, the distribution network can construct corresponding intelligent agents for non-directly controlled resources of different types of entities (such as self-optimizing resource groups, third-party independent entities, and direct control by large users), and deploy the intelligent agents in the management equipment of the non-directly controlled resources. Each intelligent agent has a corresponding state space, action space, reward function, policy network, and value network, thereby realizing the management of non-directly controlled resources of various types of entities in the distribution network through multiple intelligent agents.

[0037] Current operating data can be real-time collected electrical and / or non-electrical quantity data that reflects the latest operating status of the distribution network, such as node voltage, line power, load, and energy storage status of charge. Historical operating data can be operating status data accumulated over a period of time, such as historical voltage, historical load, and historical photovoltaic power generation.

[0038] It is understandable that the current operating data of the distribution network can be acquired in real time through sensors, measuring devices, smart meters, or communication networks. Management devices equipped with intelligent agents can collect operating data of non-directly controlled resources within the corresponding management scope in real time, store the operating data collected within a certain period of time, and upload it to the control device.

[0039] For example, the control device can communicate with various management devices to obtain current operating data such as load power, line load rate, and total rated capacity of photovoltaic power in the distribution network, as well as historical operating data such as historical photovoltaic power generation and historical load power of each node.

[0040] S102. Determine the reward weight coefficient for the current scenario based on the current running data and the historical running data.

[0041] Among them, the current scenario reward weight coefficient can be used to represent the relative importance of different task scenarios (such as renewable energy consumption, grid peak shaving, and congestion management) in the current operating state of the distribution network.

[0042] It's important to note that in a distribution network, tasks such as renewable energy consumption, grid peak shaving, and mitigating localized congestion can be accomplished through the scheduling of non-directly controlled resources. However, the operating state of a distribution network changes over time, and the relative importance of tasks varies at different times. For example, during the midday peak photovoltaic power generation period, the primary goal of the distribution network might be to maximize renewable energy consumption, incentivize energy storage charging and load consumption; at this time, the task of renewable energy consumption has the highest priority. During the evening peak electricity consumption period, the primary goal of the distribution network might be grid peak shaving to ensure system power balance; at this time, the task of grid peak shaving has the highest priority. When a critical line or transformer in the distribution network approaches its power limit, the primary goal needs to immediately shift to alleviating congestion and ensuring grid safety; at this time, the task of mitigating localized congestion has the highest priority.

[0043] Most existing resource scheduling methods rely on fixed scheduling strategies to schedule multiple non-directly controlled resources. They cannot perceive and adapt to the dynamic changes in power grid operation scenarios, cannot dynamically meet the scheduling needs of different task scenarios, and cannot maintain optimal scheduling performance in changing operating environments.

[0044] Therefore, in order to solve the above problems, this embodiment can adaptively determine the reward weight coefficient of the current scenario, so that the resource management system can dynamically determine the importance of each task scenario.

[0045] For example, this embodiment can use a pre-established evaluation model to analyze current and historical operating data, thereby evaluating the processing priority of each task scenario in the distribution network at the current moment, and setting scenario reward weight coefficients for each task scenario according to the processing priority of the task scenario.

[0046] It is understandable that the reward weight coefficient for the current scenario is dynamically calculated based on the current running data. In practical applications, it can be a parameter that changes over time, thereby enabling adaptive identification and priority processing of tasks with high current importance.

[0047] Optionally, determining the reward weight coefficient for the current scenario based on the current running data and the historical running data may include:

[0048] Based on the current operating data and the historical operating data, calculate the proportion coefficient of each scenario;

[0049] The proportion coefficients of each scenario are normalized to obtain the reward weight coefficient of the current scenario.

[0050] The percentage coefficient can be used to reflect the importance of each task scenario at the current moment.

[0051] For example, consider the scenarios of renewable energy consumption, grid peak shaving, and congestion management. For renewable energy consumption, the proportion coefficient can be calculated using indicators such as the ratio of renewable energy generation power to load power at the corresponding time and the renewable energy power change rate. For grid peak shaving, the proportion coefficient can be calculated using indicators such as the deviation between the current load and the historical average load at the corresponding time and the load change trend. For congestion management, the proportion coefficient can be calculated using indicators such as the line load rate and node voltage deviation at the corresponding time.

[0052] Specifically, the proportions of renewable energy consumption, grid peak shaving, and congestion management can be calculated and determined using the following formula.

[0053] The proportion of new energy consumption It can be:

[0054] ;

[0055] in, It can represent the proportion coefficient of new energy consumption at time t; , , These can be empirical weighting coefficients representing the penetration rate of new energy sources, the ramp-up rate, and the curtailment rate, respectively. It can be a set of distribution network nodes; This can represent the load active power of the i-th node at time t; It can represent the active power of photovoltaic power generation at the i-th node at time t; It can represent the active power of photovoltaic power generation at the i-th node at time t-1; It can represent the total rated capacity of photovoltaic power in the distribution network system; It can represent a time interval; It can represent the rated active power of photovoltaic power generation of the i-th node at time t.

[0056] The proportion of power grid peak shaving It can be:

[0057] ;

[0058] in, It can represent the proportion coefficient of power grid peak shaving at time t; , Empirical weighting coefficients that can represent load levels and load trends respectively; It can be a set of distribution network nodes; It can represent the average active power of the load at the i-th node from time t-24 to time t; This can represent the active power of the load at time t of the i-th node; It can represent the active power of the load at the i-th node at time t-1.

[0059] The proportion of congestion management It can be:

[0060] ;

[0061] in, This can represent the percentage of congestion management at time t; , These can be empirical weighting coefficients that represent line load rate and voltage deviation rate, respectively. This can represent the power of the line between the i-th node and the j-th node at time t; It can represent the maximum power that the line between the i-th node and the j-th node is allowed to operate for a long time. It can represent the voltage amplitude of the i-th node at time t; It can represent the rated voltage of the power distribution network.

[0062] After obtaining the proportion coefficients of each task scenario, they can be converted into current scenario reward weight coefficients with a sum of 1 through normalization.

[0063] S103. Based on the current scenario reward weight coefficient, determine the total reward function corresponding to the distribution network in the current scenario.

[0064] The total reward function can be a reward function corresponding to a resource management system that includes at least one management device. It can be used to guide the agents in each management device to make collaborative decisions and ensure that the resource scheduling strategy meets the needs of the current task scenario.

[0065] For example, the control device can construct a total reward function for a resource management system containing at least one management device based on the current scene reward weight coefficient. Specifically, it can obtain the reward functions of the agents in each management device and sum the products of the reward functions of each agent and the current scene reward weight coefficient as the total reward function.

[0066] Understandably, each agent in a resource scheduling system originally has its own reward function, which guides its local optimization behavior. Constructing the overall reward function is equivalent to establishing a general reward objective at the system level, allowing adjustments to the contribution of each agent's local objective to the overall system objective. For example, when the weighting coefficient for renewable energy consumption is high, agents that effectively promote photovoltaic consumption will receive higher evaluations in the overall reward, thus incentivizing all agents to collaborate and prioritize renewable energy consumption tasks. Therefore, this approach can solve the coordination problem between individual and overall objectives in multi-agent systems, achieving collaborative control among multiple agents.

[0067] S104. The management device updates the agent based on the total reward function, and generates a resource scheduling strategy using the updated agent. The non-directly controlled resources are then scheduled based on the resource scheduling strategy.

[0068] Among them, resource scheduling strategies can be control commands for non-directly controlled resources, such as adjusting power generation output, load demand, or energy storage status.

[0069] For example, the control device can send the total reward function to each management device. The management devices can then optimize and update the agent's reward function and decision network based on the total reward function. After the update, the management devices can use the updated agent to generate corresponding resource scheduling strategies according to the current state of the distribution network and distribute them to the relevant non-directly controlled resources.

[0070] Understandably, the control device can use the issued total reward function as a global command signal to guide the agents in each management device to iterate their strategies. This enables well-trained agents to perceive the power grid status in real time and instantly output the optimal scheduling instructions, thereby achieving precise and adaptive control of non-directly controlled resources.

[0071] The technical solution of this embodiment can obtain the current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of corresponding management equipment. Then, based on the current and historical operating data, the reward weight coefficient of the current scenario is determined, and based on the reward weight coefficient of the current scenario, the total reward function of the distribution network in the current scenario is determined. Finally, the management equipment updates the agent based on the total reward function, and uses the updated agent to generate a resource scheduling strategy. The non-directly controlled resources are scheduled based on the resource scheduling strategy. By determining the reward weight coefficient of the current scenario in real time and constructing the total reward function of the distribution network, the resource scheduling strategy can be dynamically adjusted according to the current operating scenario. This solves the problem of lack of scenario adaptability in existing technical solutions and improves the adaptability and flexibility of the corresponding non-directly controlled resource management.

[0072] Based on the above embodiments, the present invention also provides an optional embodiment, which can further optimize the process of determining the total reward function. The step of determining the total reward function of the distribution network in the current scenario based on the current scenario reward weight coefficient may include:

[0073] Based on the current scene reward weight coefficient, determine the target reward function of the agent in the current scene;

[0074] Based on the target reward function, generate the total reward function corresponding to the distribution network in the current scenario.

[0075] The target reward function can be the agent's expected reward function for the current task scenario. The target reward function can be determined by combining the agent's original reward function with the reward weighting coefficients for the current scenario.

[0076] For example, taking the original reward function as the reward function of the agent corresponding to the self-optimizing resource group and the task scenario at time t as the new energy consumption scenario, the target reward function of the agent corresponding to the self-optimizing resource group in the new energy consumption scenario can be determined by the following formula. :

[0077] ;

[0078] in, It can be expressed as the target reward function of the self-optimizing resource group at time t; It can be a set of nodes in a self-optimizing resource group; This can represent the reward weight coefficient for the new energy consumption scenario at time t; It can represent the purchase and sale price of electricity by the upper-level power grid at time t; It can represent the unit power regulation cost coefficient of the electric vehicle cluster at time t; It can represent the voltage offset penalty cost coefficient; This can represent the rated active power of photovoltaic power generation at the i-th node at time t; This can represent the charging and discharging power of the electric vehicle cluster at time t for the i-th node; This can represent the charging and discharging power of the energy stored at the i-th node at time t; This can represent the local active load of the self-optimizing resource group of the i-th node at time t; It can represent the active power load demand of the electric vehicle cluster at time t for the i-th node; It can represent the voltage amplitude of the i-th node in a self-optimizing resource group at time t; It can represent the reference voltage of the power distribution network system; It can represent the maximum permissible voltage offset rate.

[0079] It should be noted that, in addition to determining the target reward function for the agent corresponding to the self-optimizing resource group in the current scenario, the target reward function for the agent corresponding to other non-directly controlled resources in the distribution network in the current task scenario can also be determined. This embodiment will not elaborate further here.

[0080] Furthermore, after determining the target reward function of each agent in the current scenario, the target reward functions of all agents can be weighted and summed to obtain the total reward function of the power distribution network in the current scenario. The total reward function can guide multiple agents to collaboratively optimize at the system level.

[0081] Based on the above scheme, the step of updating the agent through the management device based on the total reward function may further include:

[0082] The management device is controlled to iteratively train the agent's policy network using a preset policy optimization algorithm with the total reward function as the optimization objective, thereby obtaining the target policy network;

[0083] The agent is updated based on the target policy network and the target reward function.

[0084] The policy optimization algorithm can be a machine learning algorithm used to update the agent's policy network. For example, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. The target policy network can be used to generate the final resource scheduling policy.

[0085] Understandably, the policy network is the agent's decision-making model, used to generate action probabilities based on state inputs. The value network is the agent's evaluation model, used to predict the value of a state.

[0086] For example, the management device can use the total reward function as the optimization objective, and adjust the parameters of the policy network and value network through iterative training to obtain the trained policy network, which serves as the target policy network. Furthermore, updating the agent can be achieved by replacing the agent's policy network and reward function with the target policy network and target reward function, and adjusting the agent's internal parameters to adapt it to the current scenario.

[0087] Optionally, the iterative training of the agent's policy network using a preset policy optimization algorithm may further include:

[0088] Determine the trajectory data generated by the interaction between the intelligent agent and the power distribution network environment;

[0089] Based on the trajectory data, the agent calculates the time difference error and advantage function at each time step;

[0090] Based on the advantage function, the policy network parameters of the agent are updated by pruning the objective function;

[0091] Based on the time difference error, the value network parameters of the agent are updated until the iteration termination condition is met.

[0092] The trajectory data can be a sequence of states, actions, and rewards generated during the execution of actions in the power distribution network environment. It includes information such as the state, actions performed, and rewards obtained by the agent at each moment, and can be used to record the interaction history between the agent and the operating environment of the power distribution network.

[0093] The specific calculation process for time difference error and dominance function can be as follows: first, determine the time difference error based on trajectory data, and then determine the dominance function based on the time difference error.

[0094] Taking a resource scheduling system with three types of agents as an example, the time difference error can be determined by the following formula:

[0095] ;

[0096] Where k∈(1,2,3) This can represent the time difference error of the k-th agent at time t; It can represent the total reward function at time t; It can represent a discount factor; , , These can represent the state spaces of the 1st, 2nd, and 3rd agents at time t, respectively. , , These can represent the state spaces of the 1st, 2nd, and 3rd agents at time t+1, respectively. It can represent the predicted state value of a value network under a given state.

[0097] Accordingly, the dominance function can be determined by the following formula:

[0098] ;

[0099] Where k∈(1,2,3) The advantage function of the k-th agent at time t can be represented; T can represent the total step size of the task. This represents the smoothing coefficient for generalized dominance estimation; This can represent the time difference error of the k-th agent at time t, and correspondingly, It can represent the time difference error of the k-th agent at time t+1, and so on.

[0100] By determining the agent's time difference error and advantage function at each time step, the parameters in the agent's policy network and value network can be further optimized.

[0101] Taking the time difference error and advantage function determined above as an example, the process of optimizing the parameters of the policy network can be as follows:

[0102] 1) Determine the policy adopted by the policy network:

[0103] ;

[0104] in, It can represent the policy adopted by the policy network; The state space of the k-th agent at time t can be represented; This can represent the action space of the k-th agent at time t; It can represent the probability that a policy network will choose a certain action in a certain state.

[0105] 2) Based on the policy adopted by the policy network, the optimization function of the policy network is determined through the pruning function and the dominance function:

[0106] ;

[0107] in, The optimization function of the policy network; Let represent the expectation at time t; It could be the probability ratio between the current policy and the old policy at time t; It can represent the advantage function of the k-th agent at time t; It can represent a clipping function; It can indicate the cropping range.

[0108] 3) Optimize the policy network using its optimization function:

[0109] ;

[0110] in, α can represent the parameters of the policy network for the k-th agent; α can represent the learning rate. It can represent the gradient values ​​of the parameters of the policy network.

[0111] The process of optimizing the parameters of a value network can be as follows:

[0112] 1) Determine the predictive value of the value network in the global state:

[0113] ;

[0114] in, It can represent the predicted value of a value network in a global state (taking three types of agents as examples); , , These can represent the state spaces of the 1st, 2nd, and 3rd agents at time t, respectively. It can represent the predicted state value of a value network under a given state.

[0115] 2) Determine the loss function of the value network based on the predicted value:

[0116] ;

[0117] in, It can represent the loss function of a value network; Let represent the expectation at time t; It can represent the total reward function at time t; It can represent a discount factor.

[0118] 3) Optimize the value network using its damage function:

[0119] ;

[0120] in, α can represent the parameters of the value network; α can represent the learning rate. It can represent the gradient values ​​of the parameters of the value network.

[0121] It should be noted that the above optimization process can be iterative until the iteration termination condition is met. The iteration termination condition can be specifically set according to the actual situation. For example, when the total system reward obtained by the agent no longer increases significantly in several consecutive optimization cycles, and the change range stabilizes within a preset threshold, the iteration can be terminated. Alternatively, the iteration can be terminated when the number of training steps reaches a preset upper limit. This embodiment does not impose specific limitations here.

[0122] Example 2

[0123] Figure 2 This is a schematic diagram of the structure of a power distribution network resource management system provided in Embodiment 2 of the present invention. Figure 2 As shown, the system may include: a control device 21 and at least one management device 22 loaded with an intelligent agent;

[0124] The control device 21 may include:

[0125] The acquisition module can be used to acquire current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of corresponding management equipment;

[0126] The first determining module can be used to determine the reward weight coefficient for the current scenario based on the current running data and the historical running data.

[0127] The second determining module can be used to determine the total reward function of the distribution network in the current scenario based on the reward weight coefficient of the current scenario;

[0128] The control module can be used to control the management device 22, update the agent based on the received total reward function, generate a resource scheduling strategy using the updated agent, and schedule the non-directly controlled resources based on the resource scheduling strategy.

[0129] The technical solution of this embodiment can obtain the current and historical operating data of non-directly controlled resources in the distribution network under the scheduling of corresponding management equipment. Then, based on the current and historical operating data, the reward weight coefficient of the current scenario is determined, and based on the reward weight coefficient of the current scenario, the total reward function of the distribution network in the current scenario is determined. Finally, the management equipment updates the agent based on the total reward function, and uses the updated agent to generate a resource scheduling strategy. The non-directly controlled resources are scheduled based on the resource scheduling strategy. By determining the reward weight coefficient of the current scenario in real time and constructing the total reward function of the distribution network, the resource scheduling strategy can be dynamically adjusted according to the current operating scenario. This solves the problem of lack of scenario adaptability in existing technical solutions and improves the adaptability and flexibility of the corresponding non-directly controlled resource management.

[0130] Optionally, the intelligent agent may include a state space, an action space, a reward function, a policy network, and a value network.

[0131] Optionally, the second determining module can be specifically used to determine the target reward function of the agent in the current scenario based on the current scenario reward weight coefficient;

[0132] Based on the target reward function, generate the total reward function corresponding to the distribution network in the current scenario.

[0133] Optionally, the control module may include: an update unit;

[0134] The update unit can be used to control the management device to iteratively train the agent's policy network using a preset policy optimization algorithm with the total reward function as the optimization objective, so as to obtain the target policy network.

[0135] The agent is updated based on the target policy network and the target reward function.

[0136] Optionally, the updating unit can be specifically used to determine the trajectory data generated by the interaction between the intelligent agent and the power distribution network environment;

[0137] Based on the trajectory data, the agent calculates the time difference error and advantage function at each time step;

[0138] Based on the advantage function, the policy network parameters of the agent are updated by pruning the objective function;

[0139] Based on the time difference error, update the value network parameters of the agent until the iteration termination condition is met.

[0140] Based on the time difference error generated by the trajectory data, the value network parameters of the agent are updated until the iteration termination condition is met.

[0141] Optionally, the first determining module can be specifically used to calculate the proportion coefficient of each scenario based on the current running data and the historical running data;

[0142] The proportion coefficients of each scenario are normalized to obtain the reward weight coefficient of the current scenario.

[0143] The control device in the power distribution network resource management system provided in the embodiments of the present invention can execute the multi-scenario non-direct control resource scheduling method based on multi-agent reinforcement learning provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0144] Example 3

[0145] Figure 3 A schematic diagram of an electronic device 30 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0146] like Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory, such as a read-only memory (ROM) 32 or a random access memory (RAM) 33, communicatively connected to the at least one processor 31. The memory stores computer programs executable by the at least one processor. The processor 31 can perform various appropriate actions and processes based on the computer program stored in the ROM 32 or loaded from storage unit 38 into the RAM 33. The RAM 33 can also store various programs and data required for the operation of the electronic device 30. The processor 31, ROM 32, and RAM 33 are interconnected via a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0147] Multiple components in electronic device 30 are connected to I / O interface 35, including: input unit 36, such as keyboard, mouse, etc.; output unit 37, such as various types of monitors, speakers, etc.; storage unit 38, such as disk, optical disk, etc.; and communication unit 39, such as network card, modem, wireless transceiver, etc. Communication unit 39 allows electronic device 30 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0148] Processor 31 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 31 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 31 performs the various methods and processes described above, such as multi-agent reinforcement learning-based multi-scenario indirect resource scheduling methods.

[0149] In some embodiments, the multi-agent reinforcement learning-based multi-scenario indirect resource scheduling method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 30 via ROM 32 and / or communication unit 39. When the computer program is loaded into RAM 33 and executed by processor 31, one or more steps of the multi-agent reinforcement learning-based multi-scenario indirect resource scheduling method described above can be performed. Alternatively, in other embodiments, processor 31 can be configured to execute the multi-agent reinforcement learning-based multi-scenario indirect resource scheduling method by any other suitable means (e.g., by means of firmware).

[0150] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0154] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0155] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0156] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0157] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-scene non-direct control resource scheduling method based on multi-agent reinforcement learning, characterized in that, The method is applied to a resource management system associated with a power distribution network, the resource management system comprising a control device and at least one management device loaded with an intelligent agent, the method being executed by the control device and comprising: obtaining current operation data and historical operation data generated by non-directly controlled resources in the power distribution network under scheduling of corresponding management devices; determining a current scenario reward weight coefficient according to the current operation data and the historical operation data; determining a total reward function corresponding to the power distribution network in the current scenario based on the current scenario reward weight coefficient; updating the intelligent agent based on the total reward function through the management device, and generating a resource scheduling strategy by using the updated intelligent agent, and scheduling the non-directly controlled resources based on the resource scheduling strategy.

2. The method of claim 1, wherein, The intelligent agent comprises a state space, an action space, a reward function, a policy network and a value network.

3. The method of claim 2, wherein, The determination of the total reward function corresponding to the power distribution network in the current scenario based on the current scenario reward weight coefficient comprises: determining a target reward function of the intelligent agent in the current scenario according to the current scenario reward weight coefficient; generating the total reward function corresponding to the power distribution network in the current scenario according to the target reward function.

4. The method of claim 3, wherein, The updating of the intelligent agent based on the total reward function comprises: controlling the management device to take the total reward function as an optimization target, and iteratively training the policy network of the intelligent agent by using a preset policy optimization algorithm to obtain a target policy network; updating the intelligent agent according to the target policy network and the target reward function.

5. The method of claim 4, wherein, The iteratively training of the policy network of the intelligent agent by using the preset policy optimization algorithm comprises: determining trajectory data generated by interaction between the intelligent agent and the power distribution network environment; calculating a time difference error and an advantage function of the intelligent agent at each time step according to the trajectory data; updating the policy network parameters of the intelligent agent by clipping a target function based on the advantage function; updating the value network parameters of the intelligent agent based on the time difference error until an iteration termination condition is met.

6. The method of claim 1, wherein, The determination of the current scenario reward weight coefficient according to the current operation data and the historical operation data comprises: calculating a proportion coefficient of each scenario based on the current operation data and the historical operation data; normalizing the proportion coefficients of the scenarios to obtain the current scenario reward weight coefficient.

7. A power distribution grid resource management system, characterized by, The method comprises: a control device and at least one management device loaded with an intelligent agent; The control device comprises: an acquisition module configured to acquire current operation data and historical operation data generated by non-directly controlled resources in the power distribution network under scheduling of corresponding management devices; a first determination module configured to determine a current scenario reward weight coefficient according to the current operation data and the historical operation data; a second determination module configured to determine a total reward function corresponding to the power distribution network in the current scenario based on the current scenario reward weight coefficient; A control module is configured to update the agent based on the received total reward function by the management device, and generate a resource scheduling strategy by using the updated agent, and schedule the non-direct control resource based on the resource scheduling strategy.

8. An electronic device, comprising: The electronic device includes: At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-agent reinforcement learning based multi-scenario non-direct control resource scheduling method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to implement the multi-agent reinforcement learning based multi-scenario non-direct control resource scheduling method in any one of claims 1-6 when executed.

10. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program implements the multi-agent reinforcement learning based multi-scenario non-direct control resource scheduling method in any one of claims 1-6 when executed by the processor.

Images