Power system migration reinforcement learning scheduling method based on flexible telescopic network reconstruction

Abstract

The invention relates to a power system migration reinforcement learning scheduling method based on flexible telescopic network reconstruction, and the method comprises the steps: determining a state space, an action space and a target reward function of a target power system scheduling model in a normal topological operation scene of a power grid, establishing a target power system scheduling model according to the state space, the action space and the target reward function; based on a preset depth deterministic gradient algorithm, training a target power system scheduling agent corresponding to the target power system scheduling model; and in a power grid topology change scene, migrating the target power system scheduling agent based on a preset migration learning strategy of flexible telescopic network reconstruction to generate a migration reinforcement learning adaptive scheduling strategy. Therefore, the problems that the adaptability of a scheduling decision model is limited under various operation scenes of a power grid, and the scheduling efficiency and reliability of a power system are difficult to improve in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the technical field of power system dispatching, and in particular to a power system migration reinforcement learning dispatching method based on flexible scalable network reconstruction. Background Art

[0002] With the accelerated advancement of the construction of new power systems, the large-scale access of multiple elements of source, grid, load and storage, the increasing proportion of new energy sources such as wind and solar with random and volatile characteristics, and the complex dispatching environment caused by different topological change scenarios such as rotation stop, fixed inspection, and increase (reduction) of equipment, have brought great challenges to the adaptability and efficiency of dispatching operation decisions. The use of traditional physical models for the preparation and optimization analysis and calculation of dispatching plans is difficult to effectively deal with the uncertainty and randomness problems brought about by the change of power grid topology. When the system calculation scale increases and the system complexity increases dramatically, the calculation efficiency is low and has limitations. On the other hand, the training of reinforcement learning cannot be separated from a large amount of "trial and error" training of the agent and the environment. When the power grid environment is diverse, the reinforcement learning algorithm will have problems such as poor generalization, dimensionality disaster and difficulty in convergence. Transfer learning can use existing knowledge to learn new knowledge faster and improve the adaptability of dispatching agents to different operating scenarios. The combination of transfer learning and reinforcement learning is expected to solve the many limitations of the current reinforcement learning algorithm in complex working conditions such as the widespread access of renewable energy, load changes, and maintenance. It is of great significance to realize the adaptive optimization decision of power grid dispatching under the operation scenarios of source and load fluctuations and topological changes.

[0003] Existing research has applied transfer reinforcement learning to the fields of power system fault identification and control to adapt to the diverse changes in the power system. Some existing technologies can be based on transfer learning technology, and propose text data entity recognition technology for power grid fault handling to achieve decision support for multiple fault types; some existing technologies can integrate deep learning, transfer learning and active learning to achieve transient stability support for power systems under changes in topology and operation mode. In addition, in the field of power dispatching, some existing technologies can also improve the generalization ability of optimized dispatching decisions of regional integrated energy systems by introducing transfer learning. The type of transfer method used determines the effect of improving the training speed of the intelligent agent, which in turn affects the improvement of the generalization ability of decision-making under various scenarios of power system changes.

[0004] Existing research on applying transfer reinforcement learning to forward-looking optimization scheduling of power systems is still preliminary, and there is no relevant research and practical results on efficient neural network migration methods for scenarios with power grid topology changes.

[0005] Among traditional optimization dispatching technologies, the classical analytical method has a good solution effect, but it requires accurate modeling of the power system. When it comes to optimization problems with large system scale and complex dynamic characteristics, modeling and solving are often difficult. Traditional heuristic algorithms, such as particle swarm optimization and genetic algorithms, have a low degree of dependence on mathematical models, but are prone to premature convergence and falling into local optimality, do not have learning and memory, and are difficult to guarantee real-time performance.

[0006] Scheduling optimization methods based on reinforcement learning algorithms are usually aimed at specific scheduling tasks, and have limitations such as high sample complexity and low sample utilization. Changes in learning tasks or environments will cause the trained model to become invalid and the model generalization ability to be poor.

[0007] In summary, the existing technology has problems such as limited adaptability of scheduling decision models under various power grid operation scenarios, which makes it difficult to improve the efficiency and reliability of power system scheduling and needs to be solved urgently. Summary of the invention

[0008] The present application provides a power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction to solve the problems of limited adaptability of scheduling decision models in various power grid operation scenarios in the existing technology and difficulty in improving the efficiency and reliability of power system scheduling.

[0009] The first aspect of the present application provides a method for power system migration reinforcement learning scheduling based on flexible scalable network reconstruction, comprising the following steps: determining the state space, action space and target reward function of the target power system scheduling model under the normal power grid topology operation scenario, so as to establish the target power system scheduling model according to the state space, the action space and the target reward function; based on a preset deep deterministic gradient algorithm, training the target power system scheduling agent corresponding to the target power system scheduling model; in the scenario of power grid topology change, based on the preset flexible scalable network reconstruction migration learning strategy, migrating the target power system scheduling agent to generate a migration reinforcement learning adaptive scheduling strategy.

[0010] Optionally, in one embodiment of the present application, the state space, action space and target reward function of the target power system dispatching model under the normal power grid topology operation scenario are determined, including: obtaining the normalized node loads at all times in the t-period look-forward window in the normal power grid topology operation scenario and the corresponding active output of the previous look-forward window, and constructing the state space according to the node loads and the corresponding active output, wherein t is a positive integer; inputting the state space into the target power system dispatching intelligent agent to obtain the active output of the generator set at all times in the t-period look-forward window corresponding to the state space, so as to construct the active output of the generator set according to the active output of the generator set. The action space; determine the active output and start-stop cost of generator set i at the τth moment in the forward-looking window of the t period, the number of moments contained in the scheduling window corresponding to the target power system scheduling model, and the number of generator sets, so as to determine the main line reward function according to the active output of the generator set, the number of moments, the number of generator sets and the start-stop cost, wherein τ and i are both positive integers; determine the unit output over-limit penalty, unit climbing over-limit penalty, and flow load rate over-limit penalty corresponding to the main line reward function, and construct the target reward function based on the main line reward function, the unit output over-limit penalty, the unit climbing over-limit penalty and the flow load rate over-limit penalty.

[0011] Optionally, in one embodiment of the present application, in the normal power grid topology operation scenario, based on a preset deep deterministic gradient algorithm, the target power system dispatching agent corresponding to the target power system dispatching model is trained, including: constructing the deep deterministic gradient algorithm based on a preset target network and experience replay mechanism, Actor target network, Critic target network and fully connected neural network; based on the Actor main network in the Actor target network, using the dispatching strategy corresponding to the target power system dispatching model to obtain the action corresponding to the current environmental state; evaluating the action through the Critic main network in the Critic target network to obtain the corresponding first evaluation result, and using the first evaluation result to calculate the corresponding first loss, and optimizing the Actor target parameters in the Actor main network to obtain the minimized loss corresponding to the first loss; updating the Acto based on a preset exponential moving average strategy. The optimized Actor target parameters in the r target network are used to obtain an updated Actor target network, and the action corresponding to the next environmental state is obtained through the updated Actor target network according to the strategy corresponding to the next environmental state; the action corresponding to the next environmental state and the next environmental state are input into the Critic target network to output a second evaluation result, and a second loss is calculated according to the first evaluation result and the second evaluation result, and the Critic target parameters in the Critic main network are optimized to obtain the minimized loss corresponding to the second loss; based on the exponential moving average strategy, the optimized Critic target parameters in the Critic target network are updated to obtain an updated Critic target network, and the target power system dispatching agent corresponding to the target power system dispatching model is trained using the updated Critic target network and the updated Actor target network.

[0012] Optionally, in one embodiment of the present application, in the scenario of power grid topology change, based on the preset transfer learning strategy of flexible scalable network reconstruction, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy, including: judging whether the input state and / or output action space corresponding to the power grid topology change scenario has changed; if the input state and / or the output action space corresponding to the power grid topology change scenario is expanded, then the input channel of the target power system dispatching model corresponding to the target power system dispatching agent is increased to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding first migration model; if the input state and / or the output action space corresponding to the power grid topology change scenario is reduced, then the neurons of the target power system dispatching model are shielded to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding second migration model; based on the first migration model or the second migration model, the target power system dispatching agent is subjected to migration training to obtain a target forward-looking dispatching strategy.

[0013] Optionally, in one embodiment of the present application, the expression of the target reward function is:

[0014]

[0015] in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the target reward function for the t-th scheduling window.

[0016] Optionally, in one embodiment of the present application, the expression of the target forward scheduling strategy in the target scenario is:

[0017]

[0018] Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

[0019] The second aspect of the present application provides a power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction, including: a modeling module, used to determine the state space, action space and target reward function of the target power system scheduling model under the normal topology operation scenario of the power grid, so as to establish the target power system scheduling model according to the state space, the action space and the target reward function; a training module, used to train the target power system scheduling agent corresponding to the target power system scheduling model based on a preset deep deterministic gradient algorithm; a migration module, used to migrate the target power system scheduling agent based on a preset flexible scalable network reconstruction migration learning strategy under the scenario of power grid topology change, so as to generate a migration reinforcement learning adaptive scheduling strategy.

[0020] Optionally, in one embodiment of the present application, the modeling module includes: a state space construction unit, which is used to obtain the normalized node loads at all times in the t-period forward-looking window in the normal topology operation scenario of the power grid and the corresponding active output output of the previous forward-looking window, and construct the state space according to the node loads and the corresponding active output, wherein t is a positive integer; an action space construction unit, which is used to input the state space into the target power system dispatching intelligent agent to obtain the active output of the generator set at all times in the t-period forward-looking window corresponding to the state space, so as to construct the action space according to the active output of the generator set; a first function determination unit, which is used Determine the active output and start-stop cost of generator set i at the τth moment in the forward-looking window of the t period, the number of moments contained in the scheduling window corresponding to the target power system scheduling model, and the number of generator sets, so as to determine the main line reward function according to the active output of the generator set, the number of moments, the number of generator sets and the start-stop cost, wherein τ and i are both positive integers; a second function determination unit is used to determine the unit output over-limit penalty, unit climbing over-limit penalty, and flow load rate over-limit penalty corresponding to the main line reward function, and construct the target reward function based on the main line reward function, the unit output over-limit penalty, the unit climbing over-limit penalty and the flow load rate over-limit penalty.

[0021] Optionally, in one embodiment of the present application, the training module includes: a construction unit, which is used to construct the deep deterministic gradient algorithm based on a preset target network and experience replay mechanism, an Actor target network, a Critic target network and a fully connected neural network; an acquisition unit, which is used to obtain an action corresponding to the current environmental state based on the Actor main network in the Actor target network and using the scheduling strategy corresponding to the target power system scheduling model; an optimization unit, which is used to evaluate the action through the Critic main network in the Critic target network to obtain a corresponding first evaluation result, and use the first evaluation result to calculate the corresponding first loss, and optimize the Actor target parameters in the Actor main network to obtain the minimized loss corresponding to the first loss; a first update unit, which is used to update the optimized Actor target parameters in the Actor target network based on a preset exponential moving average strategy, to obtain an updated Actor target network, and obtain the action corresponding to the next environmental state through the updated Actor target network according to the strategy corresponding to the next environmental state; an evaluation unit, used to input the action corresponding to the next environmental state and the next environmental state into the Critic target network to output a second evaluation result, and calculate the second loss according to the first evaluation result and the second evaluation result, and optimize the Critic target parameters in the Critic main network to obtain the minimized loss corresponding to the second loss; a second updating unit, used to update the optimized Critic target parameters in the Critic target network based on the exponential moving average strategy to obtain an updated Critic target network, and use the updated Critic target network and the updated Actor target network to train the target power system dispatching agent corresponding to the target power system dispatching model.

[0022] Optionally, in one embodiment of the present application, the migration module includes: a judgment unit, used to judge whether the input state and / or output action space corresponding to the power grid topology change scenario has changed; a space expansion unit, used to increase the input channel of the target power system dispatching model corresponding to the target power system dispatching agent if the input state and / or the output action space corresponding to the power grid topology change scenario is expanded, so as to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding first migration model; a space reduction unit, used to shield the neurons of the target power system dispatching model if the input state and / or the output action space corresponding to the power grid topology change scenario is reduced, so as to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding second migration model; a scheduling strategy unit, used to perform migration training on the target power system dispatching agent based on the first migration model or the second migration model to obtain a target forward-looking scheduling strategy.

[0023] Optionally, in one embodiment of the present application, the expression of the target reward function is:

[0024]

[0025] in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the target reward function for the t-th scheduling window.

[0026] Optionally, in one embodiment of the present application, the expression of the target forward scheduling strategy in the target scenario is:

[0027]

[0028] Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

[0029] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction as described in the above embodiment.

[0030] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction.

[0031] The fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction.

[0032] Therefore, the embodiments of the present application have the following beneficial effects:

[0033] The embodiments of the present application can determine the state space, action space and target reward function of the target power system dispatching model under the normal topology operation scenario of the power grid, so as to establish the target power system dispatching model according to the state space, action space and target reward function; based on the preset deep deterministic gradient algorithm, the target power system dispatching agent corresponding to the target power system dispatching model is trained; in the scenario of power grid topology change, based on the preset flexible scalable network reconstruction transfer learning strategy, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy. The present application designs a transfer learning method based on flexible scalable network reconstruction, migrates network parameters and network structure, and uses the historical experience and knowledge learned from the source scene to improve the learning efficiency of the agent, so that it can adapt to the dispatching tasks of the target scene more quickly, thereby realizing the transfer of the source scene dispatching knowledge to the operation requirements of the target dispatching scene with topology change. Thus, the problems that the adaptability of the dispatching decision model of the prior art is limited in the diverse operation scenarios of the power grid, and it is difficult to improve the efficiency and reliability of the power system dispatch are solved.

[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 A flowchart of a power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction provided according to an embodiment of the present application;

[0037] Figure 2 A schematic diagram of an input state expansion principle provided for an embodiment of the present application;

[0038] Figure 3 A schematic diagram of an input state reduction principle provided for an embodiment of the present application;

[0039] Figure 4 This is an example diagram of a power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction according to an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0041] Among them, 10-power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction; 100-modeling module, 200-training module, 300-migration module; 501-memory, 502-processor, 503-communication interface. DETAILED DESCRIPTION

[0042] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0043] The following describes the power system migration reinforcement learning scheduling method based on flexible retractable network reconstruction of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a power system migration reinforcement learning scheduling method based on flexible retractable network reconstruction. In this method, the state space, action space and target reward function of the target power system scheduling model under the normal topology operation scenario of the power grid are determined to establish the target power system scheduling model according to the state space, action space and target reward function; based on the preset deep deterministic gradient algorithm, the target power system scheduling agent corresponding to the target power system scheduling model is trained; in the scenario of power grid topology change, based on the preset flexible retractable network reconstruction migration learning strategy, the target power system scheduling agent is migrated to generate a migration reinforcement learning adaptive scheduling strategy. The present application designs a migration learning method based on flexible retractable network reconstruction, migrates network parameters and network structure, and uses the historical experience and knowledge learned from the source scene to improve the learning efficiency of the agent, so that it can adapt to the scheduling task of the target scene more quickly, thereby realizing the transfer of the source scene scheduling knowledge to the operation requirements of the target scheduling scene with topology change. This solves the problems of limited adaptability of the dispatching decision model in various power grid operation scenarios and difficulty in improving the efficiency and reliability of power system dispatching.

[0044] Specifically, Figure 1 A flowchart of a power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction provided in an embodiment of the present application.

[0045] like Figure 1 As shown, the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction includes the following steps:

[0046] In step S101, the state space, action space and target reward function of the target power system dispatching model under the normal power grid topology operation scenario are determined to establish the target power system dispatching model according to the state space, action space and target reward function.

[0047] The embodiments of the present application design the state space, action space and reward function to construct a power system dispatch model based on deep reinforcement learning, thereby designing an interactive environment for the economic dispatch problem of the power system.

[0048] Optionally, in one embodiment of the present application, the state space, action space and target reward function of the target power system dispatching model under the normal power grid topology operation scenario are determined, including: obtaining the normalized node loads at all times in the t-period look-forward window in the normal power grid topology operation scenario and the corresponding active power output of the previous look-forward window, and constructing a state space based on the node loads and the corresponding active power output, wherein t is a positive integer; inputting the state space into the target power system dispatching intelligent agent to obtain the active power output of the generator set at all times in the t-period look-forward window corresponding to the state space, so as to obtain the active power output of the generator set according to the active power output of the generator set. Output constructs an action space; determines the active output and start-stop cost of generator set i at the τth moment in the look-forward window of period t, the number of moments contained in the dispatch window corresponding to the target power system dispatch model, and the number of generator sets, so as to determine the mainline reward function according to the active output of the generator set, the number of moments, the number of generator sets and the start-stop cost, wherein τ and i are both positive integers; determines the unit output over-limit penalty, unit climbing over-limit penalty, and flow load rate over-limit penalty corresponding to the mainline reward function, and constructs the target reward function based on the mainline reward function, the unit output over-limit penalty, the unit climbing over-limit penalty, and the flow load rate over-limit penalty.

[0049] It should be noted that the deep reinforcement learning model of the power system dispatching model based on deep reinforcement learning constructed in the embodiment of the present application is composed of three parts: state set S, action set A and reward R of the previous action; wherein, the state set S is given by the operating environment in which the agent is located; the action set A is the corresponding action of the agent's decision output according to the input state set; R represents the reward for the current action obtained after the corresponding action is fed back to the environment for actual operation. Taking the look-ahead window corresponding to the t period as an example (the training process of the agent can be set to N rounds, each round can contain n time steps, and each time step corresponds to a look-ahead window), the environment provides the observed state S of the system at the current moment t And input to the decision-making agent, the agent decides the state S according to the current strategy tThe corresponding action A t , the action A decided t Feedback to the environment to get the corresponding reward R t , and the environment will update the next state input S t+1 The following is a detailed introduction to the construction of state space, action space and target reward function:

[0050] 1. Design state space:

[0051] Define the state input as (N Bus +N Gen )×T-dimensional vector, including the normalized node loads at all times in the look-ahead window and the corresponding active output of the previous window. The expression is as follows (taking the look-ahead window numbered t as an example, each step of exploration of the agent corresponds to a look-ahead window):

[0052]

[0053] Among them, S t is the state input vector corresponding to the look-ahead window t; T represents the total number of moments contained in a look-ahead window; is the normalized active load vector of the node corresponding to the τth moment in the look-ahead window t, with a dimension of N Bus , that is, the total number of nodes in the system; A t-1 is the normalized active power output vector of the unit corresponding to the previous window of the look-ahead window t, with a dimension of N Gen ×T.

[0054] The calculation formula for standardization is as follows:

[0055]

[0056] in, is the load of the kth node of the system corresponding to the τth time in the forward-looking time window t, and are the maximum and minimum values ​​corresponding to the kth node in the annual load curve, respectively.

[0057] 2. Design action space:

[0058] The action space is represented as follows:

[0059]

[0060] Among them, A t is the action output vector corresponding to the look-ahead window t, It is the output action vector of the agent corresponding to the τth moment in the look-forward window t, with each element in the range of [-1, 1] after the tanh function activation layer, that is, the total number of all generators in the system (traditional thermal power units, new energy units, and balancing units) and the active output of the units decided by the agent at the τth moment in the specific look-forward window t.

[0061] The denormalization formula is as follows:

[0062]

[0063] in, The output action value corresponding to the unit i at the τth moment in the look-ahead window t, which represents the output of the agent neural network in the range of [-1, 1] (it has not been normalized in the past and is not the active output value of the unit i corresponding to the actual moment); represents the active output value of unit i at the τth moment in the actual forward-looking window t after denormalization (unit: MW); i Gmax With P i Gmin Divided into the maximum and minimum allowable active output of unit i (unit: MW).

[0064] 3. Construct reward function:

[0065] Those skilled in the art should understand that in order to achieve the optimization scheduling goal, the design of the reward function needs to convert the tasks and goals expected by the model into the form of a reward function, which is received by the reinforcement learning algorithm to guide the agent to update and optimize the strategy during the interaction with the environment. The performance of the deep reinforcement learning algorithm and the quality of the reward function design jointly determine the final performance of the agent strategy training results.

[0066] The goal of the power system dispatch model based on deep reinforcement learning belongs to the extreme value of quantitative goals, that is, the total operating cost of the system generator sets at all times in each dispatch window corresponding to the dispatch model is minimized. For quantitative tasks, it can be used as the main line reward function by itself or after being processed and transformed in the corresponding form. Therefore, the formula of the reward function is as follows:

[0067]

[0068] in, is the active power output of unit i at the τth moment in the tth look-ahead window, T is the number of moments in a dispatch window, n G is the number of generator sets included in the system; and are the start-up and shutdown costs of unit i at the τth moment in the tth look-forward window, among which the start-up and shutdown costs of thermal power units are not 0, while the start-up and shutdown costs of new energy units and balancing units are always 0; a i is the coefficient of the quadratic term in the power generation cost function of unit i, b i is the coefficient of the first-order term, c i is the constant term coefficient.

[0069] It should be noted that the goal of the power system dispatching model is to minimize the total operating cost of the system generator sets corresponding to all times within the dispatching window, while the training goal of the deep reinforcement learning optimization algorithm is to maximize the discounted sum of the reward signals. Therefore, in the design process of the mainline reward function, the calculation formula of the total operating cost of the units needs to take a negative value as the mainline reward function to guide the training of the intelligent agent.

[0070] In the established deep reinforcement learning power system dispatching model, if only relying on the negative reward of the total cost of power generation, the probability of obtaining the active output of the effective unit through random exploration in the early stage of training is low, and it is difficult for the intelligent agent to find the results that can meet the safety constraints. The signal fed back to the intelligent agent by the single main line reward cannot guide the intelligent agent to explore in the right direction. The efficiency of the training sample is greatly reduced, and the deep reinforcement learning optimization algorithm will face the problem of difficulty in convergence. The above problem is mainly caused by the sparse rewards in machine learning. In order to solve the problem of sparse reward feedback signals, the embodiment of the present application can design a scheduling action penalty term based on the main line reward function, that is, by adding a penalty term to further improve the design of the reward function to obtain the final reward function (i.e., the target reward function), thereby giving the intelligent agent sufficiently strong guidance and improving the convergence performance of the deep reinforcement learning algorithm.

[0071] Optionally, in one embodiment of the present application, the expression of the target reward function is:

[0072]

[0073] in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the target reward function for the t-th scheduling window.

[0074] In the embodiment of the present application, the final reward function consists of a main line reward and three penalty terms, and its expression is as follows:

[0075]

[0076] in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the final reward of the t-th scheduling window.

[0077] Specifically, the following is a detailed description of the unit output over-limit penalty, unit ramp over-limit penalty and power flow load rate over-limit penalty involved in the final reward function:

[0078] 1. Penalty for exceeding the unit output limit:

[0079] The constraint penalty for exceeding the limit of active power output of system units in the corresponding dispatching safety operation target is calculated by calculating the deviation degree between the active power of units exceeding the limit after power flow calculation and their corresponding upper and lower limits determined by the decision-making results of the intelligent agent at all moments in each look-ahead window, and multiplying it by the corresponding weight w1 to guide the intelligent agent to make correct decisions. The expression is as follows:

[0080]

[0081] 2. Penalty for exceeding the limit of unit climbing:

[0082] This corresponds to the active power ramp constraint penalty item of the system thermal power units in the dispatching safety operation target. The intelligent agent is guided to make reasonable decisions by calculating the deviation between the active power change of the thermal power unit with over-limit climbing and its corresponding upper and lower limits, and multiplying it by the corresponding weight w2. The expression is as follows:

[0083]

[0084] Among them, when the calculation result of the active power change of thermal power unit i at the τth moment in the tth dispatch window exceeds the upper or lower climbing power limit, the corresponding limit value is calculated and added to Among them, for different scheduling windows and different moments in the look-ahead window, the calculation formula for the active power change of the thermal power unit is:

[0085]

[0086] 3. Penalty for over-limit of power flow load rate:

[0087] The constraint penalty item corresponding to the system branch flow load rate in the dispatching safety operation target. By calculating the deviation between the branch flow load rate and the corresponding upper limit value when the line is overloaded and multiplying it by the weight w3, the intelligent agent is guided to make reasonable decisions to avoid exceeding the load rate limit. The expression is shown in Eq.

[0088]

[0089] When the load rate calculation result of branch j at the τth time in the tth scheduling window exceeds the upper limit (the upper limit is 1), the over-limit value is calculated and added to middle.

[0090] In step S102, based on a preset deep deterministic gradient algorithm, a target power system dispatching agent corresponding to the target power system dispatching model is trained.

[0091] Furthermore, the embodiments of the present application also need to design a deep deterministic gradient algorithm for scheduling model training to efficiently train the forward-looking economic scheduling agent.

[0092] Optionally, in one embodiment of the present application, in a scenario where the power grid is in normal topology operation, based on a preset deep deterministic gradient algorithm, a target power system dispatching agent corresponding to the target power system dispatching model is trained, including: constructing a deep deterministic gradient algorithm based on a preset target network and experience replay mechanism, an Actor target network, a Critic target network, and a fully connected neural network; based on the Actor main network in the Actor target network, obtaining an action corresponding to the current environmental state using a dispatching strategy corresponding to the target power system dispatching model; evaluating the action through the Critic main network in the Critic target network to obtain a corresponding first evaluation result, and calculating a corresponding first loss using the first evaluation result, and optimizing the Actor target parameters in the Actor main network to obtain a minimized loss corresponding to the first loss; updating the Actor main network based on a preset exponential moving average strategy. The optimized Actor target parameters in the ctor target network are used to obtain an updated Actor target network, and the action corresponding to the next environmental state is obtained through the updated Actor target network according to the strategy corresponding to the next environmental state; the action corresponding to the next environmental state and the next environmental state are input into the Critic target network to output a second evaluation result, and the second loss is calculated according to the first evaluation result and the second evaluation result, and the Critic target parameters in the Critic main network are optimized to obtain the minimized loss corresponding to the second loss; based on the exponential moving average strategy, the optimized Critic target parameters in the Critic target network are updated to obtain an updated Critic target network, and the updated Critic target network and the updated Actor target network are used to train the target power system dispatching agent corresponding to the target power system dispatching model.

[0093] In the specific implementation process, for the multi-period rolling forward scheduling model of the power grid, the action space corresponding to the active output of the decided units is continuous. The embodiment of the present application can select the Deep Deterministic Policy Gradient (DDPG) algorithm as the algorithm for deep reinforcement learning. The DDPG algorithm is based on the target network and experience replay mechanism in Double DQN, and uses two Actor and Critic target networks, which are different from the single Actor and Critic network for delayed update to improve the stability and convergence of the agent training, and the Actor and Critic networks in the DDPG controller both use fully connected neural networks. In addition, the DDPG algorithm also adds a noise sample link before the action output to solve the problem that the algorithm has difficulties in action exploration when facing continuous action space training. The detailed algorithm flow of DDPG is as follows:

[0094] 1. The Actor main network targets the current environment state s ddpg Output action a according to the determined strategy μ ddpg ,Right now:

[0095] a ddpg =μ θ (s ddpg )

[0096] 2. The Critic main network receives and evaluates a ddpg , to get the first evaluation result Q w (s ddpg ,a ddpg ), and then use the first evaluation result to calculate the loss Loss a (i.e. the first loss), and then optimize the parameters θ (i.e. the Actor target parameters) of the Actor main network (strategy) with the goal of minimizing the loss, that is:

[0097] Loss a =-Q w (s ddpg ,a ddpg )

[0098] 3. The embodiment of the present application may use an exponential moving average (soft update) method to update the parameter θ' (i.e., the optimized Actor target parameter) of the Actor target network with a coefficient τ, that is:

[0099] θ t+1 ' = τθ + (1-τ)θ t '

[0100] 4. The Actor target network also targets the next environment state s' ddpg Output action a' according to the determined strategy μ'ddpg ,Right now:

[0101] a' ddpg =μ θ' (s' ddpg )

[0102] 5. Critic target network receives a' ddpg and the next environment state s' ddpg , output the evaluation result (i.e., the second evaluation result) Q w' (s' ddpg ,a' ddpg ), this evaluation result together with the first evaluation result Q w (s ddpg ,a ddpg ) Calculate the loss together c (i.e. the second loss), and then optimize the parameter w of the Critic main network (i.e. the Critic target parameter) with the goal of minimizing the loss, that is:

[0103] Loss c =MSE(Q w (s ddpg ,a ddpg ),r+γQ w' (s' ddpg ,a' ddpg ))

[0104] 6. The exponential moving average method is also used to update the parameter w′ of the Critic target network (that is, the optimized Critic target parameter) with the coefficient τ, that is:

[0105] ω' t+1 =τω+(1-τ)ω' t

[0106] It should be noted that the DDPG algorithm uses an experience cache mechanism to store the experience data (s ddpg,t ,a ddpg,t ,r t ,s ddpg,t+1 ) is stored in the cache, and a certain amount of empirical data is selected each time for network parameter update; in addition, in order to better find the optimal strategy, the embodiment of the present application can also add random noise to the Actor network.

[0107] In step S103, in the scenario of power grid topology change, based on the preset transfer learning strategy of flexible scalable network reconstruction, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy.

[0108] Furthermore, the embodiments of the present application also need to design a transfer learning method based on flexible scalable network reconstruction to adapt to various topology change operating scenarios, improve the learning ability and generalization ability of the intelligent agent in multiple tasks, and thus apply the transfer reinforcement learning algorithm to the forward scheduling of the power system under the topology change scenario, and obtain the transfer reinforcement learning adaptive scheduling strategy.

[0109] Therefore, the embodiments of the present application solve the problem of limited adaptability of the dispatching decision model under various power grid operation scenarios, thereby effectively improving the decision-making speed, reliability, automation and intelligence level of power system dispatching.

[0110] Optionally, in one embodiment of the present application, in a scenario where the power grid topology changes, based on a preset transfer learning strategy for flexible scalable network reconstruction, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy, including: determining whether the input state and / or output action space corresponding to the power grid topology change scenario changes; if the input state and / or output action space corresponding to the power grid topology change scenario expands, then increasing the input channel of the target power system dispatching model corresponding to the target power system dispatching agent to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding first migration model; if the input state and / or output action space corresponding to the power grid topology change scenario shrinks, then shielding the neurons of the target power system dispatching model to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding second migration model; based on the first migration model or the second migration model, performing migration training on the target power system dispatching agent to obtain a target forward-looking dispatching strategy.

[0111] It should be noted that in the face of grid topology change scenarios, the embodiments of the present application can adopt a network reconstruction method based on flexible scalability to achieve network parameter migration in heterogeneous scenarios, and use the historical experience and knowledge learned from the source scenario to improve the learning efficiency of the intelligent agent, so that it can adapt to the scheduling tasks of the target scenario more quickly.

[0112] The embodiments of the present application can redesign the neural network structure of the agent based on the characteristics of the target scene based on the flexible and scalable network reconstruction method, so that the knowledge in heterogeneous scenes can be transferred to each other. The specific neural network model migration method is as follows:

[0113] 1. When the input state of the new scene is expanded, the network can be reconstructed by adding input channels, so that the agent can achieve knowledge inheritance in heterogeneous scenes, such as Figure 2 shown.

[0114] 2. When the input state of the new scene is reduced, the network can be reconstructed by shielding neurons, such as Figure 3 shown.

[0115] 3. When the output action space changes, the network reconstruction strategy of the above input state is still continued.

[0116] Therefore, the embodiments of the present application can achieve knowledge inheritance between heterogeneous scenarios through a network reconstruction method based on flexible scalability, so that the intelligent agent has the ability to adaptively migrate in all-round scenarios.

[0117] To summarize, in operating scenarios under normal power grid topology, the embodiments of the present application can train the intelligent agent source model; in operating scenarios facing changes in power grid topology, the embodiments of the present application can adopt the above-mentioned neural network model migration method to perform migration training on the intelligent agent, so that the model can quickly adapt to the new scenario, in order to obtain a forward-looking scheduling strategy based on transfer reinforcement learning.

[0118] Optionally, in one embodiment of the present application, the expression of the target forward scheduling strategy in the target scenario is:

[0119]

[0120] Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

[0121] It should be noted that the mathematical expression of the output decision (ie, the target forward scheduling strategy) of the embodiment of the present application in the target scenario can be expressed as:

[0122]

[0123] Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

[0124] It should be noted that the above-mentioned source scenario is the scenario when the system is operating normally; the target scenario is the scenario where the power grid topology changes due to rotational shutdown, regular inspection, increase or decrease of equipment, etc.; the source network is the neural network corresponding to the power system dispatching agent under the normal operation scenario.

[0125] Therefore, the embodiments of the present application design a transfer learning method based on flexible scalable network reconstruction, redesign the neural network structure of the intelligent agent according to the characteristics of the target scenario, realize knowledge inheritance between heterogeneous scenarios, and enable the intelligent agent to have adaptive migration capabilities in all-round scenarios; in addition, the embodiments of the present application apply the transfer reinforcement learning algorithm to the forward scheduling of the power system under the scenario of topology change to generate a transfer reinforcement learning adaptive scheduling strategy.

[0126] According to the power system migration reinforcement learning scheduling method based on flexible retractable network reconstruction proposed in the embodiment of the present application, the state space, action space and target reward function of the target power system scheduling model under the normal power grid topology operation scenario are determined to establish the target power system scheduling model according to the state space, action space and target reward function; based on the preset deep deterministic gradient algorithm, the target power system scheduling agent corresponding to the target power system scheduling model is trained; in the scenario of power grid topology change, based on the preset flexible retractable network reconstruction migration learning strategy, the target power system scheduling agent is migrated to generate a migration reinforcement learning adaptive scheduling strategy. This application designs a migration learning method based on flexible retractable network reconstruction, migrates network parameters and network structure, and uses the historical experience and knowledge learned from the source scenario to improve the learning efficiency of the agent, so that it can adapt to the scheduling tasks of the target scenario more quickly, thereby realizing the transfer of the source scenario scheduling knowledge to the operation requirements of the target scheduling scenario with topology changes.

[0127] Secondly, the power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction proposed in accordance with the embodiment of the present application is described with reference to the accompanying drawings.

[0128] Figure 4 It is a block diagram of a power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction according to an embodiment of the present application.

[0129] like Figure 4 As shown, the power system migration reinforcement learning scheduling device 10 based on flexible scalable network reconstruction includes: a modeling module 100, a training module 200 and a migration module 300.

[0130] Among them, the modeling module 100 is used to determine the state space, action space and target reward function of the target power system scheduling model under the normal topology operation scenario of the power grid, so as to establish the target power system scheduling model according to the state space, action space and target reward function.

[0131] The training module 200 is used to train the target power system dispatching agent corresponding to the target power system dispatching model based on a preset deep deterministic gradient algorithm.

[0132] The migration module 300 is used to migrate the target power system dispatching agent based on the preset flexible scalable network reconstruction migration learning strategy in the scenario of power grid topology change, so as to generate a migration reinforcement learning adaptive dispatching strategy.

[0133] Optionally, in one embodiment of the present application, the modeling module 100 includes: a state space construction unit, an action space construction unit, a first function determination unit and a second function determination unit.

[0134] Among them, the state space construction unit is used to obtain the normalized node load at all times in the forward-looking window of the t period in the normal topology operation scenario of the power grid and the corresponding active output of the previous forward-looking window, and construct the state space according to the node load and the corresponding active output, wherein t is a positive integer.

[0135] The action space construction unit is used to input the state space into the target power system dispatching intelligent agent to obtain the active output of the generator set at all times in the t-period look-ahead window corresponding to the state space, so as to construct the action space according to the active output of the generator set.

[0136] The first function determination unit is used to determine the active output and start-stop cost of generator set i at the τth moment in the forward-looking window of time period t, the number of moments contained in the dispatching window corresponding to the target power system dispatching model, and the number of generator sets, so as to determine the main line reward function according to the active output of the generator set, the number of moments, the number of generator sets and the start-stop cost, wherein τ and i are both positive integers.

[0137] The second function determination unit is used to determine the unit output over-limit penalty, unit climbing over-limit penalty, and flow load rate over-limit penalty corresponding to the main line reward function, and construct a target reward function based on the main line reward function, the unit output over-limit penalty, the unit climbing over-limit penalty, and the flow load rate over-limit penalty.

[0138] Optionally, in one embodiment of the present application, the training module 200 includes: a construction unit, an acquisition unit, an optimization unit, a first updating unit, an evaluation unit and a second updating unit.

[0139] Among them, the construction unit is used to build a deep deterministic gradient algorithm based on the preset target network and experience replay mechanism, Actor target network, Critic target network and fully connected neural network.

[0140] The acquisition unit is used to acquire the action corresponding to the current environment state based on the Actor main network in the Actor target network and using the dispatch strategy corresponding to the target power system dispatch model.

[0141] The optimization unit is used to evaluate the action through the Critic main network in the Critic target network to obtain a corresponding first evaluation result, calculate the corresponding first loss using the first evaluation result, and optimize the Actor target parameters in the Actor main network to obtain the minimized loss corresponding to the first loss.

[0142] The first updating unit is used to update the optimized Actor target parameters in the Actor target network based on a preset exponential moving average strategy to obtain an updated Actor target network, and obtain an action corresponding to the next environmental state through the updated Actor target network according to the strategy corresponding to the next environmental state.

[0143] An evaluation unit is used to input the action corresponding to the next environmental state and the next environmental state into the Critic target network to output a second evaluation result, calculate the second loss according to the first evaluation result and the second evaluation result, and optimize the Critic target parameters in the Critic main network to obtain the minimized loss corresponding to the second loss.

[0144] The second updating unit is used to update the optimized critic target parameters in the critic target network based on the exponential moving average strategy to obtain an updated critic target network, and use the updated critic target network and the updated actor target network to train the target power system dispatching intelligent agent corresponding to the target power system dispatching model.

[0145] Optionally, in one embodiment of the present application, the migration module 300 includes: a judgment unit, a space expansion unit, a space reduction unit and a scheduling strategy unit.

[0146] The judgment unit is used to judge whether the input state and / or output action space corresponding to the grid topology change scenario has changed.

[0147] The space expansion unit is used to increase the input channel of the target power system dispatching model corresponding to the target power system dispatching agent if the input state and / or output action space corresponding to the power grid topology change scenario is expanded, so as to reconstruct the neural network corresponding to the target power system dispatching agent and generate the corresponding first migration model.

[0148] The space reduction unit is used to shield the neurons of the target power system scheduling model if the input state and / or output action space corresponding to the power grid topology change scenario is reduced, so as to reconstruct the neural network corresponding to the target power system scheduling agent and generate the corresponding second migration model.

[0149] The dispatching strategy unit is used to perform migration training on the target power system dispatching intelligent agent based on the first migration model or the second migration model to obtain a target forward-looking dispatching strategy.

[0150] Optionally, in one embodiment of the present application, the expression of the target reward function is:

[0151]

[0152] in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the target reward function for the t-th scheduling window.

[0153] Optionally, in one embodiment of the present application, the expression of the target forward scheduling strategy in the target scenario is:

[0154]

[0155] Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

[0156] It should be noted that the aforementioned explanation of the embodiment of the power system migration reinforcement learning scheduling method based on flexible retractable network reconstruction is also applicable to the power system migration reinforcement learning scheduling device based on flexible retractable network reconstruction in this embodiment, and will not be repeated here.

[0157] According to the embodiment of the present application, the power system migration reinforcement learning scheduling device based on flexible retractable network reconstruction proposed includes a modeling module for determining the state space, action space and target reward function of the target power system scheduling model under the normal power grid topology operation scenario, so as to establish the target power system scheduling model according to the state space, action space and target reward function; a training module for training the target power system scheduling agent corresponding to the target power system scheduling model based on a preset deep deterministic gradient algorithm; a migration module for migrating the target power system scheduling agent based on the preset flexible retractable network reconstruction migration learning strategy in the power grid topology change scenario, so as to generate a migration reinforcement learning adaptive scheduling strategy. The present application designs a migration learning method based on flexible retractable network reconstruction, migrates network parameters and network structure, and uses the historical experience and knowledge learned from the source scenario to improve the learning efficiency of the agent, so that it can adapt to the scheduling tasks of the target scenario more quickly, thereby realizing the transfer of the source scenario scheduling knowledge to the operation requirements of the target scheduling scenario with topology changes.

[0158] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0159] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0160] When the processor 502 executes the program, the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction provided in the above embodiment is implemented.

[0161] Furthermore, the electronic device further comprises:

[0162] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0163] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0164] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as a disk memory.

[0165] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0167] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0168] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction.

[0169] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction.

[0170] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in the embodiments or examples of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0171] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include the feature. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0172] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0173] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0174] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0175] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0177] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction, characterized in that: The following steps are involved: Determine the state space, action space and target reward function of the target power system dispatching model under the normal topology operation scenario of the power grid, so as to establish the target power system dispatching model according to the state space, the action space and the target reward function; Based on a preset deep deterministic gradient algorithm, training a target power system dispatching agent corresponding to the target power system dispatching model; In the scenario of power grid topology changes, based on the preset transfer learning strategy of flexible scalable network reconstruction, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy.

2. The method according to claim 1, characterized in that The state space, action space and target reward function of the target power system dispatching model under the normal power grid topology operation scenario are determined, including: Obtain the normalized node loads at all moments in the forward-looking window of period t in the normal topology operation scenario of the power grid and the corresponding active output output in the previous forward-looking window, and construct the state space according to the node loads and the corresponding active output, where t is a positive integer; Inputting the state space into the target power system dispatching intelligent agent to obtain the active output of the generator set at all times within the forward-looking window of the period t corresponding to the state space, so as to construct the action space according to the active output of the generator set; Determine the active output and start-stop cost of generator set i at the τth moment in the forward-looking window of the t period, the number of moments contained in the dispatch window corresponding to the target power system dispatch model, and the number of generator sets, so as to determine the main line reward function according to the active output of the generator set, the number of moments, the number of generator sets and the start-stop cost, wherein τ and i are both positive integers; Determine the unit output over-limit penalty, unit climbing over-limit penalty, and flow load rate over-limit penalty corresponding to the main line reward function, and construct the target reward function based on the main line reward function, the unit output over-limit penalty, the unit climbing over-limit penalty, and the flow load rate over-limit penalty.

3. The method according to claim 1, characterized in that The target power system dispatching agent corresponding to the target power system dispatching model is trained based on a preset deep deterministic gradient algorithm under the normal power grid topology operation scenario, including: Based on the preset target network and experience replay mechanism, Actor target network, Critic target network and fully connected neural network, the deep deterministic gradient algorithm is constructed; Based on the Actor main network in the Actor target network, the dispatching strategy corresponding to the target power system dispatching model is used to obtain the action corresponding to the current environmental state; Evaluate the action through the Critic main network in the Critic target network to obtain a corresponding first evaluation result, calculate a corresponding first loss using the first evaluation result, and optimize the Actor target parameters in the Actor main network to obtain a minimized loss corresponding to the first loss; Based on a preset exponential moving average strategy, the optimized Actor target parameters in the Actor target network are updated to obtain an updated Actor target network, and an action corresponding to the next environmental state is obtained through the updated Actor target network according to the strategy corresponding to the next environmental state; Inputting the action corresponding to the next environmental state and the next environmental state into the Critic target network to output a second evaluation result, calculating a second loss according to the first evaluation result and the second evaluation result, and optimizing the Critic target parameters in the Critic main network to obtain a minimized loss corresponding to the second loss; Based on the exponential moving average strategy, the optimized critic target parameters in the critic target network are updated to obtain an updated critic target network, and the updated critic target network and the updated actor target network are used to train the target power system dispatching agent corresponding to the target power system dispatching model.

4. The method according to claim 2, characterized in that: In the scenario of power grid topology change, based on the preset transfer learning strategy of flexible scalable network reconstruction, the target power system dispatching agent is migrated to generate a transfer reinforcement learning adaptive dispatching strategy, including: Determine whether the input state and / or output action space corresponding to the power grid topology change scenario has changed; If the input state and / or the output action space corresponding to the power grid topology change scenario is expanded, an input channel of the target power system dispatching model corresponding to the target power system dispatching agent is added to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding first migration model; If the input state and / or the output action space corresponding to the power grid topology change scenario is reduced, the neurons of the target power system dispatching model are shielded to reconstruct the neural network corresponding to the target power system dispatching agent and generate a corresponding second migration model; Based on the first migration model or the second migration model, migration training is performed on the target power system dispatching agent to obtain a target forward-looking dispatching strategy.

5. The method according to claim 2, characterized in that: The expression of the target reward function is: in, represents the mainline reward corresponding to the t-th look-ahead window, represents the penalty term of the ith scheduling action corresponding to the tth scheduling window, r t is the target reward function for the t-th scheduling window.

6. The method according to claim 4, characterized in that The expression of the target forward scheduling strategy under the target scenario is: Among them, x s Represents the source scene input state vector; W s represents the agent network weight of the source scene; W n represents the network weight of the newly added input channel of the target scene; b represents the neural network parameters of the source network.

7. A power system migration reinforcement learning scheduling device based on flexible scalable network reconstruction, characterized in that: include: A modeling module, used to determine the state space, action space and target reward function of the target power system dispatching model under the normal topology operation scenario of the power grid, so as to establish the target power system dispatching model according to the state space, the action space and the target reward function; A training module, used for training a target power system dispatching agent corresponding to the target power system dispatching model based on a preset deep deterministic gradient algorithm; The migration module is used to migrate the target power system dispatching agent based on a preset flexible scalable network reconstruction migration learning strategy in the scenario of power grid topology change, so as to generate a migration reinforcement learning adaptive dispatching strategy.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the power system migration reinforcement learning scheduling method based on flexible scalable network reconstruction as described in any one of claims 1 to 6.

Images