Distribution line intelligent planning method considering operation flexibility and based on deep reinforcement learning
By introducing deep reinforcement learning and a cost model for operational flexibility shortage in power distribution line planning, the problem of lack of quantitative analysis of operational flexibility in existing technologies is solved, thereby improving the economic rationality and technical robustness of line planning schemes.
Patent Information
- Application Number
- CN202511698881.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-17
AI Technical Summary
Existing power distribution line planning methods lack direct quantification and optimization of system operation flexibility, making it difficult to effectively absorb fluctuating power sources when faced with a high proportion of distributed power sources, which can easily lead to operational problems such as curtailment of solar power and voltage exceeding limits.
By establishing a comprehensive cost mathematical model that includes the cost of operational flexibility shortage, and combining it with deep reinforcement learning, an agent is trained to optimize route planning schemes, thereby achieving the scientific quantification and optimization of operational flexibility.
It enables the quantification of operational flexibility targets and long-term value assessment of line planning schemes, ensuring the economic rationality and technical robustness of planning schemes under dynamic operation, and improving the intelligence level and robustness of distribution network planning schemes.
Smart Images
Figure CN121543957A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of power distribution network planning, and more particularly to a power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility. BACKGROUND
[0002] The intermittency and volatility of distributed energy output bring adjustment pressure and operation uncertainty to the power grid, and the power grid needs to consider whether it has high operation flexibility when planning, that is, the ability of the system to quickly respond to source and load fluctuations and maintain stable operation.
[0003] However, the existing power distribution line planning method mostly focuses on the static configuration of power transmission capacity, mainly solving the problem of power transmission path; at the same time, the main goal is to minimize investment cost and network loss, lacking direct quantification and optimization mechanism of system operation flexibility. Further, the network structure planned by the above method is qualified in static indicators, but when facing actual dynamic operation scenarios with high proportion of distributed power access in the future, it often performs rigidly and cannot effectively accommodate fluctuating power, which is prone to operation problems such as light rejection and voltage out-of-limit.
[0004] Therefore, it is urgent to propose a power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility. SUMMARY
[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility, which establishes a comprehensive cost mathematical model containing operation flexibility shortage cost and combines deep reinforcement learning to solve the problem that the prior art cannot scientifically quantify and optimize operation flexibility and has low solving efficiency in complex multi-stage decision-making.
[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme: A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility, comprising the following steps: initializing a reinforcement learning environment containing operation state information and constructing a deep reinforcement learning agent; training the deep reinforcement learning agent until a preset condition is met, the training process comprising: obtaining operation state information of a next state based on the deep reinforcement learning agent and calculating a cost optimization reward using a comprehensive cost mathematical model, the comprehensive cost mathematical model containing flexibility shortage cost; updating policy network parameters of the deep reinforcement learning agent based on the cost optimization reward; inputting an initial state of a power grid to be planned into the trained deep reinforcement learning agent, and executing sequential decision-making to generate a line planning scheme.
[0007] In a preferred embodiment, the reinforcement learning environment comprising operating state information comprises: initializing a reinforcement learning environment, the reinforcement learning environment comprising an action space and a state space; the state space comprising topology information, scenario information and operating state information, the operating state information being generated by optimal power flow simulation on the topology information and the scenario information.
[0008] In a preferred embodiment, the operating state information comprises a generator output vector, a line load rate vector, a flexibility margin vector and a bus voltage amplitude vector.
[0009] In a preferred embodiment, the constructing a deep reinforcement learning agent comprises: constructing a policy network of the deep reinforcement learning agent and adopting a multi-layer perception as a function approximator; initializing an experience replay pool, the experience replay pool being used to store experience tuples generated in a deep reinforcement learning agent training process.
[0010] In a preferred embodiment, the deep reinforcement learning agent adopts a double deep Q network algorithm.
[0011] In a preferred embodiment, the calculating operating state information of a next state based on the deep reinforcement learning agent comprises: the deep reinforcement learning agent selecting a line construction action based on the action space and a current state; the reinforcement learning environment executing the line construction action and updating topology information, and generating operating state information of the next state based on updated topology information and scenario information by optimal power flow simulation.
[0012] In a preferred embodiment, the comprehensive cost mathematical model comprises: the comprehensive cost comprising investment cost, operating cost, network loss cost and flexibility shortage cost; a specific formula of the flexibility shortage cost being as follows: , wherein, is the flexibility shortage cost, , are upward and downward flexibility shortage unit penalty prices respectively, , are upward and downward flexibility regulation demands respectively, , are actual upward and downward flexibility regulation capacities respectively.
[0013] In a preferred embodiment, the calculating a cost optimization reward based on the comprehensive cost mathematical model and the operating state information of the next state comprises: calculating a comprehensive cost of the next state based on the comprehensive cost mathematical model and the operating state information of the next state; obtaining the cost optimization reward according to the comprehensive cost of the current state and the comprehensive cost of the next state.
[0014] In a preferred embodiment, the method for updating the policy network parameters of the deep reinforcement learning agent based on the cost optimization reward comprises: taking the current state, the line construction action, the cost optimization reward and the next state as an experience tuple; calculating the target value of the experience tuple based on a preset reward discount factor, the cost optimization reward and the next state, and generating a predicted value of the line construction action by the reinforcement learning agent; and updating the policy network parameters by a back propagation algorithm based on the temporal error between the target value and the predicted value.
[0015] In a preferred embodiment, the method for generating the line planning scheme by the sequential decision comprises: the deep reinforcement learning agent selects a line construction action based on the initial state of the power grid to be planned, and obtains a next state based on the line construction action and cycles until a preset planning target is met; and integrating the line construction actions performed to meet the preset planning target into the line planning scheme.
[0016] The technical effects and advantages of the power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility are as follows: 1. The method realizes the quantitative evaluation of the operation flexibility target of the line planning scheme and the long-term value evaluation by establishing a comprehensive cost mathematical model containing flexibility shortage cost and using the sequential decision characteristics of deep reinforcement learning, thereby guaranteeing the economic rationality and technical robustness of the planning scheme under dynamic operation.
[0017] 2. The method realizes the efficient solution and intelligent decision of the candidate line by initializing the reinforcement learning environment containing operation state information and combining the trained deep reinforcement learning agent, so that the agent can quickly and accurately execute deterministic decisions, thereby guaranteeing the generation of the line planning scheme considering economic and flexibility requirements within a reasonable time. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for the embodiment of the present application; Figure 2 A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for the embodiment of the present application; Figure 3 A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for the embodiment of the present application; Figure 4 A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for the embodiment of the present application; Figure 5A topological schematic diagram of a second stage of sequential planning of the power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for an embodiment of the present application. Figure 6 A topological schematic diagram of a final line planning scheme of the power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided for an embodiment of the present application. DETAILED DESCRIPTION
[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0020] Embodiment 1, Figure 1 A power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility is provided in the present application, including the following steps: S1, initializing a reinforcement learning environment containing operation state information and constructing a deep reinforcement learning agent; S2, training the deep reinforcement learning agent until a preset condition is met, and the training process includes: obtaining operation state information of a next state based on the deep reinforcement learning agent and calculating a cost optimization reward by using a comprehensive cost mathematical model, the comprehensive cost mathematical model containing a flexibility shortage cost; updating the policy network parameters of the deep reinforcement learning agent based on the cost optimization reward; S3, inputting an initial state of a power grid to be planned into the trained deep reinforcement learning agent, and executing sequential decision to generate a line planning scheme.
[0021] The present application realizes operation flexibility target quantization and long-term value evaluation of the line planning scheme by establishing a comprehensive cost mathematical model containing a flexibility shortage cost and fully utilizing the sequential decision characteristics of deep reinforcement learning, thereby guaranteeing the economic rationality and technical robustness of the planning scheme under dynamic operation. Meanwhile, the present application initializes a reinforcement learning environment containing operation state information and combines the trained deep reinforcement learning agent, so that the agent can quickly and accurately execute deterministic decisions, realizes efficient solution and intelligent decision of the candidate line, and guarantees the generation of a high-quality line planning scheme considering economic and flexibility requirements within a reasonable time. In summary, the present application solves the problems of lack of operation flexibility quantization and insufficient multi-stage decision-making ability in the prior art, and improves the intelligent level of power distribution network planning and the robustness of the scheme.
[0022] In this embodiment, the reinforcement learning environment containing operational state information in S1 includes: initializing the reinforcement learning environment, which includes an action space and a state space; the state space includes topology information, scene information, and operational state information, which is generated by performing optimal power flow simulation on the topology information and scene information.
[0023] The action space of the reinforcement learning environment is defined as a discrete and finite set of integers, representing the indices of all candidate lines. The action space can be represented as: , in, A collection of action spaces. This represents the total number of candidate routes.
[0024] The state space of the reinforcement learning environment is defined as a standardized multidimensional vector, which includes topological information, scene information, and runtime state information. The topology information is used to describe the current physical connection structure of the power grid, with the length being the total number of candidate lines. binary vector express: , in, It can be used as an indicator variable for the construction status of the line. Indicates the line Completed Indicates the line Not built; The scenario information is used to describe the fixed boundary conditions of the planning problem and remains unchanged during training; the scenario information is composed of vectors with fixed parameters, specifically including: a photovoltaic capacity vector containing the photovoltaic installed capacity of each node. Energy storage capacity vector containing the capacity and power of each node. Load vector containing the load levels of each node This includes parameters for conventional generator sets, including minimum and maximum permissible values for cost and output of each conventional generator set; The operational status information is generated by performing optimal power flow simulation calculations on topology and scenario information. The operational status information is formed by concatenating vectors of key operational indicators and then normalizing them.
[0025] In this embodiment, the operating status information includes generator output vector, line load rate vector, flexibility margin vector, and bus voltage amplitude vector; the generator output vector The actual output of each conventional generator unit; the line load rate vector The length is the sum of the lines in the system. , the line includes initial line and built line, the value of the element in the vector is the actual power flow of the line , the ratio of the maximum transmission capacity of the line ; the length of the flexibility margin vector is 2, including upward flexibility adjustment capability and downward flexibility adjustment capability ; the length of the bus voltage amplitude vector is the total number of system buses , the value of the element in the vector is the per unit value of the bus voltage, and the calculation formula of the per unit value is: , wherein, is the per unit value of the bus voltage, is the actual value of the bus voltage, is the reference value of the bus voltage.
[0026] In the embodiment, the constructing the deep reinforcement learning agent in S1 includes: constructing a policy network of the deep reinforcement learning agent and adopting a multilayer perceptron as a function approximator; and initializing an experience replay pool, which is used to store experience tuples generated in a deep reinforcement learning agent training process.
[0027] The number of neurons in the input layer of the policy network is the same as the total length of the vector of the state space, and the number of neurons in the output layer is the same as the total number of candidate lines in the action space; the multilayer perceptron has multiple hidden layers.
[0028] Key hyperparameters in the deep reinforcement learning agent training process are set, including a learning rate, a reward discount factor, an exploration rate and an exploration rate decay strategy; the learning rate is used to control the step size of policy network adjustment, and the preset range is to ; the reward discount factor is preset based on the long-term planning parameters of the power grid, and is used to measure the importance of the deep reinforcement learning agent to future rewards, and is usually set to a value between 0.9 and 0.999; the exploration rate decay strategy adopts an exponential decay function, and the exploration rate is decayed to a minimum threshold value based on an initial value and a preset decay factor, and the decay factor should be a value close to 1, for example, between 0.999 and 0.9999.
[0029] In the embodiment, the deep reinforcement learning agent in S1 adopts a double deep Q network algorithm.
[0030] The step is achieved by initializing a reinforcement learning environment containing topology information, scene information and running state information and constructing a deep reinforcement learning agent, which realizes the structural mapping of the complex power distribution network planning problem to the Markov decision process that can be solved by reinforcement learning, provides a data foundation and algorithm carrier for subsequent intelligent learning and decision making, and guarantees the implementability and technical compatibility of the scheme.
[0031] Referring to Figure 2 , the training process in S2 is a closed-loop system composed of a reinforcement learning agent and a power grid planning environment, the power grid planning environment is a reinforcement learning environment, and the training process follows the cycle mechanism of the Markov decision process: the agent learns and generates the optimal line construction strategy through the policy network, i.e., the "deep neural network" in the figure, i.e., the "action strategy" in the figure; the line construction action, i.e., the "action vector" in the figure, is selected based on the current state and sent to the environment; the environment receives the action, executes and updates the topology information of the power grid, i.e., the "grid structure" in the figure; then, the environment calculates the economic optimization reward based on the optimal power flow simulation and the comprehensive cost mathematical model, i.e., the "reward function" in the figure, and feeds back the reward value and the next state to the agent; the agent includes an experience replay pool for storing experience tuples generated during the training process, i.e., "historical data" in the figure, and finally updates the policy network parameters using the deep neural network based on these data.
[0032] The preset condition includes that the training process of the agent has reached a preset maximum number of training rounds or has converged; the maximum number of training rounds is based on experience; the convergence is specifically that when the comprehensive cost decrease amplitude of a plurality of consecutive training rounds is lower than a preset threshold, it is considered that the strategy of the agent has stably converged, and the preset threshold is based on experience.
[0033] In this embodiment, the running state information of the next state calculated based on the deep reinforcement learning agent in S2 includes: the deep reinforcement learning agent selects a line construction action based on the action space and the current state; the reinforcement learning environment executes the line construction action and updates the topology information, and generates the running state information of the next state based on the updated topology information and the scene information.
[0034] At the beginning of each training round, the environment is reset to the initial state , which is constructed according to the benchmark data of the power distribution network to be planned, specifically a multi-dimensional vector containing initial topology information, scene information and initial running state information obtained by optimal power flow simulation.
[0035] The deep reinforcement learning agent inputs the current state into its policy network, and outputs a vector equal in length to the action space through the forward propagation of the multi-layer perceptron structure in the policy network, and the elements of the vector i.e. in the current state the candidate action is executed an estimate of the expected long-term cumulative reward that can be obtained.
[0036] exploration rate of the deep reinforcement learning agent randomly selects an action from the action space for exploration, so that the candidate action with the maximum current Q value is selected for exploitation with a probability, and the exploration rate gradually decreases based on an exploration rate decay strategy as the training round progresses; finally, the deep reinforcement learning agent selects the candidate line construction action with the maximum Q value i.e. decides to build the candidate line corresponding to the index.
[0037] After the environment receives the line construction action , the candidate line corresponding to the action is parsed, and the matrix elements corresponding to the candidate line are modified based on the node-branch association matrix of the current power grid, the modification including: adding or activating the connection relationship corresponding to the candidate line in the matrix and updating the binary element corresponding to the candidate line in the topology information vector from 0 to 1, i.e. from not built to built; the modified association matrix is taken as the internal power grid topology of the next state, and the operating state information of the next state is generated through optimal power flow simulation, which takes the updated topology information and scenario information as input and aims to minimize the operating cost of the system; In this embodiment, the comprehensive cost mathematical model in S2 includes: the comprehensive cost contains investment cost, operating cost, network loss cost and flexibility shortage cost; The specific formula of the investment cost is as follows: , wherein, is the annual equivalent line construction investment cost, is the set of all candidate lines, is the unit length cost of line , and is the physical length of line ; is the investment cost annual coefficient, which is calculated based on the discount rate and the expected service life of the line.
[0038] The specific formula of the operating cost is as follows: , wherein, is the operating cost, is the unit operating cost, is the total operating time, is the Actual output of a conventional generator set connected to a busbar.
[0039] The specific formula of the network loss cost is as follows: , Wherein, is the network loss cost, is the unit loss energy price, is the operation time period, is the set of all nodes directly connected to the busbar , , are the actual current value and resistance value of the line connecting the busbar and the busbar in the time period, respectively; is the total number of nodes, which is equal to the total number of system busbars .
[0040] The specific formula of the flexibility shortage cost is as follows: , Wherein, is the flexibility shortage cost, , are the upward and downward flexibility shortage unit penalty prices, respectively, , are the upward and downward flexibility adjustment demands, respectively.
[0041] In the embodiment, the calculation of the cost optimization reward in S2 by using the comprehensive cost mathematical model and the operation state information of the next state includes: calculating the comprehensive cost of the next state based on the comprehensive cost mathematical model and the operation state information of the next state; and obtaining the cost optimization reward according to the comprehensive cost of the current state and the comprehensive cost of the next state.
[0042] The operation cost, the network loss cost and the flexibility shortage cost are calculated by substituting the operation state information of the next state into the dynamic cost term of the comprehensive cost mathematical model, and the investment cost is calculated based on the updated topology information, and all the costs are added to obtain the comprehensive cost.
[0043] The cost optimization reward is obtained by calculating the difference between the comprehensive costs of the current state and the next state; if the reward value is positive, it indicates that the line construction action reduces the system comprehensive cost, and the strategy network will be positively reinforced; if the reward value is negative, the strategy network is punished.
[0044] In the embodiment, the step of updating the policy network parameter of the deep reinforcement learning agent based on the cost optimization reward in S2 comprises: taking the current state, the line construction action, the cost optimization reward and the next state as an experience tuple; calculating a target value of the experience tuple based on a preset reward discount factor, the cost optimization reward and the next state and generating a predicted value of the line construction action by the deep reinforcement learning agent; and updating the policy network parameter by a back propagation algorithm based on a time sequence error between the target value and the predicted value.
[0045] The target value of the experience tuple is calculated by using the Bellman optimal equation, and the calculation formula of the target value is as follows:
[0046] wherein, the target value is V, the cost optimization reward is R, and the reward discount factor is γ.
[0047] The current state S is input into the policy network, and a predicted value is obtained by a forward propagation operation of the policy network parameter; the predicted value is a current value estimation of the policy network on the line construction action; and a difference between the target value and the predicted value is calculated as a time sequence error, which is used to measure the prediction accuracy of the policy network.
[0048] A loss function is generated by using the method of mean square error based on the time sequence error; the deep reinforcement learning agent calculates a partial derivative of the loss function to the policy network parameter by a back propagation algorithm, adjusts the policy network parameter by a gradient descent method by using the partial derivative and a learning rate, so as to minimize the time sequence error and make the predicted value closer to the target value.
[0049] It should be noted that the traditional planning method only determines based on static investment and network loss indicators when evaluating the advantages and disadvantages of the grid structure; the present application dynamically evaluates the actual operation adaptability of the planning scheme in a high volatility scenario by quantifying the shortage cost of operation flexibility, realizes accurate quantitative diagnosis of the economic benefit and technical robustness of the planning scheme, and ensures the forward-looking and effectiveness of the final scheme.
[0050] This step improves the decision-making ability of the agent to deal with the complexity and nonlinearity of the distribution network by the sequential interaction of the reinforcement learning agent and the environment and the updating of the policy network parameter based on the cost optimization reward, and guarantees the intelligence and global optimization of the solution.
[0051] In this embodiment, the sequential decision generation line planning scheme in S3 comprises: the deep reinforcement learning agent selects a line construction action based on the initial state of the power grid to be planned, obtains a next state based on the line construction action, and cycles until a preset planning target is met; and the line construction actions required to meet the preset planning target are integrated into a line planning scheme.
[0052] The exploration rate of the trained deep reinforcement learning agent is set to zero to make it in a deterministic decision mode; and the initial state of the power grid to be planned is input into the trained deep reinforcement learning agent. The initial state is encoded in a format required by the policy network, and includes initial topological information, scenario information, and initial operating state information.
[0053] The deep reinforcement learning agent uses its fixed policy network to evaluate all feasible line construction actions, and selects a line construction action with the minimum expected long-term comprehensive cost under the current state; and a next state of the power grid to be planned is obtained based on the line construction action.
[0054] The deep reinforcement learning agent takes the next state as a new input and repeats to continuously execute sequential decisions until a preset planning target is met; the preset planning target is a constraint condition for guiding when the decision sequence stops, and includes a technical feasibility target, a reliability target, and an economic constraint target.
[0055] All line construction actions performed by the deep reinforcement learning agent before reaching the preset planning target are integrated in sequence to generate a final line planning scheme.
[0056] This step generates a line planning scheme that takes into account economy and long-term robustness by loading the trained optimal policy network and making it enter a deterministic decision mode to execute sequential decisions, thereby improving planning efficiency and ensuring global optimization of the finally generated scheme.
[0057] In embodiment 2, to verify the effectiveness of the method, the trained deep reinforcement learning agent is applied to the extended planning of an IEEE33 node system, and a final line planning scheme is generated.
[0058] (1) Input high-proportion distributed energy scenario data.
[0059] In this embodiment, an IEEE33 node distribution network system that is modified and connected with high-proportion photovoltaic and energy storage is used as a planning object, and the voltage of the distribution network system is 12.66 kV.
[0060] The following devices are deployed on the benchmark system: new energy nodes: Bus18: 3MWp, Bus33: 2.5MWp, Bus25: 1.5MWp; energy storage (ESS) nodes: Bus18: 1MW / 2MWh, Bus6: 1.5MW / 3MWh, Bus30: 1MW / 2MWh.
[0061] The total action space of the candidate route set DRL agent is composed of the following 9 candidate routes: Table 1
[0062] (2) Initialize the deep reinforcement learning agent and the planning environment.
[0063] The intelligent planning environment for the scene is established, and an optimization function with the goal of minimizing the comprehensive cost is defined; the comprehensive cost includes investment cost, operation cost, network loss cost, and flexibility shortage cost.
[0064] (3) Perform multi-round simulation training to learn the optimal expansion strategy.
[0065] For this scene, start the interactive training of the deep reinforcement learning agent and the environment; in thousands of simulation planning rounds, the agent learns the long-term impact of different decisions on the comprehensive cost by selecting different route construction sequences, and gradually converges to the optimal strategy.
[0066] (4) Execute sequential decision generation of line planning scheme.
[0067] Determine whether the training process of the deep reinforcement learning agent has converged or has reached the preset maximum number of training rounds; load the optimal strategy network with fixed weights after training and set the agent's exploration rate to zero to enter the deterministic decision mode; input the initial state of the actual power grid to be planned into the trained deep reinforcement learning agent, and make sequential decisions by the deep reinforcement learning agent until the planning target is met, thereby generating a line planning scheme; The execution process of the sequential decision specifically includes: Figure 3 , Figure 3 The benchmark case for the power grid to be planned, i.e., the initial topology structure at t=0, is the starting point for the decision of the deep reinforcement learning agent Referring to Figure 4 , the deep reinforcement learning agent selects the optimal line construction action based on , and forms the topology structure of the next state after execution; then, referring to Figure 5 , the agent selects the action again based on the new state , forming the next topology structure; finally, referring to Figure 6After meeting the preset planning target, all construction actions are integrated in sequence to form a final line expansion scheme, Figure 6 The final line expansion scheme generated for the deep reinforcement learning agent.
[0068] The above formulas are dimensionless numerical calculations, and the formulas are obtained by software simulation of a large amount of data to obtain a formula of the latest real situation. The preset parameters in the formula are set by a person skilled in the art according to the actual situation.
[0069] The above embodiments can be realized wholly or partially by software, hardware, firmware, or any other combination. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product.
[0070] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0071] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module.
[0072] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
[0073] Finally, the above is only the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A deep reinforcement learning based power distribution line intelligent planning method considering operation flexibility, characterized in that, The method comprises the following steps: initializing a reinforcement learning environment containing operation state information and constructing a deep reinforcement learning agent; training the deep reinforcement learning agent until a preset condition is met, the training process comprising: obtaining operation state information of a next state based on the deep reinforcement learning agent and calculating a cost optimization reward by using a comprehensive cost mathematical model, the comprehensive cost mathematical model containing flexibility shortage cost; updating policy network parameters of the deep reinforcement learning agent based on the cost optimization reward; inputting an initial state of an electric power grid to be planned into the trained deep reinforcement learning agent to execute sequential decision-making and generate a line planning scheme.
2. The method of claim 1, wherein the method further comprises: The reinforcement learning environment containing operation state information comprises: initializing a reinforcement learning environment, the reinforcement learning environment containing an action space and a state space; the state space comprising topological information, scenario information and operation state information, the operation state information being generated by optimal power flow simulation on the topological information and the scenario information. 3.The power distribution line intelligent planning method based on deep reinforcement learning considering operation flexibility according to claim 2, wherein, The operation state information comprises a generator output vector, a line load rate vector, a flexibility margin vector and a bus voltage amplitude vector.
4. The method of claim 3, wherein the method further comprises: The construction of the deep reinforcement learning agent comprises: constructing a policy network of the deep reinforcement learning agent and adopting a multi-layer perception machine as a function approximator; initializing an experience replay pool, the experience replay pool being used to store experience tuples generated in the training process of the deep reinforcement learning agent.
5. The method of claim 4, wherein the method further comprises: The deep reinforcement learning agent adopts a double deep Q network algorithm.
6. The method of claim 5, wherein the method further comprises: The calculation of operation state information of a next state based on the deep reinforcement learning agent comprises: the deep reinforcement learning agent selecting a line construction action based on the action space and a current state; the reinforcement learning environment executing the line construction action and updating topological information, and generating operation state information of the next state based on the updated topological information and scenario information.
7. The method of claim 6, wherein the method further comprises: The comprehensive cost mathematical model comprises: the comprehensive cost containing investment cost, operation cost, network loss cost and flexibility shortage cost; a specific formula of the flexibility shortage cost being as follows: wherein, is the flexibility shortage cost, , are the upward and downward flexibility shortage unit penalty prices, respectively, , are the upward and downward flexibility adjustment demands, respectively, , are the actual upward and downward flexibility adjustment capacities, respectively.
8. The method of claim 7, wherein the method further comprises: The calculation of a cost optimization reward by using the comprehensive cost mathematical model and operation state information of the next state comprises: calculating a comprehensive cost of the next state based on the comprehensive cost mathematical model and the operation state information of the next state; obtaining the cost optimization reward according to the comprehensive cost of the current state and the comprehensive cost of the next state.
9. The method of claim 8, wherein the method further comprises: The updating of policy network parameters of the deep reinforcement learning agent based on the cost optimization reward comprises: taking the current state, the line construction action, the cost optimization reward and the next state as an experience tuple; calculating a target value of the experience tuple based on a preset reward discount factor, the cost optimization reward and the next state and generating a predicted value of the line construction action by using the reinforcement learning agent; updating the policy network parameters by a back propagation algorithm based on a time sequence error between the target value and the predicted value.
10. The method of claim 9, wherein the method further comprises: The execution of sequential decision-making to generate a line planning scheme comprises: the deep reinforcement learning agent selecting a line construction action based on an initial state of an electric power grid to be planned and obtaining a next state based on the line construction action and repeating the selection and the obtaining until a preset planning target is met. The line construction actions required to meet the preset planning objectives are integrated into a line planning scheme.