Load transfer-to-supply solution method and system for power distribution network

Abstract

The invention discloses a power distribution network load transfer solution method and system, and the method comprises the steps: obtaining the real-time state information of a target power distribution network, and building a reinforcement learning model of first load transfer in combination with a Markov decision process; updating the intelligent agent based on the reinforcement learning model of the first load transfer, and establishing an updated reinforcement learning model of the second load transfer; introducing an action mechanism of pre-simulation optimization, and training the reinforcement learning model of the second load transfer to obtain a reinforcement learning model of a third load transfer; based on the reinforcement learning model of the third load transfer, obtaining a load transfer solution result of the target power distribution network; through real-time monitoring and analysis of the operation state of the power distribution network, the stability and reliability of the power distribution network can be effectively improved. By establishing the reinforcement learning model, the intelligent agent can autonomously learn and optimize a load transfer strategy, so that measures can be taken quickly when a fault or load fluctuation occurs, and continuity and safety of power supply are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of load transfer solutions for distribution networks, and in particular to a method and system for solving load transfer solutions for distribution networks. Background Art

[0002] With the rapid development of smart grid technology, the automation and intelligence level of distribution networks has been continuously improved, which has put forward higher requirements for the operation and control of distribution networks. As an important part of the power system, the stability and reliability of the distribution network are directly related to the power quality of users and the overall performance of the power system. However, the distribution network will inevitably encounter various faults and abnormal conditions during operation, such as equipment aging, natural disasters, human operational errors, etc. These factors may cause power outages in some load areas and affect the normal power consumption of users.

[0003] To meet these challenges, traditional distribution network load transfer methods mainly rely on fixed preset strategies and heuristic rules. These methods often have difficulty making optimal decisions when facing complex and changing grid operating environments. In addition, traditional load transfer methods often take a long time to perform simulation iterations when restoring power supply, resulting in a slow power restoration speed, which cannot meet the modern power system's demand for rapid response.

[0004] In order to solve the above problems, deep reinforcement learning technology has been introduced into the field of load transfer in distribution networks in recent years. Deep reinforcement learning combines the powerful feature extraction ability of deep learning and the decision optimization ability of reinforcement learning, and can learn the optimal control strategy through interaction with the environment. Compared with traditional methods, the load transfer method of distribution networks based on deep reinforcement learning can automatically adapt to changes in power grid topology and make decisions quickly without human intervention, greatly improving the operating efficiency and reliability of distribution networks.

[0005] However, the complexity of the distribution network poses a challenge to deep reinforcement learning. The distribution network has a large number of switch combinations and a variety of topological structures, which makes the state space and action space very large, making it difficult for traditional deep reinforcement learning algorithms to learn effectively. In addition, the operation of the distribution network is also affected by a variety of uncertain factors, such as load fluctuations and the intermittent nature of renewable energy, which bring additional difficulties to the design of control strategies for load transfer in the distribution network. Summary of the invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for solving load transfer in a distribution network, comprising:

[0009] Obtain the real-time status information of the target distribution network, and establish a reinforcement learning model for the first load transfer by combining the Markov decision process;

[0010] updating the agent based on the reinforcement learning model of the first load transfer, and establishing an updated reinforcement learning model of the second load transfer;

[0011] A pre-simulation optimization action mechanism is introduced to train the reinforcement learning model of the second load transfer to obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, the solution result of the load transfer of the target distribution network is obtained.

[0012] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0013] The reinforcement learning model for establishing the first load transfer includes:

[0014] At each moment t, the environment provides the agent with the current distribution network state space S. The result of the agent's analysis and decision-making is the action space A. The action space A is applied to the environment. The state transition probability of the environment is P. The immediate reward function fed back to the agent by the environment is R.

[0015] For the load transfer problem of the distribution network, the state data generated by the distribution network at each moment is the state space S of the environment, so the state space S is expressed as:

[0016] S=[V,I,SW,F]

[0017] S in =[V in ,I in ,SW i ,F i ]

[0018] Where V is the voltage vector group, which is used to represent the voltage values ​​of all phases at each node in the distribution network. in is the voltage value of the nth phase of the ith node; I is the current vector group, which is used to represent the current value of each phase in all lines in the distribution network, I in is the current value of the nth phase of the ith line; SW is the state value vector of all switches in the distribution network, SW i is the state of the i-th switch, 0 means open, 1 means closed; F is the vector representing the fault state of the distribution network line, F i is the fault status of the line numbered i, 0 indicates normal, and 1 indicates a fault occurs.

[0019] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0020] The step of establishing the reinforcement learning model for the first load transfer further includes:

[0021] Only one switch is switched in one action, and the topological difference between two adjacent states is only one switch.

[0022] The load transfer ends within a limited number of actions, and an action is set to actively end the transfer;

[0023] Note that the action space A of the agent is selected as [0,2N s ] integer range, totaling 2N s +1 action, where N s is the number of operable lines;

[0024] When the action value is 2N s When , the representative does not take any action and exits, and this decision ends;

[0025] When the action value is [0,2N s -1], the calculation of A is as follows:

[0026] X=A mod 2

[0027]

[0028] Where X represents the type of switch operation, X=0 represents an open switch, X=1 represents a closed switch, and Y represents the switch number, with Y ranging from 1 to N. s .

[0029] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0030] The step of establishing the reinforcement learning model for the first load transfer further includes:

[0031] The reward function R of the model consists of the reward part RT and the penalty part RP, that is:

[0032] R=RT+RP

[0033] in:

[0034]

[0035] Among them, N op is the cumulative number of actions in this round of transfer process, P res.k is the load restored by each action, ΔP is the approximate line loss value, and a, b, and c are the offsets added to the values ​​of each factor.

[0036] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0037] The step of establishing the reinforcement learning model for the first load transfer further includes:

[0038] RP=P Volt +P Lim +P Loop +P act +P Reclose +P OverSwitch +P Seq +P 其他

[0039] Among them, P Volt Represents voltage penalty, P Lim Represents the current penalty, P Loop Represents the ring network penalty, P act Invalid action penalty, P Reclose Reclosing penalty, P OverSwitch represents the excessive switching penalty, P Seq represents the operation order penalty, P 其他 Indicates other penalties.

[0040] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0041] The step of establishing the updated reinforcement learning model for the second load transfer includes:

[0042] Since the voltage vector group V and the current vector group I in the state space S are both continuous variables, the improved DQN deep reinforcement learning algorithm is used to construct the intelligent agent of the distribution network load transfer reinforcement learning algorithm;

[0043] The improved DQN deep reinforcement learning algorithm takes the state vector as input and outputs a vector containing the Q value of each action, and the length of the output vector is the number of actions;

[0044] The value function V(s) in the improved DQN deep reinforcement learning algorithm i ) is a scalar value, which is only related to the state and has nothing to do with the action, and represents the evaluation of the environmental state factors;

[0045] The output of the advantage function A(at) is a vector, and its length is also the number of actions. A(at) is centered, that is, the average value is subtracted from the entire vector, eliminating the common environmental influence, reflecting the identifiability of the action, and used to evaluate the additional benefits brought by the action:

[0046]

[0047] Among them, ω, α, and β are the common hidden layer parameters, value function layer parameters, and advantage function layer parameters, respectively. Λ represents the set of all actions. The advantage function A is centralized to highlight the differences in actions and reflect the identifiability of actions.

[0048] As a preferred solution to the method for solving the load transfer problem of the distribution network described in the present invention, wherein:

[0049] The second load transfer reinforcement learning model is trained to obtain a third load transfer reinforcement learning model; and based on the third load transfer reinforcement learning model, the target distribution network load transfer solution is obtained, including:

[0050] Sort all actions in the current state according to their Q values ​​from large to small, using P act Compare with invalid action set to filter out invalid / duplicate actions;

[0051] Keep the original order and take out the first k actions, apply all actions one by one to the current distribution network state for simulation pre-simulation, and exclude all over-limit actions according to the state change results;

[0052] Based on the action ranking generated in the previous step, the actions are sorted in three levels according to "load recovery amount > line loss value > Q value";

[0053] The final output is the optimal action at the head end of the pre-simulation sorting as the real action.

[0054] In a second aspect, the present invention provides a distribution network load transfer solution system, comprising:

[0055] The first model building module is used to obtain the real-time status information of the target distribution network and establish a reinforcement learning model of the first load transfer in combination with the Markov decision process;

[0056] A second model building module, used to update the intelligent agent based on the reinforcement learning model of the first load transfer, and to build an updated reinforcement learning model of the second load transfer;

[0057] The training and solving module is used to introduce a pre-simulation optimization action mechanism, train the reinforcement learning model of the second load transfer, and obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, obtain the target distribution network load transfer solution result.

[0058] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when executing the computer program.

[0059] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described above when executed by a processor.

[0060] Beneficial effects of the present invention: The present invention obtains the real-time state information of the target distribution network, and establishes a reinforcement learning model for the first load transfer in combination with the Markov decision process; updates the intelligent agent according to the state space S, and establishes the updated reinforcement learning model for the second load transfer; introduces a pre-simulation preferred action mechanism, trains the reinforcement learning model for the second load transfer, and obtains the target distribution network load transfer solution result according to the reinforcement learning model for the second load transfer after training. The distribution network load transfer solution method and system of the present invention can effectively improve the stability and reliability of the distribution network by real-time monitoring and analyzing the operating status of the distribution network. By establishing a reinforcement learning model, the intelligent agent can autonomously learn and optimize the load transfer strategy, so that when a fault or load fluctuation occurs, measures can be taken quickly to ensure the continuity and safety of the power supply. In an embodiment of the present invention, the intelligent agent continuously learns and adjusts its strategy through interaction with the environment to maximize the cumulative reward. The design of the reward function R takes into account multiple factors, such as the amount of restored load, line loss value, voltage and current stability, etc., to ensure that the intelligent agent pays attention to the overall performance and safety of the system while pursuing load recovery. In addition, the present invention also introduces a pre-simulation preferred action mechanism, which evaluates and screens possible actions through simulation pre-simulation, thereby avoiding over-limit actions that may occur in actual operations, and further improving the accuracy and efficiency of load transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0062] Figure 1 A method flow chart of a method and system for solving load transfer in a distribution network provided by an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of an IEEE33 node distribution system of a distribution network load transfer solution method and system provided by an embodiment of the present invention;

[0064] Figure 3 A method for solving load transfer in a distribution network and a schematic diagram of the training effect of the number of actions and load recovery amount of the system provided by an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of the line loss rate training effect of a distribution network load transfer solution method and system provided by an embodiment of the present invention;

[0066] Figure 5 A schematic diagram showing a comparison of average training rewards of different algorithms in a method and system for solving a load transfer problem in a distribution network provided by an embodiment of the present invention;

[0067] Figure 6 A schematic diagram showing a comparison of average losses of different algorithms of a method and system for solving a load transfer problem in a distribution network provided by an embodiment of the present invention;

[0068] Figure 7 An internal structural diagram of a computer device of a distribution network load transfer solution method and system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0071] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0072] Example 1

[0073] Reference Figure 1-Figure 7 , which is the first embodiment of the present invention, and provides a method and system for solving load transfer in a distribution network, including:

[0074] Before describing the embodiments of the present application in detail, some related concepts are first explained for the sake of clarity.

[0075] DQN Deep Reinforcement Learning Algorithm: DQN (Deep Q-Network) is a deep reinforcement learning algorithm that combines the Q-learning algorithm with deep learning technology. Traditional Q-learning algorithms use a Q table to save the value function of state-action pairs. This method is suitable for situations where the state and action space are small. However, when the state space is continuous or very large, the use of Q tables becomes infeasible. The DQN algorithm uses a deep neural network to approximate the Q value function, allowing the algorithm to handle continuous state spaces and large-scale state spaces. In addition, DQN introduces an experience replay mechanism and a target network freezing technique to stabilize the training process.

[0076] Markov decision process: Markov decision process is a mathematical framework used to describe the process of an agent taking actions in an environment to achieve a certain goal. MDP consists of four main components: state set S, action set A, state transition probability P, and immediate reward R. In this process, the agent is in a certain state at each point in time. The agent chooses an action to apply to the environment. The environment transfers to the next state according to the state transition probability and gives the agent an immediate reward. The goal of the agent is to maximize the cumulative reward in a series of decision steps.

[0077] Pre-simulation-greedy action strategy: The pre-simulation-greedy action strategy is an improvement on the traditional ε-greedy strategy. In the standard ε-greedy strategy, the agent randomly selects an action with probability ε and selects the action with the highest current Q value with probability 1-ε. The pre-simulation-greedy action strategy adds a pre-simulation step, in which the agent simulates multiple possible actions and selects an action based on the results of the simulation. The purpose of this is to better explore the environment and avoid falling into a local optimal solution. The pre-simulation mechanism allows the agent to evaluate the effects of multiple possible action sequences before making a decision, so as to make a better choice. At the same time, this strategy retains some of the action mechanisms in the greedy strategy so that a certain exploration efficiency and convergence performance of the neural network can be guaranteed in complex or highly variable environments.

[0078] There are some problems in the related technologies, such as the insufficient ability of the distribution network to respond quickly to emergencies, and the problems of local overload or voltage instability that may occur during the load transfer process.

[0079] The present application provides a method for effectively solving the above-mentioned problems. Next, a plurality of embodiments will be combined to explain in detail how to implement the load transfer solution method of the distribution network.

[0080] Figure 1 A method flow chart of a method and system for solving a distribution network load transfer problem is shown, including:

[0081] S1: Obtain the real-time status information of the target distribution network, and establish a reinforcement learning model for the first load transfer in combination with the Markov decision process;

[0082] In the embodiment of the present application, the reinforcement learning model of load transfer includes a state space S, an action space A, a state transition probability P, a reward function R, and an agent, and the real-time state information of the target distribution network includes at least the voltage of all nodes of the target distribution network, the current of the branch, and the state of all switches;

[0083] In the embodiment of the present application, the real-time status information of the target distribution network is obtained, and the reinforcement learning model of the first load transfer is established in combination with the Markov decision process, including:

[0084] At each moment t, the environment provides the agent with the current distribution network state space S. The result of the agent's analysis and decision-making is the action space A. The action space A is applied to the environment. The state transition probability of the environment is P. The immediate reward function fed back to the agent by the environment is R.

[0085] In the embodiment of the present application, for the load transfer problem of the distribution network, the state data generated by the distribution network at each moment is the state space S of the environment, so the state space S is expressed as:

[0086] S=[V,I,SW,F]

[0087] S in =[V in ,I in ,SW i ,F i ]

[0088] Where V is the voltage vector group, which is used to represent the voltage values ​​of all phases at each node in the distribution network. in is the voltage value of the nth phase of the ith node; I is the current vector group, which is used to represent the current value of each phase in all lines in the distribution network, I in is the current value of the nth phase of the ith line; SW is the state value vector of all switches in the distribution network, SW i is the state of the i-th switch, 0 means open, 1 means closed; F is the vector representing the fault state of the distribution network line, F i is the fault status of the line numbered i, 0 indicates normal, and 1 indicates a fault occurs.

[0089] In the embodiment of the present application, obtaining the real-time status information of the target distribution network and establishing the reinforcement learning model of the first load transfer in combination with the Markov decision process also includes:

[0090] Only one switch is switched in one action, and the topological difference between two adjacent states is only one switch.

[0091] The load transfer ends within a limited number of actions, and an action is set to actively end the transfer;

[0092] Note that the action space A of the agent is selected as [0,2N s ] integer range, totaling 2N s +1 action, where N s is the number of operable lines;

[0093] When the action value is 2N s When , the representative does not take any action and exits, and this decision ends;

[0094] When the action value is [0,2N s -1], the calculation of A is as follows:

[0095] X=A mod 2

[0096]

[0097] Where X represents the type of switch operation, X=0 represents an open switch, X=1 represents a closed switch, and Y represents the switch number, with Y ranging from 1 to N. s .

[0098] In the embodiment of the present application, obtaining the real-time status information of the target distribution network and establishing the reinforcement learning model of the first load transfer in combination with the Markov decision process also includes:

[0099] The reward function R of the model consists of the reward part RT and the penalty part RP, that is:

[0100] R=RT+RP

[0101] Among them, the primary goal of load transfer is to restore power supply to all users as much as possible, so the most important thing is the load recovery amount. The cumulative number of loads restored by the action RL is taken as the positive correlation part of the objective function, that is:

[0102]

[0103] Among them, N op is the cumulative number of actions in this round of transfer process, P res.k The amount of load recovered for each movement.

[0104] In the embodiments of the present application, the action of the switch will affect the life of the switch, and some switches may require manual operation by personnel, so the transfer should be completed with as few actions as possible to reduce the cost of operation and maintenance and the possibility of human error, while also saving operation time. If there are too many changes in the switch, the structure of the distribution network will change too much, which will increase the difficulty of restoring the original operation mode after the fault is eliminated. Therefore, this application considers taking the number of actions Nop as the negatively correlated part of the objective function.

[0105] The line loss of the distribution network is also an important cost factor. This application uses the impedance parameters of the energized line to evaluate the line loss of the distribution network and calculates the approximate line loss value ΔP, that is:

[0106]

[0107] Where, l is the total number of energized lines, I i , Ri is the current and resistance of the ith line. The line loss value should be as small as possible, so the line loss part should be negatively correlated with the objective function RT.

[0108] In the embodiment of the present application, the final objective function RT is calculated according to the positive and negative proportional relationship of each influencing factor. At the same time, in order to facilitate the adjustment of the influence of each factor on RT to achieve the best training effect, the bias a, b, and c are added to each factor value respectively, and the calculation formula is as follows, that is:

[0109]

[0110] Among them, N op is the cumulative number of actions in this round of transfer process, P res.k is the load restored by each action, ΔP is the approximate line loss value, and a, b, and c are the offsets added to the values ​​of each factor.

[0111] In the embodiment of the present application, obtaining the real-time status information of the target distribution network and establishing the reinforcement learning model of the first load transfer in combination with the Markov decision process also includes:

[0112] RP=P Volt +P Lim +P Loop +P act +P Reclose +P OverSwitch +P Seq +P 其他

[0113] Among them, P Volt Represents voltage penalty, P Lim Represents the current penalty, P Loop Represents the ring network penalty, P act Invalid action penalty, P RecloseReclosing penalty, P OverSwitch represents the excessive switching penalty, P Seq represents the operation order penalty, P 其他 Indicates other penalties.

[0114] Among them, maintaining normal node voltage is the basic requirement for ensuring the normal operation of the distribution network. Voltage exceeding the limit may cause secondary failure of power-consuming equipment and pose a threat to the safety of users. The voltage should be kept within the allowable range of ±7%. In order to prevent the intelligent agent from exceeding the constraint conditions when acting, a high penalty is imposed for voltages exceeding the range, and no penalty is set for actions within the range.

[0115] At the same time, after the voltage exceeds the limit, the environment state should be transferred to the failure exit state, and the voltage penalty PVolt calculation method is:

[0116]

[0117] Where Ui is the voltage per unit value of each node i, U i.min and U i.max The upper and lower limits of the voltage at node i are usually 0.93 or 1.07.

[0118] When the transmission capacity exceeds the limit value of the line and transformer, it is easy to cause secondary failure of the equipment, seriously reducing the service life of the electrical equipment. This application takes the equipment current limit value as the upper limit of operation. When the current exceeds the limit, the state is transferred to the failure exit state. The current penalty P Lim The calculation method is:

[0119]

[0120] In the formula, I j is the current value of device j, I j.max is the upper limit of the current value of device j.

[0121] The distribution network should be radial when operating normally, but a ring network is allowed to exist as a short-term transition state, but it is not allowed to appear as a long-term operating state. Therefore, a ring network penalty P is set. Loop should be considered, namely:

[0122]

[0123] In the formula, g n is the restored area, G R It represents the set of radial topologies that should be maintained for the distribution network when distributed generation is not included.

[0124] When the agent performs a closing action on a currently closed switch or opens a currently open switch, that is, the action is invalid, or attempts to operate a faulty line, the action is considered invalid and a penalty is given for its invalid action, namely:

[0125]

[0126] Where Ak is the kth action of the agent in this transfer round, and OA is the set of actions that the agent has executed in this transfer round and the faulty lines in this round.

[0127] A reclosing penalty should be imposed when the switch is detected to be frequently switched from closed to open and back to closed within less than the minimum allowed time interval. This usually occurs when a transient fault is detected and reclosing should not be performed immediately to avoid equipment damage or unnecessary interference, i.e.:

[0128]

[0129] Among them, K Reclose is the reclosing penalty coefficient, t reclose Indicates the time interval between the last two switch state changes, t minReclose is the minimum time interval allowed. reclose When the state changes frequently within a certain period of time, it indicates that there is a reclosing phenomenon. At this time, the penalty given increases as the time interval becomes shorter.

[0130] When the number of switch operations in a certain period of time is N switch Exceeded the preset maximum allowed number N threshold , then it is considered that excessive switching has occurred, and excessive switching penalties should be imposed. Excessive switching may cause increased wear of the equipment and affect the life of the equipment, namely:

[0131]

[0132] Among them, K OverSwitch is the over-switching penalty coefficient, N switch Indicates the number of switch operations within a period of time, N threshold is the preset maximum allowed number. When the number of switch operations within a period of time exceeds the preset threshold, the penalty given increases with the number of times.

[0133] In some cases, the order of switch operations is crucial, such as which switch is opened first and then another switch is closed to avoid loops or other undesirable situations. If the agent's operation sequence violates the pre-defined operation order rules, an operation order penalty should be given, namely:

[0134]

[0135] Among them, K Seq is the operation order penalty coefficient, O i represents the i-th item in the actual operation sequence, E i represents the i-th item in the expected correct sequence of operations, N outOfOrder Indicates the number of operation order errors. When the operation order is wrong, the penalty given increases as the number of errors increases.

[0136] In the present embodiment, P 其他 Including power factor penalty, frequency deviation penalty and temperature over-limit penalty:

[0137] Specifically, if the power factor in the distribution network is lower than a certain threshold, it indicates that the proportion of reactive power is high, which may lead to reduced equipment efficiency and additional losses. When the power factor is lower than the set minimum value, a power factor penalty can be introduced. <PF min , then P F =K PF ·(PF min -PF), where K PF is the power factor penalty coefficient.

[0138] Frequency deviation penalty (PFreq): The frequency of the power system should be kept stable, usually 50Hz or 60Hz. If the system frequency deviates from the standard value due to load transfer, a frequency deviation penalty should be introduced. threshold , then P Freq =K Freq Δf, where K Freq is the frequency deviation penalty coefficient.

[0139] Temperature overrun penalty (PTemp): When power distribution equipment such as transformers are operated under overload conditions, the temperature may rise. If the temperature exceeds the safe operating range, a temperature overrun penalty should be introduced. max , then P Temp =K Temp ·(TT max ), where K Temp is the temperature over-limit penalty coefficient.

[0140] It should be noted that in the operation of smart distribution networks, the introduction of reward and penalty mechanisms can significantly improve the stability and reliability of the system. Through the above-mentioned reward and penalty measures, the intelligent agent can be motivated to adopt a more reasonable and effective operation strategy, thereby achieving the following specific beneficial effects:

[0141] By penalizing invalid actions, reclosing phenomena, excessive switching, and incorrect operation sequences, the intelligent agent will be more cautious when performing operations and avoid unnecessary operations, thereby improving overall operational efficiency.

[0142] By penalizing excessive switching and operating sequence errors, equipment wear can be reduced, equipment service life can be extended, and maintenance costs can be reduced.

[0143] Frequency deviation penalty and power factor penalty ensure that the frequency and power factor of the power system remain within a reasonable range, thereby ensuring the stable operation of the system.

[0144] Temperature overlimit penalties can prevent power distribution equipment from overheating due to overload, avoiding equipment damage and potential safety risks.

[0145] Through the reward and punishment mechanism, the intelligent agent will pay more attention to the radial topology of the distribution network when performing switching operations, thereby optimizing the network structure and improving power supply reliability.

[0146] When a transient fault occurs, by punishing frequent reclosing operations, unnecessary interference can be avoided and the system's response capability to transient faults can be improved.

[0147] Through the reward and punishment mechanism, the intelligent agent continuously learns and adjusts its strategy during operation, thereby improving its decision-making ability and better adapting to the complex and changing distribution network environment.

[0148] In summary, by introducing the reward and punishment mechanism, the intelligent agent can be effectively guided to adopt a more reasonable and efficient operation strategy, thereby improving the overall performance and reliability of the distribution network. These measures not only help to ensure the stable operation of the power system, but also reduce operating costs and improve economic benefits.

[0149] S2: updating the intelligent agent based on the first load transfer reinforcement learning model, and establishing an updated second load transfer reinforcement learning model;

[0150] The environmental model of the distribution network is introduced above, where the state space of the environment is S, the environmental state is used as the input of the reinforcement learning agent, and the action space of the environment is A, which is also the output of the agent. Since the voltage vector V and the current vector I in S are both continuous variables, the Q-learning algorithm cannot use continuous variables as input, so this application uses the DQN deep reinforcement learning algorithm to construct the agent of the distribution network load transfer reinforcement learning algorithm. DQN uses a deep neural network to generate Q values ​​to map states and actions to each other, continuously adjusts network parameters from a large amount of training, and seeks the optimal control strategy that satisfies the maximum return online.

[0151] When calculating the target value of the Q value, the DQN algorithm selects the action with the largest Q value in the next state s in the target network based on the reward part RT. This method of calculating the target value can easily cause DQN to overestimate the target Q value of the action. Overestimation can easily lead to uneven distribution of Q values, making the final result of deep reinforcement learning inaccurate, making it difficult to find the optimal solution. DoubleDQN has improved the update calculation method of the DQN algorithm. Since the target network lags behind the estimated network in update, DoubleDQN does not directly use the target network to calculate the target value. Instead, when estimating the next state, it first selects the action based on the estimated network, and then uses the target network to calculate the target value. This avoids blindly overestimating the Q value and improves accuracy, but it still does not consider the impact of environmental factors, and its actual use is subject to certain restrictions.

[0152] In the process of power distribution network transfer, the same action in different states may bring completely different consequences. For example, when a ring network is formed, opening a switch can break the ring, but opening the switch in other states may cause the downstream load to lose power supply. At this time, these states have different effects on the same action. In order to consider the impact of environmental factors and the rewards brought by actions at the same time, this application adopts the Dueling DQN algorithm that considers environmental impact, and divides the Q value function into environmental information rewards and action rewards, making the learning goal clearer.

[0153] In the embodiment of the present application, updating the agent according to the state space S and establishing the updated reinforcement learning model of the second load transfer includes:

[0154] Since the voltage vector group V and the current vector group I in the state space S are both continuous variables, the improved DQN deep reinforcement learning algorithm is used to construct the intelligent agent of the distribution network load transfer reinforcement learning algorithm;

[0155] The improved DQN deep reinforcement learning algorithm takes the state vector as input and outputs a vector containing the Q value of each action. The length of the output vector is the number of actions.

[0156] Improve the value function V(s) in the DQN deep reinforcement learning algorithm i ) is a scalar value, which is only related to the state and has nothing to do with the action, and represents the evaluation of the environmental state factors;

[0157] The output of the advantage function A(at) is a vector, and its length is also the number of actions. A(at) is centered, that is, the average value is subtracted from the entire vector, eliminating the common environmental influence, reflecting the identifiability of the action, and used to evaluate the additional benefits brought by the action:

[0158]

[0159] Among them, ω, α, and β are the common hidden layer parameters, value function layer parameters, and advantage function layer parameters, respectively. Λ represents the set of all actions. The advantage function A is centralized to highlight the differences in actions and reflect the identifiability of actions.

[0160] It should be noted that by adopting the improved Dueling DQN deep reinforcement learning algorithm, the intelligent agent can more accurately evaluate the rewards of different actions in a specific environment during the load transfer process of the distribution network. This algorithm separates the value function and the advantage function, allowing the intelligent agent to better understand the relationship between the environmental state and the action, and thus make more reasonable decisions.

[0161] Specifically, the output of the value function is only related to the state of the environment, and can provide the agent with an assessment of the current state of the environment. This helps the agent to quickly identify the optimal strategy under the current state when facing a complex and changing distribution network environment. The output of the advantage function highlights the differences between different actions, allowing the agent to make choices based on the additional benefits brought by the action, thereby achieving the optimal load transfer while meeting the system stability and reliability.

[0162] In addition, by centralizing the advantage function A, the common environmental influence is eliminated, allowing the agent to focus more on the differences in the actions themselves. This processing method helps the agent to more accurately evaluate the rewards of each action when facing similar environments but different consequences of different actions, thereby making more reasonable decisions.

[0163] S3: Introduce a pre-simulation optimization action mechanism, train the reinforcement learning model of the second load transfer, and obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, obtain the target distribution network load transfer solution.

[0164] In the embodiment of the present application, a pre-simulation optimization action mechanism is introduced, and the training of the reinforcement learning model for the second load transfer includes:

[0165] All actions in this state are sorted from large to small according to their Q values, and then P is used act Compare with invalid action set to filter out invalid / duplicate actions;

[0166] Keep the original order and take out the first k actions, apply all actions one by one to the current distribution network state for simulation pre-simulation, and exclude all over-limit actions according to the state change results;

[0167] Based on the action ranking generated in the previous step, the actions are sorted in three levels according to "load recovery amount > line loss value > Q value";

[0168] The final output is the optimal action at the head end of the pre-simulation sorting as the real action.

[0169] It should be noted that the load recovery amount is the primary consideration, which means that the agent gives top priority to how much load can be restored when choosing an action. This is because restoring more load means that more users can regain power supply, which is one of the main goals of load transfer in the distribution network. Therefore, giving priority to actions that can restore more load can improve the power supply situation of users as soon as possible. The line loss value is the second consideration, indicating that while ensuring that as many loads as possible are restored, it is also necessary to minimize the loss during power transmission. Reducing line loss can not only save energy, but also reduce the operating costs of power companies. Therefore, when the load recovery amount is the same, it is a wise choice to choose an action with less line loss. The Q value is the action evaluation value obtained by the agent through learning, which reflects the degree to which the agent believes that an action is good or bad in the current state. When the load recovery amount and line loss value are roughly the same, actions with higher Q values ​​usually mean better long-term benefits. Therefore, using the Q value as the third sorting criterion can help the agent make decisions that are more conducive to long-term development when faced with multiple similar options.

[0170] It should be noted that through the pre-simulation optimization action mechanism, the agent can more accurately evaluate the potential return of each action. This ensures that the selected action is verified by simulation, which can bring a large amount of load recovery while minimizing line loss, thereby improving the overall transfer efficiency and the stability of the power grid. The pre-simulation optimization mechanism makes it easier for the agent to find those truly good actions among a large number of actions, which can guide the agent to the optimal action strategy more quickly, thereby accelerating the training process of reinforcement learning. Although the pre-simulation mechanism modifies the original action mechanism of the algorithm, in order to avoid the action with the maximum Q value losing the opportunity to be verified, the practice of retaining a certain proportion of the actions with the maximum Q value in the greedy strategy is introduced. This not only increases the simulation verification opportunities for the action with the maximum Q value, but also prevents the problem of reduced training efficiency due to excessive randomness. Retaining the action with the maximum Q value as one of the candidate actions is a balance between exploring unknown actions and using known good actions, which helps the agent to explore new strategies and use existing good strategies during the learning process. Since there is a large amount of distribution network operation data in the input data, power failure may cause a sudden change in the input data, resulting in neural network learning oscillation and difficulty in convergence. Retaining the action mechanism in a certain degree of greedy action strategy can take into account the efficiency of action selection and the convergence performance of the neural network, ensuring good training results under fluctuating input data.

[0171] It should be noted that the pre-simulation optimization mechanism makes it easier for the agent to select the truly good actions from a large number of actions, accelerating the training of reinforcement learning to the optimal action strategy. However, the pre-simulation mechanism modifies the original action mechanism of the algorithm. In the existing formula, the action used to calculate the Q value target value is the action with the maximum Q value a. max In some cases, the action with the maximum Q value is not the optimal action. When the pre-simulation strategy is used, the agent will not choose the action with the maximum Q value. max , the action corresponding to the maximum Q value loses the opportunity to be simulated and verified, but its excessively large Q value participates in the calculation of the target value, resulting in an overestimation of the target value, which leads to unstable reinforcement learning training. max In order to increase the simulation verification opportunity, the proportion of random actions can be increased, but it is difficult to randomly get the action, and too much random proportion seriously affects the training efficiency of reinforcement learning, resulting in a large number of invalid samples. Therefore, the final solution is to retain a certain proportion of the actions with the largest Q value in the greedy strategy.

[0172] The pre-simulation-greedy action strategy can make it easier to find the best action during reinforcement learning training, speeding up the training. At the same time, since the input data is a large amount of distribution network operation data, power failure will cause a large amount of input data mutations. Input data with too large changes often causes neural network learning to oscillate and be difficult to converge. Retaining a certain degree of action mechanism in the greedy action strategy can take into account the efficiency of action selection and the convergence performance of the neural network, ensuring a good training effect under fluctuating input data, so that the neural network can clearly distinguish the action with the largest Q value and the truly optimal action in each state.

[0173] In the embodiment of the present application, in the conventional Dueling DQN algorithm, the agent calculates the error and updates the learning once for each action, but in fact, high-frequency learning does not increase the training speed of the neural network. In the case of high-dimensional input and output and huge neural network parameters in the present application, the neural network parameters will oscillate repeatedly, and its training loss function will also be difficult to converge. Therefore, the present application adjusts the action and learning ratio to 1 learning per 5 actions.

[0174] In the embodiments of the present application, when training a neural network, traditional optimization algorithms such as the stochastic gradient descent method SGD and Momentum use a fixed learning rate, and the learning rate needs to be repeatedly adjusted as a hyperparameter, which leads to a slow convergence speed and is prone to falling into a local optimal solution; the adaptive learning rate optimization algorithm AdaGrad automatically adjusts different learning rates for different parameters, and has a fast learning speed, but the learning rate approaches 0 as the number of iterations increases; the Adadelta optimization algorithm used in the present application can accelerate the training of the neural network without setting the learning rate for trial and error, and avoids the problem of a lower and lower learning rate.

[0175] In the embodiment of the present application, a pre-simulation optimization action mechanism is introduced, and the specific process of training the reinforcement learning model for the second load transfer is as follows:

[0176] Step 1: Determine the number of iterations N, estimate the network learning ratio P, the target network update frequency M, initialize the neural network parameters, the experience replay pool, the total number of actions n = 0, and the cumulative number of iterations episode = 0.

[0177] Step 2: A random fault occurs in the distribution network, and a new round of load transfer control begins. The distribution network state is initialized to the current fault state s0, the number of iterations episode = episode + 1, and the number of actions in this round is initialized to k = 0.

[0178] Step 3: The environment reads the distribution network state space S, inputs it into the estimation network, and obtains the action evaluation value Q.

[0179] Step 4: Select the corresponding action according to the “pre-simulation-greedy action strategy”.

[0180] Step 5: The environment executes the switch action, the environment state changes, and the action process is stored in the experience pool.

[0181] Step 6: After every P actions, randomly sample from the experience pool, calculate the loss function error, and update the estimation network.

[0182] Step 7: After every M actions, copy the estimated network parameters to the target network.

[0183] Step 8: Determine whether the state after transfer is in the end state. If not, continue from step 3. If the transfer is completed in this round, the transfer control strategy is given if the transfer is successful.

[0184] Step 9: Determine whether the total number of iterations episode reaches the set number of iterations N. If not, start the next round of transfer. If yes, end the training, output the neural network parameters, and the model training is completed.

[0185] In the embodiment of the present application, in order to further improve the efficiency and reliability of load transfer in the distribution network, a reinforcement learning model based on the pre-simulation optimal action mechanism is introduced. The model can effectively avoid possible mistakes and losses in actual operation by combining simulation rehearsal with actual operation.

[0186] Specifically, the introduction of the pre-simulation optimization action mechanism enables the agent to more accurately evaluate the potential return of each action when making load transfer decisions. In this way, the agent can choose actions more reasonably, thereby improving the overall transfer efficiency and grid stability.

[0187] During the implementation process, the current state of the distribution network is first analyzed to determine the possible action set. Then, through the pre-simulation mechanism, each action is simulated and rehearsed to evaluate its impact on the grid state. On this basis, combined with the Q value sorting, priority is given to actions that can bring larger load recovery and smaller line loss values.

[0188] In addition, in order to ensure the stability and convergence of the training process, the adaptive learning rate optimization algorithm Adadelta is also introduced in the embodiment of the present application. The algorithm can automatically adjust the learning rate according to the update of the model parameters, thereby avoiding the problems of slow convergence and local optimal solution caused by the fixed learning rate in the traditional optimization algorithm.

[0189] During the training process, the agent continuously learns and optimizes its decision-making strategy through interaction with the environment. After a certain number of actions, the agent will randomly sample from the experience pool, calculate the loss function error, and update the network parameters based on the error. In this way, the agent can gradually improve the accuracy and efficiency of its decision-making.

[0190] Finally, when the training reaches the preset number of iterations, the model training is completed. At this point, the agent has the ability to efficiently transfer loads under various distribution network failure conditions. Through practical applications, the performance of the model in the actual distribution network can be verified to ensure its reliability and effectiveness in actual operations.

[0191] In summary, the embodiment of the present application effectively improves the intelligent level of load transfer in the distribution network by introducing the pre-simulation optimization action mechanism and the adaptive learning rate optimization algorithm. This model can not only improve the transfer efficiency, but also ensure the stable operation of the power grid, and has important practical application value.

[0192] This embodiment also provides a distribution network load transfer solution system, including:

[0193] The first model building module is used to obtain the real-time status information of the target distribution network and establish a reinforcement learning model of the first load transfer in combination with the Markov decision process;

[0194] A second model building module, used to update the intelligent agent based on the reinforcement learning model of the first load transfer, and to build an updated reinforcement learning model of the second load transfer;

[0195] The training and solving module is used to introduce a pre-simulation optimization action mechanism, train the reinforcement learning model of the second load transfer, and obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, obtain the target distribution network load transfer solution result.

[0196] The above-mentioned unit modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above-mentioned modules.

[0197] This embodiment also provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for solving the load transfer of a distribution network is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0198] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0199] Obtain the real-time status information of the target distribution network, and establish a reinforcement learning model for the first load transfer by combining the Markov decision process;

[0200] updating the agent based on the reinforcement learning model of the first load transfer, and establishing an updated reinforcement learning model of the second load transfer;

[0201] A pre-simulation optimization action mechanism is introduced to train the reinforcement learning model of the second load transfer to obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, the solution result of the load transfer of the target distribution network is obtained.

[0202] Example 2

[0203] Reference Figure 2-Figure 6Tables 1 and 2 are an embodiment of the present invention, which provide a method and system for solving the load transfer problem in a distribution network. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0204] In order to verify the effectiveness of the algorithm in this application, the IEEE 33-node distribution system is used for verification. The distribution network system contains 33 nodes (numbered 1 to 33) and 37 branches, of which there are 32 section switches and 5 tie switches. The base voltage of the distribution network system is 12.66 kV, the total load is 3826.06 kW + j2366.49 kvar, the base system power is 10 MVA, and the topology is as follows: Figure 2 The top layer is numbered 18, 19, 20, 21 from left to right, the second layer is numbered 1-17 from left to right, the fourth layer is numbered 22, 23, 24 from left to right, and the third layer is numbered 24-33 from left to right;

[0205] The distribution network load transfer program based on deep reinforcement learning is written in Python. The code script is mainly divided into three parts:

[0206] (1) Environment: Using the Component Object Model (COM) interface technology of the open source three-phase distribution network flow simulation software (OpenDistributed System Simulator, OpenDSS) developed by the American Electric Power Research Institute, the distribution network modeling, simulation, operation, data extraction and other functions of OpenDSS are embedded in the deep reinforcement learning environment Python program, so that the deep reinforcement learning program can perform any operation on the IEEE-33 node distribution network during the training process. Analyze the operating status of the distribution network and output instant rewards and real-time operating status data; reset the distribution network environment before each transfer round, and automatically generate new random fault samples; execute input actions, update the distribution network status and output; generate a simulation environment and pre-simulate the optimal action.

[0207] (2) Agent part: Dueling-DQN algorithm model built based on Python's deep learning tool Tensorflow. Implement algorithm initialization and construct neural network parameters; according to the input of real-time status data of the distribution network, the neural network outputs the action Q value vector and adopts the pre-simulation-greedy action strategy to select the action; the executed actions are saved to the experience replay pool for calculating the loss function and updating the neural network parameters.

[0208] (3) Main program: The main program implements the overall iterative control, learning control, and action control functions, controls the interaction process between the intelligent agent and the environment, and is the overall process of the algorithm. It also records the interaction process between the intelligent agent and the environment and is used to output the final analysis results.

[0209] When using OpenDSS simulation to generate fault sample data, a method of randomly generating fault locations was adopted. That is, one of the 32 normally closed section switches was randomly selected to have a fault and opened directly, and the probability of each switch being selected was the same. The other five normally open contact lines did not have any faults because they were used as backup lines. Before each round of action, the fault line was cut off and simulated to determine the initial state of the distribution network. In this state, the agent will be punished for choosing to operate the fault line, and the current round of transfer decision will be ended when the state is transferred to the end state, and a new random fault will be automatically generated to start the next round of load transfer.

[0210] The deep neural network of Dueling-DQN algorithm has two public hidden layers, containing 512 and 256 neurons respectively. The value function layer and advantage function layer are both one layer. The output of one neuron of the value function is a scalar. The advantage function has 75 neurons and the output is a vector with the same number of actions. The activation function uses the linear rectified unit ReLU. For the parameter initialization in the algorithm, the number of batch training is 200, that is, 200 data are extracted for each learning to calculate the loss function, the reinforcement learning discount factor γ is 0.9, the exploration value ε is 0.95, the experience replay pool can store 1000 records, and the optimization algorithm of neural network learning uses the Adadelta algorithm. There is no need to set the learning rate, and the target network update frequency is 200.

[0211] The algorithm in this application converges after 50,000 action rounds. When making a decision on the transfer of power, the load recovery amount and operation and maintenance cost of the action are improved to the best in many aspects, such as Figure 3 shown.

[0212] At the beginning of training, the intelligent agent was not familiar with the distribution network transfer environment. The number of initial actions was large and the power supply load ratio was low, so the transfer control effect was poor. As the number of training times continued to increase, the intelligent agent continued to interact with the environment to gain experience, and learned how to restore more loads with fewer actions, thus achieving the optimal decision-making control of load transfer.

[0213] In the 14,652 rounds of training, due to a certain proportion of exploratory random actions and the inability to achieve complete transfer, 8,874 transfers were finally able to restore the full load. Figure 4In order to restore the line loss rate with the number of training times under full load, the agent learned the control strategy with the best operating cost in the later stage. The numerical labels of the left ordinate in the figure are evenly distributed from 4% to 6%;

[0214] The following is a comparative analysis of the transfer results and training effects.

[0215] (1) Comparison of load transfer results

[0216] The load transfer of the distribution network after the fault occurs is tested. Assuming that a permanent fault occurs in branch 12-13 and it is isolated, the load point 13-17 downstream of the fault point is the power-off area. The interconnection lines 8-14 and 17-32 can be used directly for transfer. However, when these two interconnection lines transfer the power-off load alone, the voltage will exceed the lower limit. Multiple actions must be used to transfer the load before transfer. The method of this application is compared with the reinforcement learning method without improved action strategy and the heuristic hybrid algorithm. The results are shown in Table 1:

[0217] Table 1 Comparison of load transfer results

[0218]

[0219] As can be seen from Table 1, the method of this application transfers part of the load on the heavy-load line to the light-load line through two connecting lines, restores all loads within a relatively small number of actions, and achieves a lower network loss level. For the reinforcement learning algorithm without improved actions, basically all attempted actions will lead to over-limit, so part of the load is eventually cut off. After the heuristic hybrid algorithm transfers the load, it still uses the heavy-load line for supply transfer. After restoring all loads, the line loss rate is 9.8% higher than that of the method of this application. The algorithm of this application basically implements the optimal control strategy.

[0220] In terms of computing time, the distribution network load transfer control method based on deep reinforcement learning does not need to spend a lot of time on simulation iteration after a fault. Its single action time is basically 0.04s, so the decision time mainly depends on the number of actions. In this example, most fault actions are 1-5 times, so the decision time of this application method is 0.04s-0.20s. The decision time of other traditional load transfer algorithms in IEEE 33 and smaller systems is compared as follows:

[0221] Table 2 Comparison of algorithm calculation speed

[0222]

[0223]

[0224] It can be seen from Table 2 that the trained deep reinforcement learning load transfer model can give control strategies in real time, restore power supply in a very short time, and achieve higher economic benefits.

[0225] (2) Comparative analysis of training effects

[0226] In order to verify the effectiveness of the distribution network load transfer control method based on Dueling-DQN proposed in this application, the scheduling method based on Dueling-DQN algorithm is compared with DQN and Double-DQN algorithms. The same reward function is set for them, and the average reward value obtained by the action in the training process is compared. Figure 5 shown. Figure 5-6 The left vertical axis value labels are 20%-100%;

[0227] In order to improve the action reward and speed up the training, the action selection method of the pre-simulation-greedy action strategy is adopted. The method of selecting actions according to the maximum Q value is retained in the deterministic strategy. Compared with the pre-simulation that completely replaces the maximum Q value selection method, it shows better convergence effect on input data training samples with smaller fluctuations. On input data samples with larger fluctuations, the latter may not converge at all.

[0228] Figure 6 The average loss of the training process of the DQN algorithm and the Dueling-DQN algorithm is compared. The Dueling-DQN algorithm shows better convergence ability and stability, and can reach optimal convergence in a shorter time. The characteristics and similarities and differences of the Q-learning algorithm and its evolutionary algorithm DQN deep reinforcement learning algorithm are pointed out that the DQN algorithm is more suitable for solving the continuous state and discrete action problems in the article. The difficulty of load transfer in the distribution network lies in the combinatorial explosion problem of the distribution network switches and the uncertainty of the topology. Therefore, a distribution network load transfer model based on deep reinforcement learning is established. The Dueling-DQN algorithm is used to divide the Q value function into environmental information return and action return, avoiding modeling of the topological uncertainty of the distribution network. For the control strategy analysis of different faults, fault types and operation modes, there is no need to adjust the model. A pre-simulation-greedy action strategy improvement algorithm is proposed for the training efficiency of the algorithm, and the action learning ratio and optimization algorithm are also adjusted for the algorithm convergence problem. It is verified by real-time training in the distribution network reinforcement learning environment of dynamic simulation. Compared with the traditional load transfer algorithm, the algorithm of this application has faster calculation speed and better action strategy. By comparing the reward function with the loss function, it is proved that the Dueling-DQN algorithm has better decision-making effect by considering the environmental return. The improved algorithm of pre-simulation-greedy action strategy retains a certain Q maximum action, which takes into account the efficiency of action selection and the convergence performance of the neural network, and improves the convergence speed of the algorithm.

[0229] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for solving load transfer in a distribution network, characterized in that: include: Obtain the real-time status information of the target distribution network, and establish a reinforcement learning model for the first load transfer by combining the Markov decision process; updating the agent based on the reinforcement learning model of the first load transfer, and establishing an updated reinforcement learning model of the second load transfer; A pre-simulation optimization action mechanism is introduced to train the reinforcement learning model of the second load transfer to obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, the solution result of the load transfer of the target distribution network is obtained.

2. The method for solving the load transfer problem in the distribution network according to claim 1, characterized in that: The reinforcement learning model for establishing the first load transfer includes: At each moment t, the environment provides the agent with the current distribution network state space S. The result of the agent's analysis and decision-making is the action space A. The action space A is applied to the environment. The state transition probability of the environment is P. The immediate reward function fed back to the agent by the environment is R. For the load transfer problem of the distribution network, the state data generated by the distribution network at each moment is the state space S of the environment, so the state space S is expressed as: S=[V,I,SW,F] S in =[V in ,I in ,SW i ,F i ] Where V is the voltage vector group, which is used to represent the voltage values ​​of all phases at each node in the distribution network. in is the voltage value of the nth phase of the ith node; I is the current vector group, which is used to represent the current value of each phase in all lines in the distribution network, I in is the current value of the nth phase of the ith line; SW is the state value vector of all switches in the distribution network, SW i is the state of the i-th switch, 0 means open, 1 means closed; F is the vector representing the fault state of the distribution network line, F i is the fault status of the line numbered i, 0 indicates normal, and 1 indicates a fault occurs.

3. The method for solving the load transfer problem in the distribution network according to claim 2, characterized in that: The step of establishing the reinforcement learning model for the first load transfer further includes: Only one switch is switched in one action, and the topological difference between two adjacent states is only one switch. The load transfer ends within a limited number of actions, and an action is set to actively end the transfer; Note that the action space A of the agent is selected as [0,2N s ] integer range, totaling 2N s +1 action, where N s is the number of operable lines; When the action value is 2N s When , the representative does not take any action and exits, and this decision ends; When the action value is [0,2N s -1], A is calculated as follows: X=A mod 2 Where X represents the type of switch operation, X=0 represents an open switch, X=1 represents a closed switch, and Y represents the switch number, with Y ranging from 1 to N. s .

4. The method for solving the load transfer problem in the distribution network according to claim 3, characterized in that: The step of establishing the reinforcement learning model for the first load transfer further includes: The reward function R of the model consists of the reward part RT and the penalty part RP, that is: R=RT+RP in: Among them, N op is the cumulative number of actions in this round of transfer process, P res.k is the load restored by each action, ΔP is the approximate line loss value, and a, b, and c are the offsets added to the values ​​of each factor.

5. The method for solving the load transfer problem in the distribution network according to claim 4, characterized in that: The step of establishing the reinforcement learning model for the first load transfer further includes: RP=P Volt +P Lim +P Loop +P act +P Reclose +P OverSwitch +P Seq +P 其他 Among them, P Volt Represents voltage penalty, P Lim Represents the current penalty, P Loop Represents the ring network penalty, P act Invalid action penalty, P Reclose Reclosing penalty, P OverSwitch represents the excessive switching penalty, P Seq represents the operation order penalty, P 其他 Indicates other penalties.

6. The method for solving the load transfer problem in the distribution network according to claim 5, characterized in that: The step of establishing the updated reinforcement learning model for the second load transfer includes: Since the voltage vector group V and the current vector group I in the state space S are both continuous variables, the improved DQN deep reinforcement learning algorithm is used to construct the intelligent agent of the distribution network load transfer reinforcement learning algorithm; The improved DQN deep reinforcement learning algorithm takes the state vector as input and outputs a vector containing the Q value of each action, and the length of the output vector is the number of actions; The value function V(s) in the improved DQN deep reinforcement learning algorithm i ) is a scalar value, which is only related to the state and has nothing to do with the action, and represents the evaluation of the environmental state factors; The output of the advantage function A(at) is a vector, and its length is also the number of actions. A(at) is centered, that is, the average value is subtracted from the entire vector, eliminating the common environmental influence, reflecting the identifiability of the action, and used to evaluate the additional benefits brought by the action: Among them, ω, α, and β are the common hidden layer parameters, value function layer parameters, and advantage function layer parameters, respectively. Λ represents the set of all actions. The advantage function A is centralized to highlight the differences in actions and reflect the identifiability of actions.

7. The method for solving the load transfer problem in the distribution network according to claim 6, characterized in that: The reinforcement learning model for the second load transfer is trained to obtain a reinforcement learning model for the third load transfer; Based on the reinforcement learning model of the third load transfer, the target distribution network load transfer solution is obtained, including: Sort all actions in the current state according to their Q values ​​from large to small, using P act Compare with invalid action set to filter out invalid / duplicate actions; Keep the original order and take out the first k actions, apply all actions one by one to the current distribution network state for simulation pre-simulation, and exclude all over-limit actions according to the state change results; Based on the action ranking generated in the previous step, the actions are sorted into three levels according to "load recovery amount > line loss value > Q value"; The final output is the optimal action at the head end of the pre-simulation sorting as the real action.

8. A distribution network load transfer solution system, characterized in that: include: The first model building module is used to obtain the real-time status information of the target distribution network and establish a reinforcement learning model of the first load transfer in combination with the Markov decision process; A second model building module, used to update the intelligent agent based on the reinforcement learning model of the first load transfer, and to build an updated reinforcement learning model of the second load transfer; The training and solving module is used to introduce a pre-simulation optimization action mechanism, train the reinforcement learning model of the second load transfer, and obtain the reinforcement learning model of the third load transfer; based on the reinforcement learning model of the third load transfer, obtain the target distribution network load transfer solution result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images