A method and product for determining the on-off state of a tie switch of a power distribution line

Abstract

The application provides a kind of power distribution line contact state determination method and product of contact, and electronic equipment obtains the power distribution network state data of each node in area;Wherein, power distribution network state data includes: each node voltage, line current, load power, photovoltaic power generation power and switch state;The power distribution network state data of each node is input into state determination model, and the target strategy output by state determination model is obtained, wherein, target strategy includes the target on-off state of each switch;Control switch agent adjusts the state of each switch based on target strategy.The application analyzes power distribution network state data using state determination model, thereby accurately determining target strategy, based on the target on-off state of each switch in target strategy, control switch agent adjusts the state of each switch based on target strategy, thereby accurately and effectively adjusting switch, effectively solve photovoltaic power consumption and load imbalance and other problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system distribution network operation, and in particular to a method for determining the on-off state of a distribution line tie switch and a product. BACKGROUND

[0002] High proportion of new energy access to distribution network has become an important direction of energy development, and the proportion of renewable energy generation represented by photovoltaic continues to increase. Due to the randomness and intermittency of distributed photovoltaic power generation, its large-scale grid connection will cause frequent voltage fluctuations of the distribution network, greatly increase the risk of system voltage out-of-limit, and increase the operation risk and operating cost of the power system.

[0003] With the large-scale distributed photovoltaic and heating period electric heating load access to the power grid, the original voltage distribution and power flow distribution of the distribution network change, and when the photovoltaic power is concentrated in the daytime and the electric heating load is concentrated in the night, a certain pressure is brought to the distribution line and transformer capacity. Therefore, it is necessary to study, on the basis of the existing distribution network architecture, to change the on-off state of the existing distribution line tie switch, change the topology of the distribution network, optimize the power flow of the distribution network, ensure the heating demand of users and maximize the consumption of distributed photovoltaic. Research on multi-loop time-sharing multiplexing transformation strategy of distribution network can effectively solve the problems of photovoltaic consumption and load imbalance, realize efficient operation of the distribution network, and has important significance for reducing network loss, improving voltage quality, and improving power supply reliability and economy. SUMMARY

[0004] The present application provides a method for determining the on-off state of a distribution line tie switch and a product, to solve the problem of inaccurate determination of the on-off state of the distribution line tie switch in the prior art, leading to photovoltaic consumption and load imbalance.

[0005] In a first aspect, the present application provides a method for determining the on-off state of a distribution line tie switch, comprising:

[0006] Obtaining distribution network state data of each node in the region; wherein the distribution network state data includes voltage, line current, load power, photovoltaic power generation power and switch state;

[0007] Inputting the distribution network state data of each node into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy includes a target on-off state of each switch;

[0008] Controlling a switch agent to adjust the state of each switch based on the target strategy.

[0009] In one possible implementation, the state determination model is a Q-network.

[0010] In a possible implementation, the state determination model is trained in the following manner:

[0011] randomly sampling a target sample from the replay buffer;

[0012] calculating a target value based on the target network and the next state in the target sample;

[0013] updating parameters of the original state determination model using the calculated target value, optimizing network parameters through a back propagation algorithm so that the predicted value of the state determination model for a given state and action approaches the target value;

[0014] based on the updated state determination model, calculating a difference between the target value and a predicted value output by the original state determination model for the sampled experience, and determining a target loss value based on the difference;

[0015] training the original state determination model based on the target loss value.

[0016] In a possible implementation, the target value is calculated based on the target network and the next state in the target sample, including:

[0017] the target value is calculated using the following formula:

[0018] y=r i +γmaxQ target (s t+1 ,a ++1 )

[0019] wherein r i is a reward value in the target sample, γ is a discount factor, Q target is the target network, s t+1 is the next state in the target sample, and a t+1 is an action that maximizes Q target (s t+1 ,a t+1 ) at state s t+1 .

[0020] In a possible implementation, the samples in the replay buffer are constructed in the following manner:

[0021] an action is selected in an ε-greedy policy at an initial state;

[0022] the action is performed in a pre-constructed environment, and an immediate reward value returned by the environment when the environment is transferred from the current initial state to a next state is obtained;

[0023] a four-tuple including the initial state, the performed action, the returned reward value, and the observed next state is stored as a sample in the replay buffer.

[0024] In a possible implementation, the objective function of the environment is constructed in the following manner:

[0025] According to the in-area network loss of the distribution network region accessing distributed photovoltaic, the balanced load, the distributed photovoltaic consumption level, the number of switch control, and the corresponding weights, respectively, the objective function is determined.

[0026] In a possible implementation, the in-area network loss of the distribution network region of the distributed photovoltaic is determined in the following manner:

[0027]

[0028] In the formula, f1 is the sum of branch active loss all day, ij is the branch connected by node i and node j; X ij is the open state of node i to node j, 1 represents closing, and 0 represents opening; N is the set of all lines of the distribution network; T is the total number of time periods; r ij is the resistance of node i to node j; t is the time period mark; I ij,t is the current of node i to node j in the tth time period; P ij and Q ij are active and reactive power of node i to node j, respectively; is the voltage of the end node of the branch ij;

[0029] The balanced load is determined in the following manner:

[0030]

[0031] In the formula, N is the set of all lines of the distribution network; L ij is the load rate of node i to node j; P ij and Q ij are active and reactive power of node i to node j, respectively; is the average load rate of the line;

[0032] The distributed photovoltaic consumption level is determined in the following manner:

[0033]

[0034] In the formula, n pv is the total number of distributed photovoltaic; T is the total time period; is the predicted value of active power of the distributed photovoltaic on node i in the tth time period, is the actual output value of active power of the distributed photovoltaic on node i in the tth time period;

[0035] The number of switch control is determined in the following manner:

[0036]

[0037] where ij represents the switch ij connecting node i and node j, N b is the total number of branches; H ij (t) is the on-off state of switch ij at time t, H ij (t-1) is the on-off state of switch ij at time t-1, which is usually represented by binary 1 and 0, 1 representing closed and 0 representing open.

[0038] In a possible implementation, the constraint condition satisfied by the state space of the environment is confirmed by the following way:

[0039] Power balance constraint:

[0040]

[0041] where, u i and u j represent the node voltage of node i and node j, respectively; i ij represents the current of branch i-j with node i as the starting point and node j as the ending point; P ij and Q ij represent the active and reactive power flowing through branch i-j, respectively; r ij and x ij represent the resistance and reactance of branch i-j, respectively; P j and Q j represent the net active and reactive load injected into node j; jk represents the set of branches connected to node j with node j as the starting point and node k as the ending point; P jk and Q jk represent the active and reactive power flowing through branch j-k;

[0042] Distributed photovoltaic power generation constraint:

[0043]

[0044] where, P pv,n and Q pv,n represent the active and reactive output of distributed photovoltaic user n, respectively; represents the maximum active output of distributed photovoltaic user n; and represent the upper and lower limits of the reactive output of distributed photovoltaic user n, respectively;

[0045] Node voltage constraint:

[0046] V min ≤ V i ≤ V max

[0047] wherein V min and V max are the minimum and maximum allowable operating values of the node voltage, respectively, and V i is the voltage value of node i.

[0048] Transformer capacity constraint:

[0049] S up ≤ aS rated

[0050] S down ≤ aS rated

[0051] wherein S rated is the rated capacity of the transformer, S up is the apparent power sent on the transformer; S down is the apparent power sent on the transformer, and a is a preset proportion value.

[0052] Branch current constraint:

[0053]

[0054] wherein I ij (t) is the amplitude of the current flowing on branch ij at time t, I ij，max (t) is the maximum amplitude of the current allowed to flow on branch ij at time t; G ij is the conductance of branch ij, B ij is the susceptance of branch ij; V i (t), V j (t) are the voltage values of node i and node j at time t, respectively; and θ ij is the phase difference between nodes ij.

[0055] In a possible implementation, the reward mechanism of the environment is confirmed by the following ways:

[0056] Node voltage reward:

[0057]

[0058] wherein A is a first preset value, U I,min is the lower limit of voltage U I，max is the upper limit of voltage, I I is the voltage value of node i at time t.

[0059] Switch transformation reward:

[0060] r2 = -B + B * p success

[0061] In the formula, B is a second preset value, p is 1 if the power flow calculation is met when the network switch transformation action is performed each time, and p is 0 if the power flow calculation is not met. success success

[0062] In a second aspect, the embodiments of the present application further provide a determination device for on-off states of tie switches of a power distribution line, the device comprising:

[0063] a obtaining module, configured to obtain power distribution network state data of each node in a region, wherein the power distribution network state data comprises voltage, line current, load power, photovoltaic power generation power and switch state;

[0064] a processing module, configured to input the power distribution network state data of each node into a state determination model, and obtain a target strategy output by the state determination model, wherein the target strategy comprises target on-off states of each switch;

[0065] a control module, configured to control a switch agent to adjust the states of each switch based on the target strategy.

[0066] In the embodiments of the present application, the electronic device obtains power distribution network state data of each node in a region, wherein the power distribution network state data comprises voltage, line current, load power, photovoltaic power generation power and switch state; inputs the power distribution network state data of each node into a state determination model, and obtains a target strategy output by the state determination model, wherein the target strategy comprises target on-off states of each switch; and controls a switch agent to adjust the states of each switch based on the target strategy. The embodiments of the present application analyze the power distribution network state data by using the state determination model, thereby accurately determining the target strategy, controlling the switch agent to adjust the states of each switch based on the target on-off states of each switch in the target strategy, accurately and effectively adjusting the switches, effectively solving problems such as photovoltaic power consumption and load imbalance, realizing efficient operation of the power distribution network, reducing network loss, improving voltage quality, and improving power supply reliability and economy. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings, which form a part of the present description, illustrate the present application and together with the written description serve to explain the principles of the present application. In the drawings:

[0068] Figure 1 FIG. 1 is a schematic diagram of a determination process for on-off states of tie switches of a power distribution line according to an embodiment of the present application;

[0069] Figure 2 FIG. 2 is a detailed process schematic diagram of determination of on-off states of tie switches of a power distribution line according to an embodiment of the present application; ​​

[0070] Figure 3 A schematic diagram of a determination structure of on-off states of a distribution line tie switch is provided for an embodiment of the present application.

[0071] Figure 4 A schematic diagram of an electronic device structure is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0072] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0073] The following detailed description is exemplary and is intended to provide further details of the present application. Unless otherwise defined, all technical terms used in the present application have the same meanings as generally understood by those skilled in the art. The terms used in the present application are only for the purpose of describing the specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0074] In order to accurately and effectively adjust the switch, effectively solve the problems of photovoltaic consumption and load imbalance, realize efficient operation of the distribution network, reduce network loss, improve voltage quality, improve power supply reliability and economy, the present application provides a determination method and product of on-off states of a distribution line tie switch.

[0075] In the embodiments of the present application, an electronic device obtains distribution network state data of each node in a region; wherein the distribution network state data includes: voltage, line current, load power, photovoltaic power generation power and switch state of each node; the distribution network state data of each node is input into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy includes a target on-off state of each switch; and a switch agent is controlled to adjust the state of each switch based on the target strategy.

[0076] Embodiment 1:

[0077] Figure 1 A schematic diagram of a determination process of on-off states of a distribution line tie switch is provided for an embodiment of the present application, as shown in Figure 1 The process includes the following steps:

[0078] S101: Obtain distribution network state data of each node in a region; wherein the distribution network state data includes: voltage, line current, load power, photovoltaic power generation power and switch state.

[0079] The determination method of on-off states of a distribution line tie switch provided by the embodiments of the present application applies an electronic device, which can be a PC or a server or other intelligent device.

[0080] In order to accurately adjust each switch, the electronic device can first acquire power distribution network state data of each node in the region, wherein the power distribution network state data includes voltage, line current, load power, photovoltaic power generation power and switch state. In a possible implementation, each power distribution network state data can be acquired by deploying a supervisory control and data acquisition (SCADA) system, a smart meter, a distributed energy monitoring system and a power distribution automation device. Specifically, the electronic device acquires the voltage amplitude and phase angle of each node from the SCADA system, acquires the active power and reactive power of each node from the smart meter, acquires the photovoltaic power generation power of each node from the distributed energy monitoring system, and acquires the voltage amplitude and phase angle of each branch from the power distribution automation device.

[0081] It should be noted that after acquiring the power distribution network state data, the electronic device can preprocess the acquired power distribution network state data, that is, to identify and repair abnormal data, remove invalid data, and fill in missing data. In addition, considering the dynamic changes of the power distribution network, the electronic device needs to monitor the power distribution network state data in real time, and set the power distribution network data update frequency to 10 minutes according to the actual demand, so as to realize real-time data acquisition and update.

[0082] S102: input the power distribution network state data of each node into a state determination model, and acquire a target strategy output by the state determination model, wherein the target strategy includes a target on-off state of each switch.

[0083] In order to determine the target on-off state of each switch, the electronic device locally stores a pre-trained state determination model. After acquiring the power distribution network state data, the electronic device can input the power distribution network state data of each node into the state determination model, and acquire a target strategy output by the state determination model. The state determination model can be a Q network.

[0084] It should be noted that the electronic device can also acquire power distribution network state data of each node in the region, photovoltaic power generation and electric heating load prediction data, and input the power distribution network state data of each node, photovoltaic power generation and electric heating load prediction data into the state determination model.

[0085] The target strategy includes a target on-off state of each switch.

[0086] In the embodiment of the application, an improved deep Q network algorithm can be used to solve the power distribution network time-sharing multiplexing transformation problem. Compared with the traditional DQN algorithm, the action space is changed. The input of the Q network is a state vector s, and the output is a Q value vector corresponding to each action. The data is processed by a denoising autoencoder to remove redundant information in the action space.

[0087] and can be added by adding Gaussian noise in the input data of the switch agent, Gaussian noise is a kind of random noise conforming to normal distribution, and the probability density function is:

[0088]

[0089] In the formula, sigma is the standard deviation; x is a random variable; mu is the mean. The de-noising process is first performed by the self-encoder, and the processed data is transmitted to the DQN for decision-making, so that the switch agent has stronger robustness to environmental changes and enhanced generalization ability.

[0090] S103: controlling the switch agent to adjust the state of each switch based on the target policy.

[0091] In the embodiments of the present application, the electronic device is connected with the switch agent, and the electronic device controls the switch agent to adjust the state of each switch based on the target policy after obtaining the target policy.

[0092] It should be noted that the state determination model proposed in the embodiments of the present application can be referred to as a power distribution network time-sharing multiplexing transformation model. The power distribution network time-sharing multiplexing transformation model can utilize the distributed photovoltaic power and user electric heating load power obtained by future prediction, and obtain an optimized transformation strategy of the power distribution network tie line switch. The power distribution network time-sharing multiplexing transformation model is mainly used to improve the voltage quality and line load imbalance problem on the basis of guaranteeing the heating demand of residents, and promote the efficient consumption of distributed photovoltaic power. According to the characteristics of distributed photovoltaic power generation and electric heating load, the specific application can be divided into the following two scenarios:

[0093] (1) Distributed photovoltaic centralized power generation scenario in daytime: When the distributed photovoltaic power is concentrated in the daytime, a large amount of photovoltaic power is sent to the power distribution network through the grid connection point, which is easy to cause line reverse power flow out-of-limit and transformer reverse heavy overload. By using the time-sharing multiplexing transformation strategy of the power distribution network tie line switch, the power distribution network topology is changed, the power transmission path of photovoltaic power generation is changed, the power flow distribution and voltage distribution of the power distribution network are optimized, and the consumption level of distributed photovoltaic power is effectively improved.

[0094] (2) Night electric heating load power supply peak scenario: In the winter heating period, the night is the peak period of electric heating load, and the line power flow out-of-limit and transformer heavy overload problems are prominent. In the scenario of heavy load at night and no photovoltaic power generation, by using the time-sharing multiplexing transformation strategy of the power distribution network tie line switch, the power distribution network topology is changed, the purpose of multi-loop power supply is achieved, the power flow distribution and voltage distribution of the power distribution network are optimized, and the safety and reliability of the residents' heating are guaranteed.

[0095] In the embodiment of the present application, the electronic device acquires power distribution network state data of each node in the region; wherein the power distribution network state data includes: voltage of each node, line current, load power, photovoltaic power generation power and switch state; the power distribution network state data of each node is input into the state determination model, and a target strategy output by the state determination model is acquired, wherein the target strategy includes a target on-off state of each switch; and the switch agent is controlled to adjust the state of each switch based on the target strategy. In the embodiment of the present application, the state determination model is used to analyze the power distribution network state data, so as to accurately determine the target strategy. Based on the target on-off state of each switch in the target strategy, the switch agent is controlled to adjust the state of each switch based on the target strategy, so as to accurately and effectively adjust the switch, effectively solve the problems of photovoltaic power consumption and load imbalance, realize efficient operation of the power distribution network, reduce network loss, improve voltage quality, and improve power supply reliability and economy.

[0096] Embodiment 2

[0097] In order to accurately determine the on-off state of each switch, on the basis of the above-mentioned embodiment, in the embodiment of the present application, the state determination model is trained by the following method:

[0098] Randomly sampling a target sample from the replay buffer;

[0099] Based on the target network and the next state in the target sample, the target value is calculated;

[0100] The calculated target value is used to update the parameters of the original state determination model, and the network parameters are optimized by the back propagation algorithm, so that the prediction value of the state determination model for the given state and action tends to the target value;

[0101] Based on the updated state determination model, the difference between the target value and the prediction value output by the original state determination model is calculated for the sampled experience, and the target loss value is determined accordingly;

[0102] The original state determination model is trained based on the target loss value.

[0103] In the model training process, the replay buffer plays a crucial role. This buffer stores the experience data generated in the process of interaction between the agent and the environment. Each experience usually contains four parts: the current state, the action taken, the reward obtained after executing the action, and the next state. The electronic device can randomly sample a target sample from the replay buffer, wherein the target sample is randomly sampled to break the time correlation between the samples, thereby improving the stability and efficiency of learning.

[0104] During training, the electronic device periodically samples a batch of experience samples from the replay buffer. This randomness ensures that the model learns not only from recent experiences but also from a diverse history of past interactions, helping it generalize better.

[0105] For the sampled target samples, the electronic device uses the target network and the next state in the target samples to calculate the target value. The target network is a lagged version of the original state determination model, used to stabilize the training process.

[0106] The target value is usually calculated by inputting the next state into the target network to obtain the state value estimate or action value estimate, and adding the expected future reward that can be obtained from the next state to the terminal state.

[0107] The electronic device uses the calculated target value to optimize the parameters of the original state determination model through the backpropagation algorithm (a gradient descent algorithm). The goal of optimization is to make the predicted value of the state determination model under a given state and action as close as possible to the target value. In this way, the state determination model can learn which action to take in a particular state to achieve the maximum long-term return.

[0108] In each iteration, the electronic device calculates the difference between the target value and the predicted value output by the model based on the updated state determination model for each set of experience sampled from the replay buffer. This difference (usually measured by some form of loss function, such as mean squared error) measures the predictive performance of the model and serves as an important indicator in the training process.

[0109] Finally, based on the calculated target loss value, the electronic device trains the original state determination model. This usually involves using an optimizer (such as Adam or RMSprop) to update the weights of the model to minimize the loss value. Through continuous iteration of this process, the model can gradually learn a better strategy to make better decisions in a given environment.

[0110] The entire process is a closed-loop feedback system, through continuous learning and adjustment from the environment, the agent can gradually improve its performance to better adapt and respond to complex environmental challenges.

[0111] Before training, the state set s i , the action set a i , the iteration step α, the exploration rate ε, and the Q network, the target network and the replay buffer can be initialized.

[0112] Embodiment 3:

[0113] To determine the target value, based on the above embodiments, in the embodiments of the present application, the target value is calculated based on the target network and the next state in the target sample, and comprises:

[0114] The target value is calculated using the following formula:

[0115] y = r i + γmax Q target (s t+1 , a t+1 )

[0116] Where r i is the reward value in the target sample, γ is the discount factor, Q target is the target network, s t+1 is the next state in the target sample, a t+1 is the action that maximizes Q t+1 (s target , a t+1 ) at state s t+1 .

[0117] Where the action space, i.e. the adjustable variable of the tie line switch agent, is used to describe the specific action of the switch time-sharing multiplexing transformation of the distribution network, and the action space of the switch transformation of the distribution network is the key to determine how the distribution network optimizes operation. The action space defines the specific operation taken by the switch agent, which usually includes the open and close states of sectionalizing switches and tie switches. The action space of the tie line switch transformation of the distribution network is effectively expressed by a set of vectors a i , with dimension n; a i represents the action state of the i-th switch, 1 represents closed, and 0 represents open.

[0118] In the process of calculating the target value, the electronic device adopts a specific formula, which is a common method for estimating future cumulative rewards in reinforcement learning, especially in Q-value-based learning methods. This formula aims to evaluate the value of a state-action pair through the weighted sum of current rewards and future potential rewards.

[0119] The formula is:

[0120] y = r i + γmax Q target (s t+1 , a t+1 )

[0121] The details are as follows:

[0122] r i represents the immediate reward obtained from the environment, i.e. performing action a t in the current state s timmediate feedback from the environment. In real-world scenarios, r i can be any real number, representing positive or negative feedback obtained from the environment.

[0123] γ is a discount factor, a constant between 0 and 1, used to adjust the importance of future rewards on the current state-action pair value assessment. The closer γ is to 1, the greater the impact of future rewards on the current value; the closer γ is to 0, the smaller the impact of future rewards on the current value. This parameter helps the algorithm strike a balance between "instant gratification" and "long-term planning". target Q is the Q-value function of the target network, which is a lagging version of the original Q-value network, used to stabilize the training process and prevent training instability due to rapid changes in Q-values. The target network regularly or slowly updates its weights to track the weight changes of the original network.

[0124] s t+1 is the next state in the target sample, i.e. the new state the environment transitions to after the agent performs action a t in the current state s t . a t+1 This is the action selected in state s t+1 , in a way (such as a greedy policy) that maximizes Q target (s t+1 ,a t+1 ). This selection process embodies the agent's ability to predict and plan for future behavior. maxQ target (s t+1 ,a t+1 ) calculates the maximum Q-value of the target network in the next state s t+1 , by iterating through all possible actions a t+1 , finding the one that maximizes the Q-value, and calculating the corresponding Q-value. This represents the maximum expected future reward that can be obtained by taking the optimal action in a given next state.

[0125] This formula calculates the target value y by combining the immediate reward and the discounted maximum expected reward in the future, which is used as a target or benchmark for updating the original Q-value network during training. In this way, it can learn how to select the optimal action in a given state to maximize its cumulative reward.

[0126] Embodiment 4:

[0127] In order to accurately train the state determination model, on the basis of the above embodiments, in the embodiments of the present application, the samples in the replay buffer are constructed in the following way:

[0128] In the initial state, select an action with an ε-greedy policy;

[0129] performing the action in the pre-constructed environment, obtaining an immediate reward value returned by the environment when the environment transitions from the current initial state to a next state;

[0130] storing a quadruple including the initial state, the performed action, the returned reward value, and the observed next state into the replay buffer as a sample.

[0131] In the embodiments of the present application, the electronic device can use an ε-greedy policy to select the action to be performed in the initial state. The ε-greedy policy is a policy that balances exploration and exploitation. Specifically, the electronic device randomly selects an action with a probability of ε, and selects the action currently considered optimal with a probability of 1-ε. Here, ε is a parameter between 0 and 1, which can be adjusted according to the learning progress.

[0132] After selecting the action, the electronic device performs the action in the pre-constructed environment. The environment performs state transition according to certain rules based on the current state and the selected action, and returns an immediate reward value to the electronic device. This reward value is a direct feedback on the effect of the electronic device performing the selected action in the current state, which can be positive (indicating reward), negative (indicating punishment), or zero (indicating no effect).

[0133] The agent stores the experience data of this interaction in the form of a quadruple into the replay buffer. This quadruple includes:

[0134] the initial state, the performed action, the returned reward value, and the next state to which the environment transitions after performing the action.

[0135] This process is repeated continuously, and as the electronic device continues to interact with the environment, a large number of experience samples are gradually accumulated in the replay buffer. These samples will be used to train the original state determination model, and the parameters of the model will be optimized through the backpropagation algorithm, so that the agent can learn how to select the optimal action in different states to maximize the cumulative reward.

[0136] In summary, the construction process of the replay buffer embodies the ideas of trial-and-error learning and experience replay in reinforcement learning. Through continuous accumulation and utilization of experience data, the agent can gradually improve its decision-making ability in complex environments.

[0137] Embodiment 5:

[0138] In order to construct the environment, on the basis of the above embodiments, in the embodiments of the present application, the objective function of the environment is constructed in the following way:

[0139] According to the network loss, balanced load, distributed photovoltaic consumption level, switch control times in the distribution network area accessing distributed photovoltaic, and the corresponding weights, the objective function is determined.

[0140] In constructing the objective function for the optimal operation of the distribution network region involving access to distributed photovoltaics, the electronic device needs to consider multiple key indicators and their corresponding weights to ensure the economy, reliability and sustainability of the system.

[0141] Among them, the network loss in the distribution network region accessing distributed photovoltaics refers to the loss of electric energy caused by resistance, reactance and other factors in the process of power transmission and distribution. Reducing network loss is an important goal to improve the efficiency of the distribution network and reduce energy waste. In the objective function, network loss can be obtained by calculating the current, voltage and impedance of each line, and it is minimized as an important cost item.

[0142] Balanced load aims to reduce the load imbalance between nodes, feeders or transformers in the distribution network to avoid overload, low load and other adverse conditions, and improve the stability and reliability of system operation. By optimizing the output plan of distributed photovoltaics, the charging and discharging strategy of energy storage systems and the switching operation of the distribution network, the balanced distribution of load in different time and different regions is realized. In the objective function, load rate balance indicators such as load rate standard deviation or coefficient of variation can be introduced to evaluate the load balance level and strive to minimize it.

[0143] The level of distributed photovoltaic consumption refers to the degree of effective utilization of distributed photovoltaic power generation in the distribution network region. Improving the consumption level helps to improve the utilization rate of renewable energy and reduce the phenomenon of abandoned light. By optimizing the grid connection strategy of distributed photovoltaics, the configuration of energy storage systems and demand side response measures, the local consumption capacity of photovoltaic power generation is improved. In the objective function, the local consumption proportion of photovoltaic power generation or the light abandonment rate can be set as the optimization goal to strive to improve the consumption level.

[0144] Frequent switching (such as tie switches, sectional switches, etc.) in the distribution network not only increases equipment wear and tear, but also may affect power supply reliability. Therefore, unnecessary switching operation times should be minimized during optimization. In constructing the objective function, the cost item of switching control times can be introduced and minimized. At the same time, consider the constraint conditions of switching operation in the optimization algorithm, such as switching action time interval limit, switching state transition logic, etc., to ensure the feasibility and practicality of the optimization result.

[0145] The allocation of weights can be considered comprehensively according to the specific needs of the distribution network, policy guidance, economic benefits and other factors. In the objective function, each cost item (such as network loss cost, load imbalance cost, light abandonment cost, switching operation cost, etc.) will be multiplied by its weight and then summed up to form the final objective function. By adjusting the value of the weight, the emphasis and balance of different optimization goals can be realized.

[0146] Example 6:

[0147] In order to determine the objective function, on the basis of the above embodiments, in the embodiments of the present application, the distributed photovoltaic power distribution network regional network loss is determined by the following method:

[0148]

[0149] In the formula, f1 is the sum of branch active loss throughout the day, and ij is the branch connected by node i and node j; X ij is the open state of node i to node j, 1 represents closing, and 0 represents opening; N is the set of all lines of the power distribution network; T is the total number of time periods throughout the day; r ij is the resistance of node i to node j; t is the time period mark; I ij,t is the current of node i to node j in the tth time period; P ij and Q ij are active and reactive power of node i to node j, respectively; is the voltage at the end of the branch ij;

[0150] The balanced load is determined by the following method:

[0151]

[0152] In the formula, N is the set of all lines of the power distribution network; L ij is the load rate of node i to node j; P ij and Q ij are active and reactive power of node i to node j, respectively; is the average load rate of the line;

[0153] The distributed photovoltaic power consumption level is determined by the following method:

[0154]

[0155] In the formula, n pv is the total number of distributed photovoltaics; T is the total time period; is the predicted value of the active power of the distributed photovoltaic at node i in the tth time period, is the actual output value of the active power of the distributed photovoltaic at node i in the tth time period;

[0156] The switch control times are determined by the following method:

[0157]

[0158] In the formula, ij represents the switch ij connected by node i and node j, N b is the total number of branches; H ij (t) is the opening and closing state of the switch ij in the tth time period, H ij(t-1) is the on-off state of the t-1 period switch ij, the on-off state is usually represented by binary 1 and 0, 1 represents closing, and 0 represents opening.

[0159] Embodiment 7:

[0160] In order to construct the environment, on the basis of the above embodiments, in the embodiments of the present application, the constraint conditions satisfied by the state space of the environment are confirmed by the following methods:

[0161] Power balance constraint:

[0162]

[0163] Where, u i and u j respectively represent the node voltage of node i and node j; i ij represents the current of branch i-j with node i as the starting point and node j as the ending point; P ij and Q ij respectively represent the active and reactive power flowing through branch i-j; r ij and x ij respectively represent the resistance and reactance of branch i-j; P j and Q j respectively represent the net active and reactive load injected into node j; jk represents the branch set connected to node j with node j as the starting point and node k as the ending point; P jk and Q jk respectively represent the active and reactive power flowing through branch j-k;

[0164] Distributed photovoltaic power generation constraint:

[0165]

[0166] Where, P pv,n and Q pv,n respectively represent the distributed photovoltaic active and reactive power output of photovoltaic user n; represents the maximum value of the distributed photovoltaic active power output of user n; and respectively represent the upper and lower limits of the distributed photovoltaic reactive power output of user n;

[0167] Node voltage constraint:

[0168] V min ≤ V i ≤ V max

[0169] In the formula, V min and V max are the minimum and maximum values of the allowable operation of the node voltage, respectively, and V iVi(t) is the voltage value of node i at time t.

[0170] Transformer capacity constraint:

[0171] S up ≤ aS rated

[0172] S down ≤ aS rated

[0173] In the formula, S rated is the rated capacity of the transformer, S up is the apparent power sent on the transformer; S down is the apparent power sent on the transformer, and a is a preset proportion value.

[0174] In one possible implementation, to ensure that the transformer does not run under heavy load when the night electric heating load is centrally supplied and the daytime distributed photovoltaic power is centrally generated, the transformer sent power and the transformer sent power are limited to not exceed 80% of the rated capacity of the transformer, and therefore the preset proportion value can be 80%.

[0175] Branch current constraint:

[0176]

[0177] In the formula, I ij (t) is the amplitude of the current flowing through the branch ij at time t, I ij，max (t) is the maximum amplitude of the current allowed to flow through the branch ij at time t; G ij is the conductance of the branch ij, B ij is the susceptance of the branch ij; V i (t) and V j (t) are the voltage values of nodes i and j at time t, respectively; θ ij is the phase difference between nodes ij.

[0178] The branch current constraint can ensure that the branch current of the power distribution network does not exceed the limit when the night electric heating load is centrally supplied and the daytime distributed photovoltaic power is centrally generated.

[0179] It should be noted that the state space is one of the key steps for the agent to make decisions, and the state space represents the key information of the power distribution network at a certain moment, and should satisfy the above constraint conditions.

[0180] Embodiment 8:

[0181] To determine the reward value, on the basis of the above embodiments, in the embodiments of the application, the reward mechanism of the environment is confirmed by the following way:

[0182] Node voltage reward:

[0183]

[0184] Among them, A is the first preset value, and U I,min is the lower voltage limit U I，max is the upper voltage limit, and U I is the voltage value of node i at time t.

[0185] Switch transformation reward:

[0186] r2 = -B + B*p success

[0187] In the formula, B is the second preset value. Each time the network switch transformation action is executed, if the power flow calculation is satisfied, then p success is 1, and if the power flow calculation is not satisfied, p success is 0.

[0188] The basic framework of the reinforcement learning in this application includes five parts: agent, environment, state, action, and reward. It is a kind of learning that maps from the environmental state to the action, and the goal is to make the agent obtain the maximum cumulative reward during the interaction with the environment. In order to apply reinforcement learning to the efficient accommodation of distributed photovoltaics, this application constructs a reinforcement learning environment model suitable for network topology optimization according to the characteristics of the distribution network topology.

[0189] In the embodiment of this application, the electronic device can determine the reward value based on the node voltage reward and the switch transformation reward.

[0190] This application proposes a multi-loop time-sharing multiplexing transformation strategy for the distribution network based on reinforcement learning. First, a mathematical model is established for the multi-loop time-sharing multiplexing transformation problem of the distribution network: taking the minimum network loss, balanced load, distributed photovoltaic accommodation level, and switch control times as the objective function, and establishing the constraint conditions of the distribution network; secondly, an environment for simulating the distribution network is constructed based on deep reinforcement learning, including the state space, action space, and reward mechanism; finally, an improved deep Q network is used to train and solve the model, and finally the transformation strategy of the distribution network line switch is obtained to guide the time-sharing multiplexing transformation of the distribution network tie-line switch.

[0191] Figure 2 It is a detailed process schematic diagram for determining the on-off state of the tie switch of the distribution line provided in the embodiment of this application.

[0192] By Figure 2It can be seen that the electronic device can obtain power distribution network state data of each node in the region, photovoltaic power generation and electric heating load prediction data, input the power distribution network state data of each node, photovoltaic power generation and electric heating load prediction data into a state determination model, obtain a target strategy output by the state determination model, and control the switch agent to adjust the state of each switch based on the target strategy. Wherein, the state determination model is a pre-constructed power distribution network time-sharing service transformation reinforcement learning environment, and an improved DQN algorithm is used to train the model.

[0193] The power distribution network multi-loop time-sharing service transformation strategy based on reinforcement learning provided by the embodiments of the application has the following advantages:

[0194] (1) The deployed power distribution network state detection device can monitor the state of the power distribution network in real time. The real-time acquired data is processed and then input into a deep reinforcement learning model. The deep reinforcement learning trains the switch agent to enable it to make decisions quickly, and the real-time optimization of the time-sharing service transformation of the power distribution network line can be realized according to the decisions.

[0195] (2) The power distribution network time-sharing service transformation model is constructed based on the Markov decision process of reinforcement learning. The decision process can obtain a high-quality switch strategy of the power distribution network line through continuous search and trial in a dynamic environment, and improve the reliability and economy of the operation of the power distribution network after the distributed photovoltaic and electric heating load is connected.

[0196] (3) For the problem of the power distribution network time-sharing service transformation strategy based on reinforcement learning, the improved deep Q network algorithm is used to solve the problem. The algorithm reduces the dimension of the action space on the basis of the original deep Q network and introduces noise into the input sample features to learn more robust data coding. In the training process, it is not necessary to sample all the opening and closing states of the switches, which avoids local optimization and improves the data utilization efficiency.

[0197] Embodiment 9

[0198] Figure 3 The device includes:

[0199] The acquisition module 301 is configured to acquire power distribution network state data of each node in the region. The power distribution network state data includes voltage, line current, load power, photovoltaic power generation power and switch state.

[0200] The processing module 302 is configured to input the power distribution network state data of each node into a state determination model, and obtain a target strategy output by the state determination model. The target strategy includes target on-off states of each switch.

[0201] The control module 303 is configured to control the switch agent to adjust the state of each switch based on the target policy.

[0202] In a possible implementation, the processing module 302 is further configured to train the state determination model by the following method:

[0203] randomly sampling a target sample from the replay buffer;

[0204] calculating a target value based on a next state in the target sample and the target network;

[0205] updating parameters of the original state determination model using the calculated target value, optimizing network parameters through a back propagation algorithm, so that the predicted value of the state determination model for a given state and action tends to be close to the target value;

[0206] based on the updated state determination model, calculating a difference between the target value and a predicted value output by the original state determination model for the sampled experience, and determining a target loss value based on the difference;

[0207] training the original state determination model based on the target loss value.

[0208] In a possible implementation, the processing module 302 is further configured to calculate the target value by the following formula:

[0209] y = r i + γmaxQ target (s t+1 ,a t+1 )

[0210] wherein r i is a reward value in the target sample, γ is a discount factor, Q target is the target network, s t+1 is the next state in the target sample, and a t+1 is an action that maximizes Q t+1 (s target ,a t+1 ) at state s t+1 .

[0211] In a possible implementation, the processing module 302 is further configured to construct the sample in the replay buffer by the following method:

[0212] selecting an action in an initial state by an ε-greedy policy;

[0213] performing the action in a pre-constructed environment, and obtaining an immediate reward value returned by the environment when the environment is transferred from the current initial state to a next state;

[0214] A quadruple including an initial state, an executed action, a returned reward value, and an observed next state is stored into a replay buffer as a sample.

[0215] In a possible implementation, the processing module 302 is further configured to construct the objective function of the environment by the following method:

[0216] According to the grid loss in the distribution network region accessing the distributed photovoltaic, the balanced load, the distributed photovoltaic consumption level, the number of switch control, and the corresponding weights, respectively, the objective function is determined.

[0217] In a possible implementation, the processing module 302 is further configured to confirm the grid loss in the distribution network region of the distributed photovoltaic by the following method:

[0218]

[0219] In the formula, f1 is the sum of branch active power losses throughout the day, and ij is the branch connected by node i and node j; X ij is the opening and closing state of node i to node j, 1 represents closing, and 0 represents opening; N is the set of all lines of the distribution network; T is the total number of time periods throughout the day; r ij is the resistance of node i to node j; t is the time period index; I ij,t is the current of node i to node j in the tth time period; P ij and Q ij are the active and reactive power of node i to node j, respectively; is the voltage of the end node of the branch ij;

[0220] The balanced load is confirmed by the following method:

[0221]

[0222] In the formula, N is the set of all lines of the distribution network; L ij is the load rate of node i to node j; P ij and Q ij are the active and reactive power of node i to node j, respectively; is the average load rate of the line;

[0223] The distributed photovoltaic consumption level is confirmed by the following method:

[0224]

[0225] In the formula, n pv is the total number of distributed photovoltaics; T is the total time period; is the predicted value of the active power of the distributed photovoltaic on node i in the tth time period, Pij(t) is the actual active power output value of the distributed photovoltaic at node i at time period t;

[0226] The number of switch control times is determined by the following method:

[0227]

[0228] wherein ij represents the switch ij connected by node i and node j, N b N is the total number of branches; H ij (t) is the on-off state of switch ij at time period t, H ij (t-1) is the on-off state of switch ij at time period t-1, and the on-off state is usually represented by binary 1 and 0, 1 representing closed and 0 representing open.

[0229] In a possible implementation, the processing module 302 is further configured to determine that the state space of the environment satisfies the constraint condition by the following method:

[0230] Power balance constraint:

[0231]

[0232]

[0233] wherein u i and u j respectively represent the node voltage of node i and node j; i ij represents the current of branch i-j with node i as the starting point and node j as the ending point; P ij and Q ij respectively represent the active and reactive power flowing through branch i-j; r ij and x ij respectively represent the resistance and reactance of branch i-j; P j and Q j respectively represent the net active and reactive load injected into node j; jk represents the branch set connected to node j with node j as the starting point and node k as the ending point; P jk and Q jk respectively represent the active and reactive power flowing through branch j-k;

[0234] Distributed photovoltaic power generation constraint:

[0235]

[0236] wherein P pv,n and Q pv,n respectively represent the active and reactive output of the distributed photovoltaic of photovoltaic user n; represents the maximum active output of the distributed photovoltaic of user n; and upper and lower limits of the distributed photovoltaic reactive power output of the user n are represented by a table;

[0237] Node voltage constraint:

[0238] V min ≤V i ≤V max

[0239] wherein V min and V max are the minimum and maximum values of the allowable operation of the node voltage, respectively, and V i is the voltage value of the node i.

[0240] Transformer capacity constraint:

[0241] S up ≤aS rated

[0242] S down ≤aS rated

[0243] wherein S rated is the rated capacity of the transformer, S up is the apparent power sent on the transformer; S down is the apparent power sent on the transformer, and a is a preset proportion value.

[0244] Branch current constraint:

[0245]

[0246] wherein I ij (t) is the amplitude of the current flowing through the branch ij at the time period t, I ij，max (t) is the maximum amplitude of the current allowed to flow through the branch ij at the time period t; G ij is the conductance of the branch ij, B ij is the susceptance of the branch ij; V i (t), V j (t) are the voltage values of the node i and the node j at the time period t, respectively; θ ij is the phase difference between the nodes ij.

[0247] In a possible implementation, the processing module 302 is further configured to confirm the reward mechanism of the environment by:

[0248] Node voltage reward:

[0249]

[0250] wherein A is a first preset value, U I,min is the voltage lower limit UI，max is a voltage upper limit, U I is a voltage value of node i at time t.

[0251] Switching transformation reward:

[0252] r2 = -B + B * p success

[0253] In the formula, B is a second preset value, and p success is 1 if the power flow calculation is met, and p success is 0 if the power flow calculation is not met.

[0254] Embodiment 7:

[0255] Figure 4 An electronic device structure schematic diagram provided by the embodiment of the application, on the basis of the above embodiments, the embodiment of the application further provides an electronic device, as shown in the figure, comprising: a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, the memory 403 complete mutual communication through the communication bus 404; Figure 4

[0256] The memory 403 stores a computer program, and when the program is executed by the processor 401, the processor 401 executes the following steps:

[0257] Obtain power distribution network state data of each node in the region; wherein the power distribution network state data includes: voltage, line current, load power, photovoltaic power generation power and switch state;

[0258] Input the power distribution network state data of each node into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy includes a target on-off state of each switch;

[0259] Control the switch agent to adjust the state of each switch based on the target strategy.

[0260] In a possible implementation, the state determination model is a Q network.

[0261] In a possible implementation, the state determination model is trained by the following method:

[0262] Randomly sample a target sample from a replay buffer;

[0263] Calculate a target value based on a target network and a next state in the target sample;

[0264] ​The calculated target value is used to update the parameters of the original state determination model. The network parameters are optimized through the backpropagation algorithm so that the predicted value of the state determination model for a given state and action approaches the target value.

[0265] Based on the updated state determination model, for the sampled experience, the difference between the target value and the predicted value output by the original state determination model is calculated, and the target loss value is determined accordingly.

[0266] The original state determination model is trained based on the target loss value.

[0267] In one possible implementation, the target value is calculated based on the target network and the next state in the target sample, including:

[0268] The target value is calculated using the following formula:

[0269] y = r i +γnax Q target (s t+1 ,a t+1 )

[0270] Where, r i Let Q be the reward value in the target sample, γ be the discount factor, and Q be the reward value. target For the target network, s t+1 For the next state in the target sample, a t+1 For state s t+1 Time Q target (s t+1 ,a t+1 The biggest action.

[0271] In one possible implementation, the samples in the playback buffer are constructed in the following manner:

[0272] In the initial state, an ε-greedy strategy is used to select actions;

[0273] Perform the action in a pre-built environment and obtain the instantaneous reward value returned when the environment transitions from the current initial state to the next state;

[0274] Store the quadruple containing the initial state, the action performed, the returned reward value, and the observed next state as a sample in the replay buffer.

[0275] In one possible implementation, the objective function of the environment is constructed in the following manner:

[0276] The objective function is determined based on the network loss, load balancing, distributed photovoltaic absorption level, number of switching control operations, and their respective weights within the distribution network area connected to distributed photovoltaic power.

[0277] In a possible implementation, the distributed photovoltaic power distribution network area network loss is confirmed by the following method:

[0278]

[0279] In the formula, f1 is the sum of branch active loss throughout the day, and ij is the branch connected by node i and node j; X ij is the open state of node i to node j, 1 represents closing, and 0 represents opening; N is the set of all lines of the power distribution network; T is the total number of time periods; r ij is the resistance of node i to node j; t is the time period mark; I ij,t is the current of node i to node j in the tth time period; P ij and Q ij are active and reactive power of node i to node j, respectively; is the voltage of the end node of the branch ij;

[0280] The balanced load is confirmed by the following method:

[0281]

[0282]

[0283] In the formula, N is the set of all lines of the power distribution network; L ij is the load rate of node i to node j; P ij and Q ij are active and reactive power of node i to node j, respectively; is the average load rate of the line;

[0284] The distributed photovoltaic consumption level is confirmed by the following method:

[0285]

[0286] In the formula, n pv is the total number of distributed photovoltaics; T is the total time period; is the predicted value of the active power of the distributed photovoltaic on node i in the tth time period, is the actual output value of the active power of the distributed photovoltaic on node i in the tth time period;

[0287] The switch control times are confirmed by the following method:

[0288]

[0289] In the formula, ij represents the switch ij connected by node i and node j, N b is the total number of branches; H ij(t) the on-off state of switch ij at time t, H ij (t-1) the on-off state of switch ij at time t-1, the on-off state is usually represented by binary 1 and 0, 1 represents closed, and 0 represents open.

[0290] In a possible implementation, the constraint condition satisfied by the state space of the environment is confirmed by the following manner:

[0291] Power balance constraint:

[0292]

[0293] Wherein, u i and u j respectively represent the node voltage of node i and node j; i ij represents the current of branch i-j with node i as the starting point and node j as the ending point; P ij and Q ij respectively represent the active and reactive power flowing through branch i-j; r ij and x ij respectively represent the resistance and reactance of branch i-j; P j and Q j respectively represent the net active and reactive load injected into node j; jk represents the branch set connected to node j with node j as the starting point and node k as the ending point; P jk and Q jk respectively represent the active and reactive power flowing through branch j-k;

[0294] Distributed photovoltaic power generation constraint:

[0295]

[0296] Wherein, P pv,n and Q pv,n respectively represent the distributed photovoltaic active and reactive power output of photovoltaic user n; represents the maximum value of the distributed photovoltaic active power output of user n; and respectively represent the upper and lower limits of the distributed photovoltaic reactive power output of user n;

[0297] Node voltage constraint:

[0298] V min ≤ V i ≤ V max

[0299] In the formula, V min and V max respectively represent the minimum value and the maximum value of the allowable operation of the node voltage, and V i is the voltage value of node i.

[0300] Transformer capacity constraint:

[0301] S up ≤ aS rated

[0302] S down ≤ aS rated

[0303] wherein S rated is the rated capacity of the transformer, S up is the apparent power sent to the transformer; S down is the apparent power sent from the transformer, and a is a preset proportion value;

[0304] Branch current constraint:

[0305]

[0306] wherein I ij (t) is the amplitude of the current flowing through branch ij at time t, I ij，max (t) is the maximum amplitude of the current allowed to flow through branch ij at time t; G ij is the conductance of branch ij, B ij is the susceptance of branch ij; V i (t), V j (t) are the voltage values of node i and node j at time t, respectively; θ ij is the phase difference between nodes ij.

[0307] In one possible implementation, the reward mechanism of the environment is confirmed by the following ways:

[0308] Node voltage reward:

[0309]

[0310] wherein A is a first preset value, U I,min is the lower limit of voltage U I，max is the upper limit of voltage, U I is the voltage value of node i at time t.

[0311] Switch transformation reward:

[0312] r2 = -B + B * p success

[0313] wherein B is a second preset value, and p success is 1 if the network switch transformation action is executed and the power flow calculation is satisfied, and p success is 0 if the network switch transformation action is executed and the power flow calculation is not satisfied.

[0314] The communication bus mentioned by the server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0315] The communication interface is used for communication between the electronic device and other devices.

[0316] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0317] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0318] Embodiment 8:

[0319] On the basis of the above-mentioned embodiments, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program executable by an electronic device, when the program runs on the electronic device, the electronic device executes the following steps:

[0320] Obtaining power distribution network state data of each node in the region; wherein the power distribution network state data includes voltage, line current, load power, photovoltaic power generation power and switch state;

[0321] Inputting the power distribution network state data of each node into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy includes a target on-off state of each switch;

[0322] Controlling a switch agent to adjust the state of each switch based on the target strategy.

[0323] In a possible implementation, the state determination model is a Q network.

[0324] In one possible implementation, the state determination model is trained in the following manner:

[0325] Randomly sample the target sample from the playback buffer;

[0326] Calculate the target value based on the target network and the next state in the target sample;

[0327] The calculated target value is used to update the parameters of the original state determination model. The network parameters are optimized through the backpropagation algorithm so that the predicted value of the state determination model for a given state and action approaches the target value.

[0328] Based on the updated state determination model, for the sampled experience, the difference between the target value and the predicted value output by the original state determination model is calculated, and the target loss value is determined accordingly.

[0329] The original state determination model is trained based on the target loss value.

[0330] In one possible implementation, the target value is calculated based on the target network and the next state in the target sample, including:

[0331] The target value is calculated using the following formula:

[0332] y = r i +ymax Q target (s t+1 ,a t+1 )

[0333] Where, r i Let Q be the reward value in the target sample, γ be the discount factor, and Q be the reward value. target For the target network, s t+1 For the next state in the target sample, a t+1 For state s t+1 Time Q target (s t+1 ,a t+1 The biggest action.

[0334] In one possible implementation, the samples in the playback buffer are constructed in the following manner:

[0335] In the initial state, an ε-greedy strategy is used to select actions;

[0336] Perform the action in a pre-built environment and obtain the instantaneous reward value returned when the environment transitions from the current initial state to the next state;

[0337] Store the quadruple containing the initial state, the action performed, the returned reward value, and the observed next state as a sample in the replay buffer.

[0338] In a possible implementation, the objective function of the environment is constructed in the following manner:

[0339] According to the network loss in the distribution network region accessing distributed photovoltaic, the balanced load, the distributed photovoltaic consumption level, the number of switch control, and the corresponding weights, respectively, the objective function is determined.

[0340] In a possible implementation, the network loss in the distribution network region of the distributed photovoltaic is determined in the following manner:

[0341]

[0342] In the formula, f1 is the sum of branch active loss all day, ij is the branch connected by node i and node j; X ij is the open state of node i to node j, 1 represents closing, and 0 represents opening; N is the set of all lines of the distribution network; T is the total number of time periods; r ij is the resistance of node i to node j; t is the time period mark; I ij,t is the current of node i to node j in the tth time period; P ij and Q ij are the active and reactive power of node i to node j, respectively; is the voltage of the end node of the branch ij;

[0343] The balanced load is determined in the following manner:

[0344]

[0345] In the formula, N is the set of all lines of the distribution network; L ij is the load rate of node i to node j; P ij and Q ij are the active and reactive power of node i to node j, respectively; is the average load rate of the line;

[0346] The distributed photovoltaic consumption level is determined in the following manner:

[0347]

[0348] In the formula, n pv is the total number of distributed photovoltaics; T is the total time period; is the predicted value of the active power of the distributed photovoltaic on node i in the tth time period, is the actual output value of the active power of the distributed photovoltaic on node i in the tth time period;

[0349] The number of switch control is determined in the following manner:

[0350]

[0351] where ij represents the switch ij connecting node i and node j, N b is the total number of branches; H ij (t) is the on-off state of switch ij at time t, H ij (t-1) is the on-off state of switch ij at time t-1, which is usually represented by binary 1 and 0, 1 representing closed and 0 representing open.

[0352] In a possible implementation, the constraint condition satisfied by the state space of the environment is confirmed by the following way:

[0353] Power balance constraint:

[0354]

[0355]

[0356] where u i and u j represent the node voltage of node i and node j, respectively; i ij represents the current of branch i-j with node i as the starting point and node j as the ending point; P ij and Q ij represent the active and reactive power flowing through branch i-j, respectively; r ij and x ij represent the resistance and reactance of branch i-j, respectively; P j and Q j represent the net active and reactive load injected into node j; jk represents the set of branches connected to node j with node j as the starting point and node k as the ending point; P jk and Q jk represent the active and reactive power flowing through branch j-k, respectively;

[0357] Distributed photovoltaic power generation constraint:

[0358]

[0359] where P pv,n and Q pv,n represent the active and reactive output of distributed photovoltaic user n, respectively; represents the maximum active output of distributed photovoltaic user n; and represent the upper and lower limits of the reactive output of distributed photovoltaic user n, respectively;

[0360] Node voltage constraint:

[0361] Vmin ≤V i ≤V max

[0362] wherein V min and V max are the minimum and maximum values of the allowable operation of the node voltage, respectively, V i is the voltage value of the node i.

[0363] Transformer capacity constraint:

[0364] S up ≤aS rated

[0365] S down ≤aS rated

[0366] wherein S rated is the rated capacity of the transformer, S up is the apparent power sent on the transformer; S down is the apparent power sent on the transformer, and a is a preset proportion value.

[0367] Branch current constraint:

[0368]

[0369]

[0370] wherein I ij (t) is the amplitude of the current flowing on the branch ij at the time period t, I ij，max (t) is the maximum amplitude of the current allowed to flow on the branch ij at the time period t; G ij is the conductance of the branch ij, B ij is the susceptance of the branch ij; V i (t), V j (t) are the voltage values of the node i and the node j at the time period t, respectively; θ ij is the phase difference between the nodes ij.

[0371] In a possible implementation, the reward mechanism of the environment is confirmed by the following ways:

[0372] Node voltage reward:

[0373]

[0374] wherein A is a first preset value, U I,min is the lower limit of the voltage U I，max is the upper limit of the voltage, I I is the voltage value of the node i at the time t.

[0375] Switching transformation reward:

[0376] r2 = -B + B*p success

[0377] wherein B is a second preset value, p is 1 if the power flow calculation is satisfied when the network switching transformation action is performed each time, and p is 0 if the power flow calculation is not satisfied. success success

[0378] From the technical common sense, the present application can be realized by other embodiments without departing from the spirit or essential characteristics. Therefore, the above disclosed embodiments are only examples, and are not the only ones. All changes within the scope of the present application or within the scope equivalent to the present application are included in the present application.

[0379] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0380] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in the flowcharts and / or block diagrams Figure 1 The functions specified in the flowcharts and / or block diagrams

[0381] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in the flowcharts and / or block diagrams Figure 1 The functions specified in the flowcharts and / or block diagrams

[0382] ​​These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide the function of implementing the processes specified in the flowchart Figure 1 one flow or multiple flows and / or the functions specified in the block Figure 1 one flow or multiple flows and / or the functions specified in the block

[0383] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the above embodiments of the present application are described in detail, those skilled in the art should understand: the specific embodiments of the present application can be modified or replaced by the same, without departing from the spirit and scope of the present application of any modification or equivalent replacement, it should be covered within the scope of protection of the claims of the present application.

Claims

1. A method for determining the on-off state of a tie switch of a power distribution line, characterized by, The method comprises: obtaining power distribution network state data of each node in the region; wherein the power distribution network state data comprises voltage, line current, load power, photovoltaic power generation power and switch state; inputting the power distribution network state data of each node into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy comprises a target on-off state of each switch; controlling a switch agent to adjust the state of each switch based on the target strategy; the state determination model is obtained by training in the following manner: randomly sampling a target sample from a replay buffer; calculating a target value based on a target network and a next state in the target sample; updating parameters of an original state determination model using the calculated target value, optimizing network parameters through a back propagation algorithm, so that the prediction value of the state determination model for a given state and action tends to approach the target value; based on the updated state determination model, calculating the difference between the target value and the prediction value output by the original state determination model for the sampled experience, and determining a target loss value based on the difference; training the original state determination model based on the target loss value; the samples in the replay buffer are constructed in the following manner: selecting an action in an initial state with an ε-greedy strategy; executing the action in a pre-constructed environment to obtain an immediate reward value returned by the environment when the environment transitions from the current initial state to a next state; storing a four-tuple containing the initial state, the executed action, the returned reward value and the observed next state as a sample in the replay buffer; the constraint conditions satisfied by the state space of the environment are confirmed in the following manner: power balance constraint: in, and Let i and j represent the node voltages, respectively. This represents the current in branch ij, which starts at node i and ends at node j. and These represent the active and reactive power flowing through branch ij, respectively; and Let represent the resistance and reactance of branch ij, respectively; and These represent the net active and reactive loads injected into node j, respectively; jk represents the set of branches connected to node j, starting from node j and ending at node k. and These represent the active and reactive power flowing through branch jk, respectively; distributed photovoltaic power generation constraint: wherein, , represent the distributed PV active and reactive power output of the photovoltaic user n, respectively; denotes the maximum of the distributed PV active power output of the user n; and denote the upper and lower limit of the distributed PV reactive power output of the user n, respectively; node voltage constraint: wherein and are the minimum and maximum values of the allowed operation of the node voltage, respectively, is the voltage value of the node i; transformer capacity constraint: In the formula, is the rated capacity of the transformer, is the apparent power sent on the transformer; is the apparent power sent on the transformer, a is a preset proportion value; branch current constraint: wherein is the amplitude of the current flowing through branch ij at time t, is the maximum amplitude of the current allowed to flow through branch ij at time t; is the conductance of branch ij, is the susceptance of branch ij; , are the voltage values of node i and node j, respectively, at time t; is the phase difference between nodes ij.

2. The method of claim 1, wherein, the state determination model is a Q network.

3. The method of claim 1, wherein, calculating a target value based on a target network and a next state in the target sample comprises: calculating the target value using the following formula: wherein, is a reward value in the target sample, is a discount factor, is a target network, is a next state in the target sample, is an action that maximizes a value of when the state is 4. The method of claim 1, wherein, the objective function of the environment is constructed in the following manner: determining the objective function according to the network loss, balanced load, distributed photovoltaic consumption level, switch control times and respective weights in the distributed photovoltaic power distribution network region.

5. The method of claim 4, wherein, the network loss in the distributed photovoltaic power distribution network region is confirmed in the following manner: wherein is the sum of all branch active power losses over the whole day, and ij is the branch connecting node i and node j; is the open state of the branch from node i to node j, 1 means closed and 0 means open; and N is the set of all lines of the distribution network. T is the total number of time periods; Rij is the resistance from node i to node j; t is the time period index; Iij(t) is the current from node i to node j at time period t; and Pij(t) and Qij(t) are the real and reactive power from node i to node j, respectively; Vij is the voltage at the end nodes of branch ij. the balanced load is confirmed in the following manner: where N is the set of all lines of the power distribution network; is the load rate from node i to node j; and are the active and reactive power from node i to node j, respectively; is the average load rate of the line. the distributed photovoltaic consumption level is confirmed in the following manner: In the formula, is the total number of distributed photovoltaics; T is the total time period; is the predicted value of the active power of the distributed photovoltaics at node i in the t time period, is the actual output value of the active power of the distributed photovoltaics at node i in the t time period; the switch control times are confirmed in the following manner: where ij represents the switch ij connected between node i and node j, is the total number of branches; is the on-off state of switch ij at time t, is the on-off state of switch ij at time t-1, which is usually represented by binary 1 and 0, 1 representing closed and 0 representing open.

6. The method of claim 1, wherein, the reward mechanism of the environment is confirmed in the following manner: node voltage reward: Wherein, A is a first preset value, is a lower limit of voltage is an upper limit of voltage, is a voltage value of node i at time t; switch transformation reward: In the formula, is a second preset value, and if the power flow calculation is met when the network switch transformation action is executed each time, is 1, if the power flow calculation is not met, is 0.

7. A power line circuit tie switch on-off state determination product, characterized by, the product comprises: an acquisition module configured to obtain power distribution network state data of each node in the region; wherein the power distribution network state data comprises voltage, line current, load power, photovoltaic power generation power and switch state; a processing module configured to input the power distribution network state data of each node into a state determination model to obtain a target strategy output by the state determination model, wherein the target strategy comprises a target on-off state of each switch; a control module configured to control a switch agent to adjust the state of each switch based on the target strategy; the state determination model is obtained by training in the following manner: randomly sampling a target sample from the replay buffer; calculating a target value based on a target network, a next state in the target sample; updating parameters of the original state determination model using the calculated target value, optimizing network parameters through a back propagation algorithm, so that the prediction value of the state determination model for a given state and action tends to the target value; based on the updated state determination model, calculating the difference between the target value and the prediction value output by the original state determination model for the sampled experience, and determining the target loss value accordingly; training the original state determination model based on the target loss value; the samples in the replay buffer are constructed in the following way: selecting an action in an initial state with an ε-greedy strategy; executing the action in the pre-constructed environment to obtain the immediate reward value returned by the environment when the environment transitions from the current initial state to the next state; storing the four-tuple containing the initial state, the executed action, the returned reward value and the observed next state as a sample in the replay buffer; the constraint conditions satisfied by the state space of the environment are confirmed in the following way: power balance constraint: where, Vi and Vj represent the node voltage of node i and node j, respectively; Vi and Vj represent the node voltage of node i and node j, respectively; Iij represents the current of branch i-j, which starts at node i and ends at node j; Pi and Qj represent the active and reactive power injected into node j, respectively; Pi and Qj represent the active and reactive power injected into node j, respectively; Rij and Xij represent the resistance and reactance of branch i-j, respectively; Rij and Xij represent the resistance and reactance of branch i-j, respectively; Pi and Qj represent the active and reactive power injected into node j, respectively; Pi and Qj represent the active and reactive power injected into node j, respectively; Pi and Qj represent the active and reactive power injected into node j, respectively; Pi and Qj represent the active and reactive power injected into node j, respectively; distributed photovoltaic power generation constraint: wherein, , Pn, Qnrepresent the distributed PV active and reactive power of the photovoltaic user n, respectively; Pn,maxrepresents the maximum of the distributed PV active power of the user n; and Pn,minand Pn,maxrepresent the upper and lower limits of the distributed PV reactive power of the user n, respectively; node voltage constraint: wherein and are the minimum and maximum values of the allowed operation of the node voltage, respectively, is the voltage value of the node i; transformer capacity constraint: wherein is the rated capacity of the transformer, is the apparent power delivered to the transformer; is the apparent power delivered from the transformer, a is a preset proportion value; branch current constraint: wherein is the amplitude of the current flowing through branch ij at time t, is the maximum amplitude of the current allowed to flow through branch ij at time t; is the conductance of branch ij, is the susceptance of branch ij; , are the voltage values at node i and node j, respectively, at time t; is the phase difference between nodes ij.

Images