Improved double-agent d3qn-based method for n-1 additional risk load transfer in urban power grid

Abstract

The application belongs to the technical field of city power grid risk prevention and topology reconstruction, and is based on a city power grid N-1 new risk load transfer method improved by double intelligent agents D3QN. An optimization index with better characterization for N-1 risk is newly built, a three-stage model is used to avoid local optimum and weight balance problems caused by multi-objective, double intelligent agents are set according to the load transfer characteristics to reduce the action space dimension and solve the multi-loop network problem, a pre-action-change exploration value selection strategy is used to replace the traditional greedy strategy, and the training rate is improved from the action selection and sample quality. The method does not require prior knowledge and realizes gradual interaction with the power grid environment, can better guarantee the risk prevention and control capability than the experience decision, and can obtain a city power grid N-1 risk prevention and control strategy more in line with the engineering practice.

Description

Technical Field

[0001] This invention belongs to the field of urban power grid risk prevention and topology reconfiguration technology, and in particular relates to a method for transferring newly added risky loads in urban power grids N-1 based on an improved dual-agent D3QN. Background Technology

[0002] In urban power grids, topology design and coordination with automatic transfer switches (ATS) reduce or offset most of the adverse effects of N-1 events. However, N-1 events can disrupt this coordination, potentially leading to large-scale power outages or cascading failures due to N-1-1 risks. Because urban power grids have numerous backup lines, load transfer can be achieved by adjusting switch combinations to change the power supply path of the load. Therefore, load transfer is often used by dispatching departments as a means of controlling risk. However, load transfer plans are based on the experience of dispatching personnel, and their effectiveness depends on their experience level. The lack of practical interaction makes it difficult to comprehensively analyze and control potential risks.

[0003] Existing commonly used risk indicators are insufficient to simultaneously address specific severity representation capabilities, applicability to N-1 scenarios, and the guarantee of overall risk control effectiveness. There is an urgent need to explore risk analysis and control from the perspective of N-1 scenarios, and to establish new evaluation indicators. Currently, there are three relatively mature and widely used methods for urban power grid reconfiguration problems. One is mathematical optimization algorithms, including multi-stage optimization methods, nonlinear programming methods, and dynamic programming methods, which transform load transfer into solving a mathematical problem. Another is heuristic algorithms, such as hierarchical tree search and branch switching methods, which can utilize heuristic rules to reduce the search space and problem complexity, but have a large search range, are difficult to search globally, and are difficult to adapt to distribution network reconfiguration decisions with mixed variables and numerous nonlinear constraints. A third is metaheuristic algorithms, such as particle swarm optimization, genetic algorithms, and ant colony optimization. These algorithms are robust, but cannot guarantee the optimality of the solution, and have long solution times.

[0004] Therefore, how to quickly obtain load transfer strategies that take into account interaction, provide information support to the power grid sector, and reduce the potential risks of N-1 addition, the extent of power outage losses and the possibility of cascading failures after N-1-1, has become a problem of concern and urgent need to be solved by urban power grids. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a method for transferring newly added risky loads in urban power grid N-1 based on an improved dual-agent D3QN. For the newly added risks in urban power grid N-1, a new risk optimization index is created to represent the risks. The method is optimized in three stages. The improved dual-agent D3QN method is obtained by improving deep reinforcement learning to address the characteristics of the load transfer problem, and the risk prevention and control transfer strategy is solved.

[0006] The method for transferring newly added risky loads (N-1) to urban power grids based on the improved dual-agent D3QN includes the following steps, which are performed sequentially.

[0007] Step 1: Analyze the new risks N-1 in the power grid and introduce automatic transfer switching devices to establish load transfer optimization indicators for the prevention and control of new risks;

[0008] Step 2: Based on the priority of the optimization indicators established in Step 1, the load transfer optimization is divided into three stages, and the three-stage model is converted into a mathematical model of deep reinforcement learning.

[0009] Step 3: Based on the deep reinforcement learning D3QN algorithm model, set up dual agents and dual experience pools to analyze the closure and opening of load transfer, reduce the dimension of action space, and improve the utilization rate of excellent samples.

[0010] Step 4: Based on the D3QN algorithm, set a pre-action - changing exploration value selection strategy to replace the greedy strategy in the original D3QN algorithm. Enable immediate reward knowledge to add a new selection strategy and assign two exploration values ​​that change with the training process for each strategy. Accumulate high-quality samples, improve the training rate, and output risk control and transfer strategy.

[0011] The load transfer optimization index for step one is:

[0012] f1 = min(N) R )

[0013] Where, N R The number of single-supply substations in the power grid with only one backup power supply path;

[0014] f2=min(P R )

[0015] Among them, P R The load carried by a risky load busbar in a power grid with only one backup power supply path;

[0016] f3=min(P lmax )

[0017] Among them, P lmax This represents the highest branch load rate in the power grid. When the highest load rate is below 70%, this value is 0.

[0018] The division of the model into two and three stages in step two is based on the following criteria:

[0019] In the first phase, ensure that the substation has two power supply paths, including a backup power supply path, to minimize the risk of the substation being supplied by a single power source. Substations in the risk of being supplied by a single power source will transfer part of the load to the power supply path of the superior or same-level substation through the split operation mode, or use the backup automatic transfer to form two power supply paths for the substation, thus getting rid of the risk of the substation being supplied by a single power source.

[0020] In the second phase, the number of substations with single power supply will be maintained, and the maximum load will be transferred to the standby automatic transfer protection area that is in the pre-operation state to ensure two power supply paths and minimize the risk of load loss.

[0021] The third stage maintains the optimization results of the first two stages and transfers the load downstream of high-load-rate branches to low-load-rate branches, balancing the load rate of each branch to ensure stable power supply from the power grid.

[0022] The two agents established in step three learn the closing and opening operations in load transfer, and only one agent outputs the actual action for each action. The reward function for the two agents is:

[0023]

[0024] In the formula, R1, R2, R3, and R4 represent the agent's reward components, and P1, P2, P3, and P4 represent the agent's penalty components.

[0025] The reward portion is as follows:

[0026] With the goal of reducing risky loads and the number of single-supply substations, the rewards that the agent can obtain in the first and second phases are established:

[0027]

[0028] R2 = P trans

[0029] In the formula, N del , and P loadi P represents the cumulative reduction in the number of single-supply substations and the total active load of the i-th substation; trans This represents the cumulative amount of risky loads that have been transformed into safe loads.

[0030] After the objectives of the first and second phases are achieved, the third phase of optimization will begin, focusing on reducing the line load rate.

[0031]

[0032] In the formula, r represents the reward for the current state R1 and R2, P3 represents the penalty function related to the radial movement of the current state, and P m0 and P m1 This indicates the load rate of the branch with the highest load rate before and after the current action;

[0033] The load transfer process can cause a ring network in the distribution network. By setting a bonus R4, the radial distribution structure can be restored after each load transfer.

[0034]

[0035] In the formula, g represents the distribution network topology state after the current action, and G represents the combination of all radial distribution network structures in the power grid;

[0036] The punishment part is as follows:

[0037] Establish a penalty for situations where the agent's actions do not meet the voltage amplitude constraints:

[0038]

[0039] In the formula, U i U represents the per-unit voltage value of the i-th node, while U imin and U imax This indicates the lower and upper limits of the voltage per unit value for this node;

[0040] Penalties are set based on the transmission power limits of each branch voltage level and equipment model:

[0041]

[0042] In the formula, P i Let P represent the power transmitted on the i-th branch, and Pi represent the power transmitted on the i-th branch. imax This indicates the maximum power that can be transmitted on this branch;

[0043] Set a penalty for the switch action based on the switch operation time:

[0044]

[0045] In the formula, N o This represents the current number of switch actions, when N o When the value is 8, this penalty offsets the reward for reducing risk.

[0046] Based on the risks associated with the transformer and load, the following penalties are set:

[0047]

[0048] In the formula, N del0 N del1 P represents the cumulative reduction in the number of single-supply substations before and after the action. trans0 ,P trans1 This represents the cumulative risk load before and after the action.

[0049] In step three, the dual experience pools store sample data in two separate experience pools, ranked from highest to lowest, based on the Q-value and reward value. The Q-function and Q-value represent a value function and a value under a specific condition in deep reinforcement learning, which can be expressed by two neural networks in D3QN as follows:

[0050]

[0051] In the formula, s i Let a represent the i-th state. i Let A represent the i-th action, θ represent the parameters of the neural network, and A a For the set of all actions, center the action vectors.

[0052] The strategy for selecting the pre-action / change exploration value in step four is as follows:

[0053] The instant reward value is used as a reference for the next action selection. Before each action selection, if the Q value is not suitable, a pre-action calculation flow is performed to obtain the state and instant reward set at the next moment. A selection mechanism for the action with the maximum instant reward is added to enable the agent to accumulate high-quality samples in the early exploration process.

[0054] A greedy strategy is adopted to ensure the convergence of the algorithm. Two exploration values ​​ε are set, and exploration is carried out in combination with three strategies. The exploration values ​​ε1 and ε2 are set, and random numbers in [0,1] are selected. Different action selection strategies are adopted depending on whether the values ​​fall in different regions: ① the action with the maximum immediate reward; ② the greedy strategy action; ③ the random action; the formulas for the changes of ε1 and ε2 are as follows:

[0055]

[0056]

[0057] In the formula, m is the current number of episodes, M is the maximum number of episodes to train, and ε 1max =0.95, ε 1min =0.55, ε 2max =0.45, ε 2min =0.05.

[0058] Through the above design scheme, the present invention can bring the following beneficial effects:

[0059] This invention presents an improved dual-agent D3QN method for transferring newly added risky loads (N-1) in urban power grids. It introduces a new optimization index that better represents N-1 risks, utilizes a three-stage model to avoid local optima and weight balancing problems caused by multi-objectives, and reduces the action space dimension and solves the multi-loop network problem by designing a dual-agent system specifically for load transfer characteristics. A pre-action-change exploration value selection strategy replaces the traditional greedy strategy, improving training speed through action selection and sample quality. This method requires no prior knowledge and achieves gradual interaction with the power grid environment, ensuring better risk control capabilities than experience-based decision-making, thus yielding an urban power grid N-1 risk control strategy that better aligns with engineering realities. Attached Figure Description

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0061] Figure 1 This is a schematic diagram of the process for the transfer of newly added risky loads in urban power grid N-1 based on the improved dual-agent D3QN method of the present invention.

[0062] Figure 2 This is a schematic diagram of the dual-agent D3QN training process for the urban power grid N-1 newly added risky load transfer method based on the improved dual-agent D3QN of the present invention.

[0063] Figure 3 This is a schematic diagram illustrating the change of the exploration value ε in the greedy strategy of the urban power grid N-1 newly added risky load transfer method based on the improved dual-agent D3QN of this invention.

[0064] Figure 4 This is a schematic diagram of the load transfer method based on the dual-agent D3QN for the N-1 newly added risky load in the urban power grid, which is based on the improved dual-agent D3QN of the present invention.

[0065] Figure 5 This is a schematic diagram of a local power grid topology in an embodiment of the urban power grid N-1 newly added risk load transfer method based on the improved dual-agent D3QN of the present invention.

[0066] Figure 6 This diagram illustrates the comparison of average training rewards using different algorithms in an embodiment of the urban power grid N-1 newly added risky load transfer method based on the improved dual-agent D3QN of the present invention.

[0067] Figure 7 This is a schematic diagram of the load transfer switch switching position in an embodiment of the urban power grid N-1 newly added risk load transfer method based on the improved dual-agent D3QN of the present invention.

[0068] Figure 8 This diagram illustrates the load rate comparison of different load transfer strategies in an embodiment of the urban power grid N-1 newly added risky load transfer method based on the improved dual-agent D3QN of the present invention.

[0069] Figure 9 This is a schematic diagram comparing the third stage and multiple objectives of the embodiment of the urban power grid N-1 newly added risk load transfer method based on the improved dual-agent D3QN of the present invention. Detailed Implementation

[0070] A method for transferring newly added risky loads to urban power grids based on the improved dual-agent D3QN approach, such as... Figure 1 As shown, it includes the following steps:

[0071] Step 1: Analyze the new risks in the power grid N-1, and establish load transfer optimization indicators for the prevention and control of new potential risks, taking into account the automatic transfer device;

[0072] Step 2: Divide the supply optimization into three stages according to the priority of optimization indicators, and convert the three-stage model into a mathematical model of deep reinforcement learning;

[0073] Step 3: Based on the D3QN method, set up two agents and two experience pools to analyze the closing and opening actions of load transfer, while reducing the dimension of action space to avoid multi-loop network problems and reduce the discard of excellent samples.

[0074] Step 4: Based on the D3QN method, set up a pre-action-changing exploration value selection strategy to replace the traditional greedy strategy. Enable immediate reward knowledge to add new selection strategies and assign two exploration values ​​that change with the training process for each strategy. This can quickly accumulate high-quality samples while ensuring convergence and high training rate, and finally output a risk control and transfer strategy.

[0075] Specifically, step one involves conducting an N-1 risk analysis and establishing optimization indicators for the automatic start-up device:

[0076] The topology design and coordination of automatic transfer switches (ATS) in urban power grids reduce or offset most of the adverse effects of N-1 events. However, N-1 events will disrupt this coordination, posing a potential risk of large-scale power outages or cascading failures caused by N-1-1. After an N-1 event occurs, to prevent actual power outage losses, ATS near the N-1 will automatically activate. However, this simultaneously results in substations and load buses lacking ATS protection. These substations and load buses, which previously had two power supply paths (including backup ATS), now only have one power supply path (currently supplying power) after N-1. Therefore, the best way to mitigate the N-1 risk is to ensure that these substations and load buses have two power supply paths (including backup ATS) again through other ATS protection. Considering the occurrence of N-1 and the previous power transfer leading to overload on some branches, the following optimization indicators are established:

[0077] f1 = min(N) R (1)

[0078] Where N R The number of single-supply substations in the power grid with only one power supply path (including backup).

[0079] f2=min(P R (2)

[0080] Where P R The load carried by a risky load bus in the power grid with only one power supply path (including backup).

[0081] f3=min(P lmax (3)

[0082] Where P lmax This represents the highest branch load rate in the power grid. If the highest load rate is below 70%, this value is 0.

[0083] Specifically, its three-stage division and three-stage load transfer model are transformed into a mathematical model for deep reinforcement learning:

[0084] Considering the strict control of power grid level 3 events in urban power grids (when any power outage occurs in urban power grid, the number of substations with a total voltage loss of 110kV or above is less than one, and the load loss is less than 50MW), load transfer will be carried out in three stages according to the risk optimization priority.

[0085] In the first phase, efforts should be made to ensure that substations have two power supply paths (including backups), thus minimizing the risk of substations relying solely on a single power supply. Substations facing this risk can transfer part of their load to the power supply paths of higher-level or same-level substations through split operation, or utilize automatic transfer switches to create two power supply paths, thereby eliminating the risk of substations relying solely on a single power supply.

[0086] In the second phase, without increasing the number of substations with single power supply, the load will be transferred as much as possible to the backup automatic transfer protection area that is in the pre-operation state, so as to ensure that there are two power supply paths (including backup), that is, to minimize the risk of load loss.

[0087] In the third stage, without compromising the optimization results of the first two stages, the load downstream of high-load-rate branches should be transferred to low-load-rate branches as much as possible. Appropriately balancing the load rate of each branch is conducive to the stable power supply of the power grid and avoids cascading failures.

[0088] Unlike the simultaneous optimization of multi-objective models, three-stage optimization avoids the problem of balancing the weight setting by avoiding excessive load factor during load risk optimization which would result in minimal substation risk optimization effect.

[0089] The state space should be considered in relation to the factors influencing the quality of topology reconfiguration decisions. For the problem of load transfer using topology reconfiguration, the most fundamental variable is the state of each switch in the power grid. Let the states of the switches of lines not exceeding N-1 voltage levels, bus couplers, and transformers above a certain voltage level be considered as quantities in the state space:

[0090] S = [X, N] R ,P R (4)

[0091] In the formula: X is the set of state variables of the non-faulty branch switch, N R P represents the number of risky substations. R This refers to the risk load.

[0092] The specific operation in the load transfer strategy consists of a series of switching actions. Considering that the output load transfer strategy takes into account the sequence of switching actions, each action during the learning process should only involve switching one switch. The switching operations of the N-1 line, transformer, and bus tie switch are selected as the action space.

[0093]

[0094] In the formula: a is a 0-1 variable that determines whether to switch a certain switch in the power grid. Only one variable in the vector is 1, which means that the switch is switched. When a0 = 1, it means that no action is taken. m is the number of switches in the open state and n is the number of switches in the closed state.

[0095] Each action has exactly one variable set to 1, ensuring that each action ultimately selects to switch only one switch. If voltage exceedances, line overloads, or islanding occur during the agent's learning process, the agent can use its learning to avoid these actions in subsequent power transfers and discover the optimal power transfer strategy.

[0096] The agent needs to determine the merits of a given state based on the reward function. For the load transfer strategy mentioned in this paper, the reward function needs to consider the risk and safety load amounts, the number of substations supplying power, changes in line load rates, and relevant grid constraints, such as node voltages, branch transmission limits, and whether the distribution network topology is radial. The total number of switching actions during a single load transfer is also used as one of the criteria for evaluating the merits of the transfer strategy. The two agents will each have different reward functions set; the content involved in these reward functions is described below.

[0097] 1) Rewards

[0098] The primary goal of load shifting is to mitigate the risk of the grid recurring N-1-1 after N-1 and improve reliability. Therefore, the most important thing is to reduce the amount of risky loads and the number of single-supply substations. The following are the rewards that the agent can obtain in the first and second phases, respectively.

[0099]

[0100] R2 = P trans (7)

[0101] In the formula N del , and P loadi P represents the cumulative reduction in the number of single-supply substations and the total active load of the i-th substation; trans This represents the cumulative amount of risky loads that have been transformed into safe loads.

[0102] When the risk load and the number of single-supply substations cannot be reduced further, the third stage of optimization is entered, which involves reducing the line load rate within permissible limits:

[0103]

[0104] In the formula, r represents the reward for the current state R1 and R2, P3 represents the penalty function related to the radial movement of the current state, and P m0 and P m1 This indicates the load rate of the branch with the highest load rate before and after the current action.

[0105] Load transfer inevitably leads to ring networks in the distribution network. To ensure the restoration of the radial distribution structure after each load transfer, an incentive is established:

[0106]

[0107] In the formula, g represents the distribution network topology state after the current action, and G represents the combination of all radial distribution network structures in the power grid.

[0108] 2) Punishment section

[0109] Maintaining the normal node voltage is one of the basic requirements for ensuring the normal operation of the power grid. The voltage amplitude should be kept within the upper and lower limits. Penalties are given for situations where the agent's actions do not meet this constraint:

[0110]

[0111] In the formula U i U represents the per-unit voltage value of the i-th node, while U imin and U imax This indicates the lower and upper limits of the voltage per unit value for this node.

[0112] When the transmission power of a branch exceeds its limit, the probability of equipment failure increases significantly. Different voltage levels and different equipment models have corresponding transmission power limits, and penalties are set accordingly.

[0113]

[0114] In the formula P i Let P represent the power transmitted on the i-th branch, and Pi represent the power transmitted on the i-th branch. imax This indicates the upper limit of power transmitted on this branch.

[0115] Load transfer should be completed with as few operations as possible to reduce maintenance costs and the possibility of errors, save operation time, and prevent excessive changes in the distribution network structure that would increase the difficulty of restoring the original operation mode after fault elimination. The following switch operation costs should be considered:

[0116]

[0117] In the formula, N o This represents the current number of switch actions, when No When the number of switches is 8, this penalty will offset the risk reduction reward. The dispatching department believes that the number of such switching actions should be within eight.

[0118] Certain load transfer actions may cause the risks to transformers and loads to increase rather than decrease, therefore the following penalties are added:

[0119]

[0120] In the formula N del0 N del1 P represents the cumulative reduction in the number of single-supply substations before and after the action. trans0 ,P trans1 This represents the cumulative risk load before and after the action.

[0121] Specifically, urban power grid load transfer involves a large number of binary variables representing switch states, resulting in a vast action space that makes existing optimization methods very difficult to solve. In this load transfer problem, the purpose of closing a switch is to connect branches and reduce the load risk of substations and busbars by utilizing automatic transfer switches; the purpose of opening a switch is to break loops and form a radial network. During the learning process, the agent often closes switches continuously, forming multiple loop networks, making it difficult to recover the radial network and hindering training effectiveness. Based on the different purposes of the two actions, two agents are set up for separate training and centralized interaction. This ensures the sequential action of closing and opening switches during load transfer while dividing the original action space into two subspaces, improving the training rate.

[0122] like Figure 2 and Figure 4 As shown, two agents with different action spaces are set up to learn the closing and opening operations in load transfer, respectively. Only one agent outputs the actual action for each operation. The state spaces are set to be identical so that the two agents can share a changing state. The reward functions for both agents are as follows:

[0123]

[0124] Due to storage limitations, the D3QN experience pool has a limited capacity. The load transfer process involves a series of switching actions, resulting in a massive state space where the agent encounters a large proportion of useless samples. To address this, samples are categorized into two groups based on Q-value and reward, and placed into two separate experience pools. When updating the network, samples from both pools are proportionally drawn as data, ensuring a sufficient number of high-quality samples are available for the agent to learn from first and preventing the discarding of excellent samples.

[0125] The specific details of setting the pre-action - change exploration value selection strategy include:

[0126] Traditional random greedy strategies select the action with the maximum Q value with a high probability and select random actions with a low probability. This requires a lot of exploration and computation and is prone to getting trapped in local optima. Even if random actions are used to escape local optima, it is difficult to affect the learning direction.

[0127] Considering that the reward given by the agent is a judgment of the action, which also partially affects the Q-value, if the immediate reward value is used as a reference for the next action selection, there will be a higher probability of finding the optimal action before the neural network is fully trained, i.e., before the Q-value is suitable.

[0128] Before each action selection, a pre-action computation flow is performed to obtain the possible next-moment states and immediate reward sets. This information is used to add a selection mechanism that chooses the action with the maximum immediate reward, enabling the agent to quickly accumulate high-quality samples in the early stages of exploration. To ensure algorithm convergence, this paper retains a greedy strategy, requiring two exploration values ​​ε to be set, and exploring using a combination of three strategies.

[0129] In summary, the set exploration values ​​ε1 and ε2 will vary depending on... Figure 3 The values ​​change, and a random number within the range [0,1] is selected. Different action selection strategies are adopted depending on where the number falls: ① the action with the maximum immediate reward; ② the greedy strategy action; ③ the random action. The formulas for the changes in ε1 and ε2 are as follows:

[0130]

[0131]

[0132] In the formula, m is the current number of episodes, M is the maximum number of episodes to train, and ε 1max =0.95, ε 1min =0.55, ε 2max =0.45, ε 2min =0.05.

[0133] Example: Taking a local power grid model of a certain region as an example, the simulation environment pypower is used; such as Figure 5 As shown,

[0134] N-1 occurs at SB220-JM2201M. In the original topology, the JM substation busbar operates separately and is simultaneously powered by both SB and DH substations. N-1 disrupts the ring structure formed by JM, SB, and DH substations, causing the JM220kV bus tie automatic transfer switch to activate. This reduces the potential power supply path for these three 220kV substations, but SB substation still maintains two potential power supply paths and is not at risk. Furthermore, JM substation is powered only by DH substation, and DH substation is powered only by SZ substation; both are in a single-supply risk state. Some of the 110kV substations they supply, such as LC, LF, LH, GC, SG substations, and the CF substation below them, are also in a single-supply risk state. Load risk exists within WJ, YG, HB, TXL, and HL substations.

[0135] The neural network of the dual-agent D3QN consists of an input layer, hidden layers, a value function layer, a dominance function layer, and an output layer. The hidden layers of the two agents have 72 and 64 neurons, and 24 and 16 neurons, respectively. The value function layers of both agents have only one neuron, and the dominance function layers have 64 and 16 neurons, respectively. The discount factor γ is 0.97, the experience replay pool capacity is 1000 samples for both agents, the number of learning samples is 200, the initial learning rate is 0.001 and will be reduced as the learning progresses to achieve better convergence. The specific setting of the exploration value ε is specified in the requirements document.

[0136] Improving the training performance of dual-agent D3QN compared to other algorithms, for example... Figure 6 As shown, in the early stages of training, the agent's actions easily cause the flow of data to become non-convergent, resulting in low reward levels. The improved DA-D3QN, however, introduces an immediate reward mechanism through its action selection mechanism, achieving a higher average reward compared to other methods. In the later stages of training, all methods still exhibit some oscillation because the agent continuously attempts random actions to avoid getting trapped in local optima.

[0137] Table 1 shows a comparison between the load transfer strategies proposed by the intelligent method and the experience-based decision-making strategies.

[0138] Table 1 Comparison of Load Transfer Results

[0139]

[0140] Load transfer location such as Figure 7 As shown, the improved dual-agent D3QN provides the following specific ideas for the power transfer strategy: the SB220kV station supplies power to the DH110kV2M bus in series and supplies power to half of the buses of the LC, CF, LH, LF, YG, and QY stations; the ZH220kV station supplies power to the JM110kV2M bus in series and supplies power to half of the buses of the SG and GC stations; half of the load of the CF, LH, and LF stations is transferred to the JM110kV1M bus to reduce the load.

[0141] The empirical decision-making method (contingency plan) proposes a power transfer strategy where the ZH220kV and BB220kV substations supply power to the JM110kV 1M bus and DH110kV 1M bus, respectively. While this strategy involves fewer power transfer switch operations, it overlooks the risks associated with numerous transformers and loads, as well as the long series supply path. A comparison of the N-1-1 verification for both strategies is summarized in Table 2, along with a load factor comparison. Figure 8 As shown.

[0142] Table 2 Comparison of the most severe consequences of N-1-1

[0143]

[0144]

[0145] Figure 9 This section compares the effects of three-stage optimization and multi-objective optimization. Multi-objective optimization, which optimizes all three parts together without a standardized three-stage optimization order, is more likely to cause training to get stuck in local optima. In the early stages, the agent may choose more actions with a high proportion of load, risk, and reward, resulting in excessive load on some branches and failure to optimize the risk of surrounding single-supply substations.

[0146] If the closed GC-HQ line is chosen instead of the b2 line, the immediate reward values ​​of the two are similar, but the former cannot continue to optimize the risk of single supply at the SG station, and the total reward value will be lower than that of the latter. This is also the reason why the average reward value of multi-objective optimization is slightly lower than that of three-stage optimization.

[0147] To demonstrate the effectiveness of the newly proposed optimization indicators, a comparative optimization will be conducted using commonly used risk indicators (the sum of the products of each fault probability and the severity of its consequences). The fault probability is related to the line load level. The optimization strategies for both are shown in Table 3 below.

[0148] Table 3 Comparison of Optimization Indicator Effects

[0149]

[0150] After the power transfer strategy was implemented, the N-1 risk of the power grid was reduced to a minimum. Therefore, this method can be used to intelligently prevent and control the N-1 risk of the power grid and has good applicability.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for transferring newly added risky loads (N-1) in urban power grids based on an improved dual-agent D3QN approach, characterized by: The steps are as follows, and the steps are performed in sequence. Step 1: Analyze the new risks N-1 in the power grid and introduce automatic transfer switching devices to establish load transfer optimization indicators for the prevention and control of new risks; Step 2: Based on the priority of the optimization indicators established in Step 1, the load transfer optimization is divided into three stages, and the three-stage model is converted into a mathematical model of deep reinforcement learning. Step 3: Based on the deep reinforcement learning D3QN algorithm model, set up dual agents and dual experience pools to analyze the closure and opening of load transfer, reduce the dimension of action space, and improve the utilization rate of excellent samples. Step 4: Based on the D3QN algorithm, set a pre-action-changing exploration value selection strategy to replace the greedy strategy in the original D3QN algorithm. Enable the immediate reward knowledge to add a new selection strategy and arrange two exploration values ​​for each strategy that change with the training process. Accumulate high-quality samples, improve the training rate, and output a risk control and transfer strategy. The two agents established in step three learn the closing and opening operations in load transfer, and only one agent outputs the actual action for each action. The reward function for the two agents is: ， In the formula, R1, R2, R3, and R4 are the reward part of the intelligent agent, P1 is the penalty for the intelligent agent's action process not meeting the voltage amplitude constraint, P2 is the transmission power limit penalty corresponding to the voltage level and equipment model of each branch, P3 is the switching action penalty, and P4 is the transformer and load risk penalty. The reward portion is as follows: With the goal of reducing risky loads and the number of single-supply substations, the rewards that the agent can obtain in the first and second phases are established: ， ， In the formula, and This represents the cumulative reduction in the number of single-supply substations and the total active load carried by the i-th substation. This represents the cumulative amount of risky loads that have been transformed into safe loads. After the objectives of the first and second phases are achieved, the third phase of optimization will begin, focusing on reducing the line load rate. , In the formula, Indicates the current state and The reward This represents the penalty function related to the current state and radial movement. and This indicates the load rate of the branch with the highest load rate before and after the current action; The load transfer process can cause a ring network in the distribution network. By setting a bonus R4, the radial distribution structure can be restored after each load transfer. , In the formula, This indicates the state of the distribution network topology after the current action. It represents the radial structure combination of all distribution networks in the power grid.

2. The urban power grid N-1 newly added risk load transfer method based on improved dual-agent D3QN according to claim 1, characterized in that: The load transfer optimization index for step one is: , in, The number of single-supply substations in the power grid with only one backup power supply path; , in, The load carried by a risky load bus in the power grid that has only one backup power supply path; , in, This represents the highest branch load rate in the power grid. When the highest load rate is below 70%, this value is 0.

3. The urban power grid N-1 newly added risk load transfer method based on improved dual-agent D3QN according to claim 1, characterized in that: The division of the model into two and three stages in step two is based on the following criteria: In the first phase, ensure that the substation has two power supply paths, including a backup power supply path, to minimize the risk of the substation being supplied by a single power source. Substations in the risk of being supplied by a single power source will transfer part of the load to the power supply path of the superior or same-level substation through the split operation mode, or use the backup automatic transfer to form two power supply paths for the substation, thus getting rid of the risk of the substation being supplied by a single power source. In the second phase, the number of substations with single power supply will be maintained, and the maximum load will be transferred to the standby automatic transfer protection area that is in the pre-operation state to ensure two power supply paths and minimize the risk of load loss. The third stage maintains the optimization results of the first two stages and transfers the load downstream of high-load-rate branches to low-load-rate branches, balancing the load rate of each branch to ensure stable power supply to the power grid.

4. The urban power grid N-1 newly added risk load transfer method based on improved dual-agent D3QN according to claim 1, characterized in that: The dual-agent penalty part mentioned in step three is as follows: Establish a penalty for situations where the agent's actions do not meet the voltage amplitude constraints: , In the formula, This represents the per-unit voltage value of the i-th node, while and This indicates the lower and upper limits of the voltage per unit value for this node; Penalties are set based on the transmission power limits of each branch voltage level and equipment model: , In the formula, This represents the power transmitted on the i-th branch, while This indicates the maximum power that can be transmitted on this branch; Set a penalty for the switch action based on the switch operation time: , In the formula, Indicates the current number of switch actions, when In such cases, this penalty offsets the reward for reducing risk; Based on the risks associated with the transformer and load, the following penalties are set: , In the formula, , This represents the cumulative reduction in the number of single-supply substations before and after the action; , This represents the cumulative risk load before and after the action.

5. The urban power grid N-1 newly added risk load transfer method based on improved dual-agent D3QN according to claim 1, characterized in that: In step three, the dual experience pools store sample data in two separate experience pools, ranked from highest to lowest, based on the Q-value and reward value. The Q-function and Q-value are value functions in deep reinforcement learning, which can be expressed by two neural networks in D3QN as follows: , In the formula, This represents the i-th state. This represents the i-th action. The parameters of the neural network, For the set of all actions, center the action vectors.

6. The urban power grid N-1 newly added risk load transfer method based on improved dual-agent D3QN according to claim 1, characterized in that: The strategy for selecting the pre-action / change exploration value in step four is as follows: The instant reward value is used as a reference for the next action selection. Before each action selection, if the Q value is not suitable, a pre-action calculation flow is performed to obtain the state and instant reward set for the next moment. A selection mechanism for the action with the maximum instant reward is added to enable the agent to accumulate high-quality samples in the early exploration process. A greedy strategy is used to ensure the convergence of the algorithm, and two exploration values ​​are set. The exploration is conducted using a combination of three strategies; the exploration value is set. and And take a random number in [0,1], and take different action selection strategies depending on which area it falls in: ① the action with the maximum immediate reward; ② the greedy strategy action; ③ the random action; and The formula for change: , , In the formula, This represents the current number of episodes. The maximum number of training elements in an episode, where , , , .

Images