Novel power system toughness evaluation method based on deep reinforcement learning

Abstract

The invention relates to the technical field of power grid evaluation, in particular to a novel power system toughness evaluation method based on deep reinforcement learning, which comprises the following steps: acquiring operation data based on a power system; executing a Markov decision process once to obtain a training sample and a pre-decision error action set, storing the training sample into a sample library, and updating a toughness evaluation agent based on the sample library; the Markov decision process is repeatedly executed, and training of the toughness evaluation agent is achieved until the preset training frequency is met; screening out a shortest pre-decision action set based on the plurality of pre-decision fault action sets; and inputting the shortest pre-decision action set into the toughness index function to obtain a toughness evaluation result of the power system. According to the method, multiple times of evaluation can be repeatedly carried out based on the operation data under different power flow working conditions, so that the different power flow working conditions of the power system can be taken into consideration, and the power supply capability maintaining toughness of the power system under different conditions when the power supply line network collapses is comprehensively analyzed.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid assessment, and in particular to a novel power system resilience assessment method based on deep reinforcement learning. Background Art

[0002] At present, with the large-scale access of renewable energy and power electronic equipment to the power grid, the traditional power system is gradually transforming into a new power system. The reliable integration of renewable energy generation and other forms of distributed generation and local loads has led to the rapid development of new power systems. As the scale of new power systems becomes larger and larger, their structures become more and more complex. The strong intermittent nature and weak support of renewable energy generation and the rapidly growing load demand on the demand side of the power system make their operation more and more variable. This means that when the power system is subject to fault disturbances, malicious attacks or wrong decision-making operations, it is easy for the connection network within the power system to cause large-scale flow transfer, causing serious chain failures or transient instability, and ultimately threatening the safe and stable operation of the new power system.

[0003] In response to the fault disturbances, malicious attacks or decision-making errors that the power system may suffer, in addition to relying on expert experience to evaluate the power system, existing technologies have also begun to introduce artificial intelligence technologies represented by deep learning and reinforcement learning, such as using a cascading fault device to simulate the power system cascading fault process. Adjustment strategy or use Monte Carlo method to generate scenarios to simulate the power system for resilience assessment. However, the existing research schemes not only occupy a large amount of processor computing resources, but also take into account the characteristic parameters of each node in the power system and the power loss of the transmission line. It is impossible to consider the differences in the operating data of the power system under different power flow conditions, and it is impossible to provide scientific and effective guidance for the construction, maintenance and repair of related power systems. Summary of the invention

[0004] The purpose of this application is to propose a power system analysis and evaluation method for the existing various complex and changeable new power systems, so that the construction, maintenance and repair of related power systems can be guided by scientific and effective information.

[0005] In order to achieve the above objectives, the present application provides a novel power system resilience assessment method based on deep reinforcement learning, comprising the following steps:

[0006] Acquiring power system operation data based on the power system;

[0007] Perform a Markov decision process to obtain a number of agent training samples and pre-decision error action sets, and then store the several agent training samples in a sample library and update the resilience evaluation agent based on the sample library;

[0008] Repeating the Markov decision process several times to train the resilience assessment agent until a preset number of training times is met, and obtaining several pre-decision error action sets;

[0009] Based on the plurality of pre-decision error action sets, a shortest pre-decision action set is selected;

[0010] Constructing a resilience index function, and then inputting the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result;

[0011] The Markov decision process includes:

[0012] Build a fault simulation model based on the preset power system;

[0013] The preset resilience assessment agent is based on the power system operation data, selects lines to execute pre-decision error actions, realizes an interaction process with the fault simulation model, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes.

[0014] The above-mentioned new power system resilience assessment method based on deep reinforcement learning realizes the Markov decision process of the agent and the power system fault simulation model by selecting lines to perform pre-decision error actions based on the power system operation data through the resilience assessment agent, which fully considers the characteristic parameters and transmission losses in each line on the complex network structure of the power system, and reduces the computing resource occupation of the power system resilience assessment process, thereby improving the assessment accuracy and efficiency. Furthermore, this method can repeatedly evaluate the same power system based on the operating data under different power flow conditions, so that it can take into account different power flow conditions of the power system, and comprehensively analyze the resilience of the power system to maintain its power supply capacity when facing the collapse of the power line network under different conditions, so that the construction, maintenance and repair of related power systems can be guided by scientific and effective information.

[0015] In a possible implementation, the step of constructing a fault simulation model based on a preset power system includes:

[0016] Based on the preset power system, the initial power system operation data is obtained, and then the power system topology is established;

[0017] All line flows of the power system topology are calculated based on the initial power system operation data, and the power system topology and the line flows are used as fault simulation models.

[0018] It should also be noted that the initial power system operation data includes the initial load of each load node and the initial actual active power of each power generation source. The power system topology includes the connection relationship between the various components in the power system. These components include power generation sources, transmission lines, distribution networks and load nodes.

[0019] Specifically, the calculation formula for calculating all line flows of the power system topology is as shown in the following formula (1):

[0020]

[0021] Among them, P i represents the active power injected into node i in the power system topology, Q i represents the reactive power injected by node i, V i represents the voltage amplitude of node i, δ i represents the voltage phase angle of node i, Y ij With B ij is the position admittance corresponding to node i and node j.

[0022] In this implementation, by constructing a fault simulation model that includes the power system topology and line flow, the operating data, structure and properties of the preset power system are comprehensively and accurately reflected, laying the foundation for subsequent accurate power system resilience assessment and analysis.

[0023] In a possible implementation, the preset resilience assessment agent implements an interaction process with the fault simulation model by selecting a line to perform a pre-decision error action based on the power system operation data, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes, including:

[0024] The preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line;

[0025] updating the fault simulation model and recording all lines without power flow as out-of-service lines;

[0026] Performing line power flow adjustment on the fault simulation model to maintain active power balance of the fault simulation model;

[0027] Recalculating all line flows of the fault simulation model to obtain a number of overloaded lines;

[0028] After recording the overloaded line with the smallest capacity as a shutdown line, returning to the step of updating the fault simulation model and recording all lines without power flow as shutdown lines until there is no overloaded line;

[0029] The total number of suspended lines is recorded. If the total number of suspended lines is less than the preset collapse threshold, the resilience assessment agent reselects a line and returns to execute the above steps. If the total number of suspended lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round.

[0030] Specifically, after executing the pre-decision error action and updating the fault simulation model, the power system topology in the fault simulation model may be separated into multiple subnets, where the lines in the subnet without power supply have no power flow, and all the lines without power flow are recorded as out-of-service lines. Adjusting the line flow of the fault simulation model to maintain the active power balance of the fault simulation model also means adjusting the line flow of each subnet in the fault simulation model to maintain the active power balance of each subnet.

[0031] It should also be noted that the overloaded line may also be referred to as a power flow overloaded line. When the active power transmitted by the line exceeds its short-term emergency capacity, the line is considered to be an overloaded line.

[0032] In this implementation, by recording lines without power flow and overloaded lines as out-of-service lines and judging the state of the fault simulation model according to the number of out-of-service lines, the fault conditions of the power system when it is subjected to fault disturbances, malicious attacks or erroneous decisions are simply and effectively simulated, reducing the computing resource usage of the power system resilience assessment process and improving the comprehensiveness and accuracy of the assessment results.

[0033] In a possible implementation, the resilience assessment agent includes a preset Q estimation network, a preset Q target network and a preset sample library.

[0034] In a possible implementation, the preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line, including:

[0035] Based on the fault simulation model, the power system operation data is integrated into the power flow state s k ;

[0036] The power flow state s k The input is sent to the resilience assessment agent, and then the preset Q estimation network in the resilience assessment agent is used to calculate the circuit with the maximum Q estimation value through the greedy strategy as the circuit for executing the pre-decision error action. The calculation formula can be expressed as formula (2):

[0037]

[0038] Among them, a k Indicates the available routes. It represents the line with the largest Q estimate. Indicates selection The probability of k A represents the greed coefficient when the kth pre-decision error action is taken, k represents the set of routes available for selection when the kth pre-decision error occurs. represents the state of line v before the kth pre-decision error action, It means that line v is operating normally before the kth pre-decision error action. It means that line v is shut down before the kth pre-decision error action.

[0039] It should also be noted that the power system operation data is integrated into the flow state s k The calculation formula can be expressed as formula (3):

[0040]

[0041] in, is the active power of the jth load, is the active power output by the xth power source, is the active power on the nth transmission line.

[0042] In this implementation, the line with the largest Q estimate is calculated through a greedy strategy, so that the line selected each time is the line with the greatest impact on the system resilience after comprehensive consideration of the current state of the fault simulation model and the complex line relationship between each line. This ensures that the resilience assessment can comprehensively consider the decision-making errors that have the greatest impact on the system, improves the accuracy of the resilience assessment, and effectively reduces redundant operations in the power system resilience assessment process, improves computing efficiency, and reduces the computational burden of the processor.

[0043] In a possible implementation, adjusting the line power flow of the fault simulation model to maintain the active power balance of the fault simulation model includes:

[0044] Obtain the current load P of each load node in the fault simulation model l i The actual active power of each power source

[0045] Based on the current load P of each load node l i The actual active power of each power source The active power imbalance ΔP is calculated and its calculation formula is expressed as:

[0046] Among them, P load is the current total load of the fault simulation model, PG is the current total power generation of the fault simulation model;

[0047] Obtain the output upper limit of each power generation source in the fault simulation model Output limit

[0048] If the active power unbalance Cutting off part of the load in the fault simulation model;

[0049] If the active power imbalance ΔP < 0, and the absolute value of the active power imbalance Cut off some of the power generation sources in the fault simulation model until

[0050] Obtain the droop coefficient D of each generating source in the fault simulation model i , based on the droop coefficient D of each power source i Adjust the active power output of each power generation source in the fault simulation model, wherein the active power output adjustment amount of each power generation source is It is expressed as formula (4):

[0051]

[0052] When the active power output of any power source exceeds its output limit Output limit Part of the active power of any power generation source is distributed to the remaining power generation sources, so as to maintain the active power balance of the fault simulation model.

[0053] It should also be noted that when the power system topology in the fault simulation model is separated into multiple sub-grids, line flow adjustment must be performed for each sub-grid to maintain the active power balance of each sub-grid.

[0054] Specifically, when the active power unbalance amount When , it means that there is an active power shortage in the current fault simulation model or the current subnet, and the power generation source has insufficient upward capacity. At this time, it is necessary to cut off part of the load to ensure its active power balance. The expression of load cutting is shown in the following formula (5):

[0055]

[0056] in, represents the rounding up operation, and ζ represents the minimum load removal ratio.

[0057] When the active power imbalance ΔP<0, and the absolute value of the active power imbalance When , it means that there is active power surplus in the current fault simulation model or the current subnet, and the generation source is insufficient to adjust the capacity. At this time, it is necessary to cut off some generation sources in ascending order of the generation source adjustment capacity until the generation source is satisfied. To ensure its active power balance.

[0058] In this implementation, through accurate power calculation and adjustment strategy with clear priority, the characteristic parameters of each node in the power system and the power loss of the transmission line are fully considered to simulate the impact of pre-decision error actions on the power system, thereby improving the accuracy of the power system resilience assessment.

[0059] In a possible implementation, the total number of outage lines is recorded. If the total number of outage lines is less than a preset collapse threshold, the resilience assessment agent reselects a line to return to execute the above steps. If the total number of outage lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round, including:

[0060] Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overloaded line after each pre-decision error action is executed is obtained. Then construct the end mark T of the Markov decision process over , which is expressed as follows (6):

[0061]

[0062] Among them, N c is the preset crash threshold;

[0063] Based on the Markov decision process end mark T over The value is 1, which means that a Markov decision process is completed.

[0064] It should also be noted that The expression of is shown in formula 7:

[0065]

[0066] in, represents the state of line v before the k+1th pre-decision error action, N represents the total number of initial running lines before executing the pre-decision error action, It means that line v is operating normally before the k+1th pre-decision error action. It means that line v is shut down before the k+1th pre-decision error action.

[0067] In a possible implementation, storing the plurality of agent training samples in a sample library and updating the resilience assessment agent based on the sample library includes:

[0068] Based on the Markov decision process end mark T over Constructing the reward function Re k , its expression is shown in formula (8):

[0069]

[0070] Among them, λ1 is the cost of a single pre-decision error action, λ2 is the profit of the line shutdown caused by the pre-decision error action, λ3 is the one-time reward when the fault simulation model presents a preset collapse state, N k is the total number of newly suspended lines after the kth pre-decision error action when there is no overloaded line, represents the state of line v before the kth pre-decision error action, represents the state of line v before the k+1th pre-decision error action, and N represents the total number of initial operation lines before executing the pre-decision error action;

[0071] Obtain the importance of each sample in the sample library, and then based on the reward function Re k Calculate the sampling probability of each sample in the sample library based on the importance of each sample;

[0072] Based on the sampling probability of each sample, combined with the preset importance sampling weighted correction, a number of samples are obtained in the sample library through random priority sampling;

[0073] updating the Q estimation network based on a number of the samples;

[0074] Based on the Q estimation network, the importance of each sample in the sample library is updated, thereby completing the update of the resilience assessment agent.

[0075] Specifically, the reward function Re k The sampling probability of each sample in the sample library is calculated based on the importance of each sample. The calculation process is shown in the following formulas (10), (11) and (12):

[0076]

[0077] Among them, p j is the importance of the jth sample in the sample library, is the time difference error of the sample, Represents the average reward of the Markov decision process in which the sample is located. sumj represents the cumulative total reward of the Markov decision process, N sumjrepresents the length of the pre-decision error action set obtained at the end of the Markov decision process. η is the upper limit of the sample importance, α is the sample importance coefficient, α represents the participation of random priority sampling, α=1 represents complete random priority sampling, and α=0 represents uniform random sampling. P j represents the sampling probability of the jth sample, and D is the capacity of the sample library.

[0078] Furthermore, the importance sampling weighted correction expression is shown in the following formula (13):

[0079]

[0080] Among them, Φ is a set of samples obtained by random priority sampling, ψ j～Φ Represents the importance weight of the jth sample in the Φ sample set. m sim Represents the number of samples in the Φ sample set. β is the importance sampling weighted correction coefficient, and its value range is β∈[0,1]. β=0 means no weighted correction is made to the loss function, and β=1 means a complete weighted correction is made to the loss function.

[0081] Specifically, random priority sampling will select samples with smaller time difference errors and larger average rewards of the Markov decision process, which will introduce bias when accelerating the convergence speed in the early stage of agent training. Therefore, it is necessary to correct the loss function when updating the Q estimation network in combination with the weighted correction of importance sampling to limit the influence of random priority sampling. Among them, the more important the sample, the greater the restriction on its loss function, so as to reduce the influence of the loss function on the update of the Q estimation network.

[0082] Furthermore, the calculation formula for updating the Q estimation network based on a plurality of the samples is shown in the following formula (14):

[0083]

[0084] Among them, θ i represents the parameters of the Q estimation network during the i-th network update, Δ j～Φ (θ i ) is the parameter correction value provided for the j-th sample. represents the time difference error of the jth sample, The expression of is shown in formula 15:

[0085]

[0086] in, It represents the estimated value of Q, and its expression is shown in formula (16). represents the Q target value, and its expression is shown in Formula 17:

[0087]

[0088] in, Represents the parameters of the Q target network in the agent during the ith network update. After the Q estimation network is updated, the time differential error of the batch of samples is recalculated based on the Q estimation network, and then the importance of each sample in the sample library is updated, thereby completing the update of the resilience assessment agent.

[0089] In this implementation, by designing an improved priority experience replay strategy, the Q estimation network is trained to update the importance of samples in the sample library, thereby improving the efficiency and accuracy of the agent update, thereby reducing the error of the power system resilience assessment results, so that the new power system resilience assessment method based on deep reinforcement learning in this application can be applied to large-scale and complex power systems.

[0090] In a possible implementation, constructing a resilience index function, and then inputting the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result includes:

[0091] Based on the fault simulation model, the total number N of initial operating lines before executing the pre-decision error action is obtained;

[0092] Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overload line after each pre-decision error action is executed is obtained, and then the resilience index function is constructed, which is expressed as formula (18) and formula (19):

[0093]

[0094]

[0095] Where N is the total number of initial operation lines before executing the pre-decision error action, is the total number of outage lines when there is no overload line after the k-th pre-decision error action is executed, G(k) is the ratio of the total number of remaining operating lines to the total number of initial operating lines when there is no overload line after the k-th pre-decision error action is executed, k e is the total number of pre-decision error actions included in the shortest pre-decision action set, RX k is the power system resilience index corresponding to the kth pre-decision error action result;

[0096] The power system resilience index corresponding to each pre-decision error action result is used as the power system resilience assessment result.

[0097] In this implementation, G(k) is used to reflect the damage degree of the fault simulation model after the kth pre-decision error action, and then the resilience of the power system is systematically and quantitatively evaluated, where RX k It directly reflects the resilience of the power system after the kth pre-decision error action, making the resilience assessment results more interpretable and providing scientific and effective data support for the construction, maintenance and repair of the power system.

[0098] The present application also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, any possible power system resilience assessment method described above is implemented.

[0099] The novel power system resilience assessment method and computer device based on deep reinforcement learning described in this application have at least the following advantages over the prior art:

[0100] The novel power system resilience assessment method based on deep reinforcement learning described in the present application realizes the Markov decision process of the intelligent agent and the power system fault simulation model by selecting lines to perform pre-decision error actions based on the power system operation data through the resilience assessment intelligent agent, which fully considers the characteristic parameters and transmission losses in each line on the complex network structure of the power system, and reduces the computing resource occupation of the power system resilience assessment process, thereby improving the assessment accuracy and efficiency. Furthermore, this method can repeatedly perform multiple assessments based on the operating data under different power flow conditions for the same power system, so that it can take into account different power flow conditions of the power system, and comprehensively analyze the resilience of the power system to maintain its power supply capacity when facing the collapse of the power line network under different conditions, so that the construction, maintenance and repair of related power systems can be guided by scientific and effective information. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 It is a flow chart of a new power system resilience assessment method based on deep reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0103] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.

[0104] refer to Figure 1 The embodiment of the present invention provides a novel power system resilience assessment method based on deep reinforcement learning, which specifically includes:

[0105] S101. Acquire power system operation data based on the power system.

[0106] S102. Execute a Markov decision process to obtain a number of agent training samples and a pre-decision error action set, and then store the several agent training samples in a sample library and update the resilience assessment agent based on the sample library.

[0107] S103, repeatedly executing the Markov decision process several times to implement training of the resilience assessment agent until a preset number of training times is met, and obtaining several pre-decision error action sets.

[0108] S104: based on the plurality of pre-decision error action sets, screen out the shortest pre-decision action set.

[0109] S105. Construct a resilience index function, and then input the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result.

[0110] The Markov decision process includes:

[0111] Build a fault simulation model based on the preset power system;

[0112] The preset resilience assessment agent is based on the power system operation data, selects lines to execute pre-decision error actions, realizes an interaction process with the fault simulation model, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes.

[0113] In one embodiment, the constructing of a fault simulation model based on a preset power system includes:

[0114] Based on the preset power system, the initial power system operation data is obtained, and then the power system topology is established;

[0115] All line flows of the power system topology are calculated based on the initial power system operation data, and the power system topology and the line flows are used as fault simulation models.

[0116] It should also be noted that the initial power system operation data includes the initial load of each load node and the initial actual active power of each power generation source. The power system topology includes the connection relationship between the various components in the power system. These components include power generation sources, transmission lines, distribution networks and load nodes.

[0117] Specifically, the calculation formula for calculating all line flows of the power system topology is as shown in the following formula (1):

[0118]

[0119] Among them, P i represents the active power injected into node i in the power system topology, Q i represents the reactive power injected by node i, V i represents the voltage amplitude of node i, δ i represents the voltage phase angle of node i, Y ij With B ij is the position admittance corresponding to node i and node j.

[0120] In this embodiment, by constructing a fault simulation model including the power system topology and line flow, the operating data, structure and properties of the preset power system are comprehensively and accurately reflected, laying the foundation for subsequent accurate power system resilience assessment and analysis.

[0121] In one embodiment, the preset resilience assessment agent implements an interaction process with the fault simulation model by selecting a line to perform a pre-decision error action based on the power system operation data, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes, including:

[0122] The preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line;

[0123] updating the fault simulation model and recording all lines without power flow as out-of-service lines;

[0124] Performing line power flow adjustment on the fault simulation model to maintain active power balance of the fault simulation model;

[0125] Recalculating all line flows of the fault simulation model to obtain a number of overloaded lines;

[0126] After recording the overloaded line with the smallest capacity as a shutdown line, returning to the step of updating the fault simulation model and recording all lines without power flow as shutdown lines until there is no overloaded line;

[0127] The total number of suspended lines is recorded. If the total number of suspended lines is less than the preset collapse threshold, the resilience assessment agent reselects a line and returns to execute the above steps. If the total number of suspended lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round.

[0128] Specifically, after executing the pre-decision error action and updating the fault simulation model, the power system topology in the fault simulation model may be separated into multiple subnets, where the lines in the subnet without power supply have no power flow, and all the lines without power flow are recorded as out-of-service lines. Adjusting the line flow of the fault simulation model to maintain the active power balance of the fault simulation model also means adjusting the line flow of each subnet in the fault simulation model to maintain the active power balance of each subnet.

[0129] It should also be noted that the overloaded line may also be referred to as a power flow overloaded line. When the active power transmitted by the line exceeds its short-term emergency capacity, the line is considered to be an overloaded line.

[0130] In this embodiment, by recording lines without power flow and overloaded lines as out-of-service lines and judging the state of the fault simulation model according to the number of out-of-service lines, the fault conditions of the power system when it is subjected to fault disturbance, malicious attack or erroneous decision-making operation are simply and effectively simulated, thereby reducing the computing resource usage of the power system resilience assessment process and improving the comprehensiveness and accuracy of the assessment results.

[0131] In one embodiment, the resilience assessment agent includes a preset Q estimation network, a preset Q target network and a preset sample library.

[0132] In one embodiment, the preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line, including:

[0133] Based on the fault simulation model, the power system operation data is integrated into the power flow state s k ;

[0134] The power flow state s k The input is sent to the resilience assessment agent, and then the preset Q estimation network in the resilience assessment agent is used to calculate the circuit with the maximum Q estimation value through the greedy strategy as the circuit for executing the pre-decision error action. The calculation formula can be expressed as formula (2):

[0135]

[0136] Among them, a k Indicates the available routes. It represents the line with the largest Q estimate. Indicates selection The probability of k A represents the greed coefficient when the kth pre-decision error action is taken, k represents the set of routes available for selection when the kth pre-decision error occurs. represents the state of line v before the kth pre-decision error action, It means that line v is operating normally before the kth pre-decision error action. It means that line v is shut down before the kth pre-decision error action.

[0137] It should also be noted that the power system operation data is integrated into the flow state s k The calculation formula can be expressed as formula (3):

[0138]

[0139] in, is the active power of the jth load, is the active power output by the xth power source, is the active power on the nth transmission line.

[0140] In this embodiment, the line with the largest Q estimation value is calculated through a greedy strategy, so that the line selected each time is the line with the greatest impact on the system resilience after comprehensive consideration of the current state of the fault simulation model and the complex line relationship between the lines, ensuring that the resilience assessment can comprehensively consider the decision-making errors that have the greatest impact on the system, thereby improving the accuracy of the resilience assessment, and effectively reducing redundant operations in the power system resilience assessment process, improving computing efficiency, and reducing the computing burden of the processor.

[0141] In one embodiment, adjusting the line power flow of the fault simulation model to maintain the active power balance of the fault simulation model includes:

[0142] Obtain the current load P of each load node in the fault simulation model l i The actual active power of each power source

[0143] Based on the current load P of each load node l i The actual active power of each power source The active power imbalance ΔP is calculated and expressed as:

[0144] Among them, P load is the current total load of the fault simulation model, P G is the current total power generation of the fault simulation model;

[0145] Obtain the output upper limit of each power generation source in the fault simulation model Output limit

[0146] If the active power unbalance Cutting off part of the load in the fault simulation model;

[0147] If the active power imbalance ΔP < 0, and the absolute value of the active power imbalance Cut off some of the power generation sources in the fault simulation model until

[0148] Obtain the droop coefficient D of each generating source in the fault simulation model i , based on the droop coefficient D of each power source i Adjust the active power output of each power generation source in the fault simulation model, wherein the active power output adjustment amount of each power generation source is It is expressed as formula (4):

[0149]

[0150] When the active power output of any power source exceeds its output limit Output limit Part of the active power of any power generation source is distributed to the remaining power generation sources, so as to maintain the active power balance of the fault simulation model.

[0151] It should also be noted that when the power system topology in the fault simulation model is separated into multiple sub-grids, line flow adjustment must be performed for each sub-grid to maintain the active power balance of each sub-grid.

[0152] Specifically, when the active power unbalance amount When , it means that there is an active power shortage in the current fault simulation model or the current subnet, and the power generation source has insufficient upward capacity. At this time, it is necessary to cut off part of the load to ensure its active power balance. The expression of load cutting is shown in the following formula (5):

[0153]

[0154] in, represents the rounding up operation, and ζ represents the minimum load removal ratio.

[0155] When the active power imbalance ΔP<0, and the absolute value of the active power imbalance When , it means that there is active power surplus in the current fault simulation model or the current subnet, and the generation source is insufficient to adjust the capacity. At this time, it is necessary to cut off some generation sources in ascending order of the generation source adjustment capacity until the generation source is satisfied. To ensure its active power balance.

[0156] In this embodiment, through accurate power calculation and a clear priority adjustment strategy, the characteristic parameters of each node in the power system and the power loss of the transmission line are fully considered to simulate the impact of pre-decision errors on the power system, thereby improving the accuracy of the power system resilience assessment.

[0157] In one embodiment, the total number of outage lines is recorded. If the total number of outage lines is less than a preset collapse threshold, the resilience assessment agent reselects a line to return to execute the above steps. If the total number of outage lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round, including:

[0158] Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overloaded line after each pre-decision error action is executed is obtained. Then construct the end mark T of the Markov decision process over , which is expressed as follows (6):

[0159]

[0160] Among them, N c is the preset crash threshold;

[0161] Based on the Markov decision process end mark T over The value is 1, which means that a Markov decision process is completed.

[0162] It should also be noted that The expression of is shown in formula 7:

[0163]

[0164] in, represents the state of line v before the k+1th pre-decision error action, N represents the total number of initial running lines before executing the pre-decision error action, It means that line v is operating normally before the k+1th pre-decision error action. It means that line v is shut down before the k+1th pre-decision error action.

[0165] In one embodiment, storing the plurality of agent training samples in a sample library and updating the resilience assessment agent based on the sample library includes:

[0166] Based on the Markov decision process end mark T over Constructing the reward function Re k , its expression is shown in formula (8):

[0167] Re k =-λ1+λ2Nk +λ3T over (Formula 8)

[0168]

[0169] Among them, λ1 is the cost of a single pre-decision error action, λ2 is the profit of the line shutdown caused by the pre-decision error action, λ3 is the one-time reward when the fault simulation model presents a preset collapse state, N k is the total number of newly suspended lines after the kth pre-decision error action when there is no overloaded line, represents the state of line v before the kth pre-decision error action, represents the state of line v before the k+1th pre-decision error action, and N represents the total number of initial operation lines before executing the pre-decision error action;

[0170] Obtain the importance of each sample in the sample library, and then based on the reward function Re k Calculate the sampling probability of each sample in the sample library based on the importance of each sample;

[0171] Based on the sampling probability of each sample, combined with the preset importance sampling weighted correction, a number of samples are obtained in the sample library through random priority sampling;

[0172] updating the Q estimation network based on a number of the samples;

[0173] Based on the Q estimation network, the importance of each sample in the sample library is updated, thereby completing the update of the resilience assessment agent.

[0174] Specifically, the reward function Re k The sampling probability of each sample in the sample library is calculated based on the importance of each sample. The calculation process is shown in the following formulas (10), (11) and (12):

[0175]

[0176] Among them, p j is the importance of the jth sample in the sample library, e TDj is the time difference error of the sample, Represents the average reward of the Markov decision process in which the sample is located. sumj represents the cumulative total reward of the Markov decision process, N sumj represents the length of the pre-decision error action set obtained at the end of the Markov decision process. η is the upper limit of the sample importance, α is the sample importance coefficient, α represents the participation of random priority sampling, α=1 represents complete random priority sampling, and α=0 represents uniform random sampling. Pj represents the sampling probability of the jth sample, and D is the capacity of the sample library.

[0177] Furthermore, the importance sampling weighted correction expression is shown in the following formula (13):

[0178]

[0179] Among them, Φ is a set of samples obtained by random priority sampling, ψ j～Φ Represents the importance weight of the jth sample in the Φ sample set. m sim Represents the number of samples in the Φ sample set. β is the importance sampling weighted correction coefficient, and its value range is β∈[0,1]. β=0 means no weighted correction is made to the loss function, and β=1 means a complete weighted correction is made to the loss function.

[0180] Specifically, random priority sampling will select samples with smaller time difference errors and larger average rewards of the Markov decision process, which will introduce bias when accelerating the convergence speed in the early stage of agent training. Therefore, it is necessary to correct the loss function when updating the Q estimation network in combination with the weighted correction of importance sampling to limit the influence of random priority sampling. Among them, the more important the sample, the greater the restriction on its loss function, so as to reduce the influence of the loss function on the update of the Q estimation network.

[0181] Furthermore, the calculation formula for updating the Q estimation network based on a plurality of the samples is shown in the following formula (14):

[0182]

[0183] Among them, θ i represents the parameters of the Q estimation network during the i-th network update, Δ j～Φ (θ i ) is the parameter correction value provided for the j-th sample. represents the time difference error of the jth sample, The expression of is shown in formula 15:

[0184]

[0185] in, It represents the estimated value of Q, and its expression is shown in formula (16). represents the Q target value, and its expression is shown in Formula 17:

[0186]

[0187] in, Represents the parameters of the Q target network in the agent during the ith network update. After the Q estimation network is updated, the time differential error of the batch of samples is recalculated based on the Q estimation network, and then the importance of each sample in the sample library is updated, thereby completing the update of the resilience assessment agent.

[0188] In this embodiment, the importance of samples in the sample library is updated by training the Q estimation network, thereby improving the efficiency and accuracy of the agent update, thereby reducing the error of the power system resilience assessment results, so that the new power system resilience assessment method based on deep reinforcement learning in this application can be applied to large-scale and complex power systems.

[0189] In a possible embodiment, constructing a resilience index function, and then inputting the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result includes:

[0190] Based on the fault simulation model, the total number N of initial operating lines before executing the pre-decision error action is obtained;

[0191] Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overloaded line after each pre-decision error action is executed is obtained, and then a resilience index function is constructed, which is expressed as:

[0192]

[0193]

[0194] Where N is the total number of initial operation lines before executing the pre-decision error action, is the total number of outage lines when there is no overload line after the k-th pre-decision error action is executed, G(k) is the ratio of the total number of remaining operating lines to the total number of initial operating lines when there is no overload line after the k-th pre-decision error action is executed, k e is the total number of pre-decision error actions included in the shortest pre-decision action set, RX k is the power system resilience index corresponding to the kth pre-decision error action result;

[0195] The power system resilience index corresponding to each pre-decision error action result is used as the power system resilience assessment result.

[0196] In this embodiment, G(k) is used to reflect the damage degree of the fault simulation model after the kth pre-decision error action, and then the resilience of the power system is systematically and quantitatively evaluated, where RX kIt directly reflects the resilience of the power system after the kth pre-decision error action, making the resilience assessment results more interpretable and providing scientific and effective data support for the construction, maintenance and repair of the power system.

[0197] In addition, an embodiment of the present application also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the power system resilience assessment method as described in any of the above embodiments is implemented.

[0198] The computer device may specifically be a terminal or a server. The computer device includes a processor, a memory, a network interface, a display, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an input impedance modeling method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0199] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD), etc.

[0200] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A new power system resilience assessment method based on deep reinforcement learning, characterized in that: include: Acquiring power system operation data based on a preset power system; Perform a Markov decision process to obtain a number of agent training samples and pre-decision error action sets, and then store the several agent training samples in a sample library and update the resilience evaluation agent based on the sample library; Repeating the Markov decision process several times to train the resilience assessment agent until a preset number of training times is met, and obtaining several pre-decision error action sets; Based on the plurality of pre-decision error action sets, a shortest pre-decision action set is selected; Constructing a resilience index function, and then inputting the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result; The Markov decision process includes: Build a fault simulation model based on the preset power system; The preset resilience assessment agent is based on the power system operation data, selects lines to execute pre-decision error actions, realizes an interaction process with the fault simulation model, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes.

2. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 1, it is characterized in that: The method of constructing a fault simulation model based on a preset power system includes: Based on the preset power system, the initial power system operation data is obtained, and then the power system topology is established; All line flows of the power system topology are calculated based on the initial power system operation data, and the power system topology and the line flows are used as fault simulation models.

3. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 2, it is characterized in that: The preset resilience assessment agent implements an interaction process with the fault simulation model by selecting a line to perform a pre-decision error action based on the power system operation data, and obtains an agent training sample, until the fault simulation model presents a preset collapse state after multiple interaction processes, including: The preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line; updating the fault simulation model and recording all lines without power flow as outage lines; Performing line power flow adjustment on the fault simulation model to maintain active power balance of the fault simulation model; Recalculating all line flows of the fault simulation model to obtain a number of overloaded lines; After recording the overloaded line with the smallest capacity as a shutdown line, returning to the step of updating the fault simulation model and recording all lines without power flow as shutdown lines until there is no overloaded line; The total number of suspended lines is recorded. If the total number of suspended lines is less than the preset collapse threshold, the resilience assessment agent reselects a line and returns to execute the above steps. If the total number of suspended lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round.

4. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 3, it is characterized in that: The resilience assessment agent includes a preset Q estimation network, a preset Q target network and a preset sample library.

5. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 4, it is characterized in that: The preset resilience assessment agent selects a line based on the power system operation data, and then executes a pre-decision error action to shut down the line, including: Based on the fault simulation model, the power system operation data is integrated into the power flow state s k ; The power flow state s k The input is sent to the resilience assessment agent, and then the preset Q estimation network in the resilience assessment agent is used to calculate the line with the maximum Q estimation value through the greedy strategy as the line for executing the pre-decision error action. The calculation formula can be expressed as: Among them, a k Indicates the available routes. It represents the line with the largest Q estimate. Indicates selection The probability of k A represents the greed coefficient when the kth pre-decision error action is taken, k represents the set of routes available for selection when the kth pre-decision error occurs. represents the state of line v before the kth pre-decision error action, It means that line v is operating normally before the kth pre-decision error action. It means that line v is shut down before the kth pre-decision error action.

6. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 5, it is characterized in that: The line power flow is adjusted for the fault simulation model to maintain the active power balance of the fault simulation model, including: Obtain the current load P of each load node in the fault simulation model l i The actual active power of each power source Based on the current load P of each load node l i The actual active power of each power source The active power imbalance ΔP is calculated and its calculation formula is expressed as: Among them, P load is the current total load of the fault simulation model, P G is the current total power generation of the fault simulation model; Obtain the output upper limit of each power generation source in the fault simulation model Output limit If the active power unbalance Cutting off part of the load in the fault simulation model; If the active power imbalance ΔP < 0, and the absolute value of the active power imbalance Cut off some of the power generation sources in the fault simulation model until Obtain the droop coefficient D of each generating source in the fault simulation model i , based on the droop coefficient D of each power source i Adjust the active power output of each power generation source in the fault simulation model, wherein the active power output adjustment amount of each power generation source is It is expressed as: When the active power output of any power source exceeds its output limit Output limit Part of the active power of any power generation source is distributed to the remaining power generation sources, so as to maintain the active power balance of the fault simulation model.

7. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 6, it is characterized in that: The total number of outage lines is recorded. If the total number of outage lines is less than a preset collapse threshold, the resilience assessment agent reselects a line to return to execute the above steps. If the total number of outage lines is greater than or equal to the collapse threshold, it is considered that the fault simulation model presents a preset collapse state in this round, including: Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overloaded line after each pre-decision error action is executed is obtained. Then construct the end mark T of the Markov decision process over , which is expressed as: Among them, N c is the preset crash threshold; Based on the Markov decision process end mark T over The value is 1, which means that a Markov decision process is completed.

8. According to a new power system resilience assessment method based on deep reinforcement learning according to claim 7, it is characterized in that: The step of storing a plurality of the agent training samples in a sample library and updating the resilience assessment agent based on the sample library comprises: Based on the Markov decision process end mark T over Constructing the reward function Re k , whose expression is: Re k =-λ1+λ2N k +λ3T over Among them, λ1 is the cost of a single pre-decision error action, λ2 is the profit of the line shutdown caused by the pre-decision error action, λ3 is the one-time reward when the fault simulation model presents a preset collapse state, N k is the total number of newly suspended lines after the kth pre-decision error action when there is no overloaded line; Obtain the importance of each sample in the sample library, and then based on the reward function Re k Calculate the sampling probability of each sample in the sample library based on the importance of each sample; Based on the sampling probability of each sample, combined with the preset importance sampling weighted correction, a number of samples are obtained in the sample library through random priority sampling; updating the Q estimation network based on a number of the samples; Based on the Q estimation network, the importance of each sample in the sample library is updated, thereby completing the update of the resilience assessment agent.

9. A novel power system resilience assessment method based on deep reinforcement learning according to claim 8, characterized in that: The constructing of the resilience index function, and then inputting the shortest pre-decision action set into the resilience index function to obtain a power system resilience assessment result, includes: Based on the fault simulation model, the total number N of initial operating lines before executing the pre-decision error action is obtained; Based on the fault simulation model and the shortest pre-decision action set, the total number of outage lines when there is no overloaded line after each pre-decision error action is executed is obtained, and then a resilience index function is constructed, which is expressed as: Where N is the total number of initial operation lines before executing the pre-decision error action, is the total number of outage lines when there is no overload line after the k-th pre-decision error action is executed, G(k) is the ratio of the total number of remaining operating lines to the total number of initial operating lines when there is no overload line after the k-th pre-decision error action is executed, k e is the total number of pre-decision error actions included in the shortest pre-decision action set, RX k is the power system resilience index corresponding to the kth pre-decision error action result; The power system resilience index corresponding to each pre-decision error action result is used as the power system resilience assessment result.

10. A computer device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the method described in any one of claims 1 to 9 is implemented.

Images