A power transmission network outage maintenance scheduling optimization method based on Q learning

Abstract

The application provides a power grid outage maintenance scheduling optimization method based on Q learning, which comprises the following steps: generating a Gantt chart by acquiring an original power grid topology file, and establishing a state judgment rule and a target function of reinforcement learning; integrating maintenance plans of subordinate units to the Gantt chart to form an initial maintenance plan; calculating an initial power flow collection by using a built-in power flow solver of Matlab, and evaluating a power grid operation state; in the agent training process, an epsilon-greedy method is used to select a maintenance action, the Gantt chart and the topology file are updated, and the Q value is updated through a Bellman equation, so that the maintenance plan is optimized; and the Gantt chart of a final outage maintenance plan is output after the training is completed. The power grid scheduling problem is abstracted as a Q learning problem, the outage scheduling is optimized by using a Q learning algorithm through reasonable constraints and a target function, the work intensity of dispatchers is reduced, the talent training cost is reduced, the scheduling efficiency and reliability are improved, and the key technical problems of power grid scheduling are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grid planned outage scheduling, in particular to a power transmission network outage maintenance scheduling optimization method based on Q learning. BACKGROUND

[0002] In order to enable the power transmission network to operate stably for a long time, reasonably arranging the outage maintenance plan of the power transmission network in the scheduling area not only relates to the normal production and life of the power users, but also reflects the service quality and operation efficiency of the power grid dispatching department.

[0003] At present, the main means for regional power transmission network outage scheduling is artificial arrangement. With the development of social economy, the complexity of the topology of the regional power transmission network is increasing, and the types of power users in the same region are increasing, which makes it difficult for artificial arrangement to be reasonable. At the same time, due to the high time cost of learning scheduling, the scheduling cannot quickly cultivate talents to meet the development of the regional power grid.

[0004] In addition, when facing special period scheduling, since most of the dispatchers' schemes are based on experience priority, the proposed outage scheduling scheme may not be the most appropriate. These factors together cause the power transmission network outage plan proposed by the current artificial may have potential risks that are not conducive to the long-term safe and reliable operation of the power transmission network, affecting power supply continuity, reliability, reducing user power supply satisfaction, increasing user complaints, and increasing the work burden of power grid workers.

[0005] Therefore, it is urgent to propose a power grid outage maintenance plan optimization arrangement method based on Q learning, which uses the Q learning concept in reinforcement learning in artificial intelligence to realize the reasonable arrangement of outage scheduling, reduce the work intensity of dispatchers, and reduce the training cost of scheduling. SUMMARY

[0006] In view of the deficiencies in the prior art, the present application aims to provide a power transmission network outage maintenance scheduling optimization method based on Q learning. The problems raised in the above background technology are solved.

[0007] In order to achieve the above-mentioned purpose, the present application is realized by the following technical scheme: a power transmission network outage maintenance scheduling optimization method based on Q learning, comprising the steps of:

[0008] Generating a corresponding Gantt chart by acquiring the original topology file of the power grid, and establishing the state judgment rule and the objective function of the current intelligent agent reinforcement learning;

[0009] Integrating the maintenance plan of the subordinate unit into the Gantt chart to form an initial maintenance plan;

[0010] Using the built-in power flow solver of matlab to calculate the initial power flow set, and evaluating the initial operation state of the power grid;

[0011] In the double cycle nested process of the agent training, the maintenance action is selected by the epsilon-greedy method, and the Gantt chart and the topology file are updated, and the Q value is updated by using the Bellman equation, so that the maintenance plan is optimized, and the safety and efficiency of the power grid operation are improved.

[0012] After the agent training is completed, the Gantt chart corresponding to the final power outage maintenance plan is output, so that the efficiency and reliability of the power grid dispatching are ensured.

[0013] Compared with the prior art, the beneficial effects of the present application are:

[0014] By abstracting the power grid dispatching problem as a Q learning problem and setting reasonable constraints and objective functions, the Q learning algorithm in reinforcement learning is used to optimize the power outage dispatching, which can reduce the work intensity of the dispatching personnel, reduce the dispatching personnel training cost, and improve the efficiency and reliability of the power outage dispatching. At the same time, through iteration training and optimization, the present application can provide the optimal power outage maintenance plan for the power grid, ensure the long-term stable operation of the power grid, improve the power supply continuity and reliability, reduce user complaints, and effectively solve the key technical problems in power grid dispatching. BRIEF DESCRIPTION OF DRAWINGS

[0015] The disclosure of the present application will be described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes, and are not intended to limit the scope of protection of the present application. In the drawings, the same reference numerals are used to refer to the same parts. Among them:

[0016] Figure 1 A flowchart of the power outage maintenance plan optimization arrangement method based on Q learning proposed in an embodiment of the present application is shown.

[0017] Figure 2 A power outage topology diagram of an embodiment IEEE39 node system is shown.

[0018] Figure 3 A Gantt chart G0 corresponding to the initial power outage plan is shown.

[0019] Figure 4 A power outage plan diagram G after optimization by the agent is shown. *

[0020] Figure 5 A training effect sum-reward diagram is shown. DETAILED DESCRIPTION

[0021] ​It is easy to understand that, according to the technical solutions of the present application, those skilled in the art can propose various structures and implementation modes that can be replaced with each other without changing the essential spirit of the present application. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solutions of the present application, and should not be regarded as the whole or as a limitation or restriction on the technical solutions of the present application.

[0022] The present application will be further described in detail below in conjunction with the drawings, but not as a limitation on the present application.

[0023] As an understanding of the technical concept and implementation principle of the present application, the main purpose of the power grid outage maintenance scheduling optimization method based on Q learning proposed by the present application is to solve the problem of how to abstract the scheduling problem as a Q learning problem in the prior art, and set a reasonable constraint and target function dimension for the Q learning agent, so that it can give the optimal power grid outage scheduling of the current area within a reasonable training time.

[0024] In specific implementation, as shown in Figure 1 As an embodiment of the present application, the power grid outage maintenance scheduling optimization method based on Q learning proposed by the present application includes the following steps:

[0025] S1, generate the corresponding Gantt chart by obtaining the original topology file of the power grid, and establish the state judgment rule and the target function of the current agent reinforcement learning, to provide a clear initial state and optimization direction for the power grid maintenance.

[0026] S2, integrate the subordinate unit maintenance plan into the Gantt chart to form an initial maintenance plan, and ensure the systematicness and coordination of the maintenance plan;

[0027] S3, use the built-in power flow solver of matlab to calculate the initial power flow set representing the running state of the power grid under the initial outage maintenance plan, and evaluate the running state of the power grid under the initial maintenance plan;

[0028] S4, in the double-loop nested process of agent training, select the maintenance action by epsilon-greedy method and update the Gantt chart and topology file, calculate the reward value reflecting the running state of the power grid and the quality of the maintenance plan, and update the Q value by using Bellman equation, to optimize the maintenance plan and improve the safety and efficiency of the power grid operation;

[0029] S5, after the agent training is completed, output the Gantt chart corresponding to the final outage maintenance plan and the training effect test, to verify the effectiveness of the optimization scheduling method and ensure the high efficiency and reliability of the power grid scheduling.

[0030] In one embodiment of the present invention, it should be noted that, in step S1, the specific implementation method of obtaining the original topology file of the power grid and generating the corresponding Gantt chart is as follows: obtaining the original topology file of the power grid and generating the corresponding two-dimensional Gantt chart G according to n，row，col (n=1, 2, ..., is the number of Gantt charts generated under this patented method), row=30 is the number of days in the month, col is the total number of repairable equipment in the current regional power grid (including the number of transmission lines and motors in the current topology), G in the Gantt chart i,j =1 means that the jth device works normally on the i-th day, G i,j =0 means that the jth device needs to be shut down for maintenance on the i-th day. The initial Gantt chart should not have any equipment for maintenance. G can be used when the Gantt chart power outage plan is not modified. n Direct expression.

[0031] When establishing the state judgment rules of the current agent reinforcement learning, it is first necessary to determine the number of transmission lines branch and the number of motors gen through the original topology file of the power grid, and calculate the total number of repairable equipment col in the power grid. After providing the data basis for the state judgment rule, the state judgment rule C is composed of a single rule c1, C = {c1}, where the single rule c1 ensures that on any given day, the number of devices k that are out of power for maintenance is month No more than the total number of devices that can be repaired on the power grid minus the maximum number of devices in operation on that day row is the number of days, is the number of devices running on that day. That is, the state judgment rule of the current agent reinforcement learning is:

[0032] C={c1}(2)

[0033]

[0034] In one embodiment of the present invention, k month The number of devices that experienced power outages on a single day during the month when there was a power outage for maintenance. It refers to the amount of equipment in operation on a day in the month when there is a power outage for maintenance.

[0035] Example: Figure 2 As shown, according to the current topology, it can be determined that branch=46, gen=10, col=branch+gen=56.

[0036] In step S1, it is necessary to establish the objective function F based on the state judgment rule C of the intelligent agent reinforcement learning by considering the minimization of voltage deviation of the grid nodes, the minimum active network loss of the entire network transmission line, and the convergence of the power flow. The specific implementation method is as follows:

[0037] According to the obtained Gantt chart Gn Modify the original topology file and use the built-in power flow solver in Matlab to calculate the power flow set PF of the current month including the power outage maintenance plan in units of days n The optimization result objectives are set as follows: after optimization, the number of nodes exceeding the voltage limit in the entire network is the least, the active power loss of the transmission lines in the entire network is the lowest, and the power flow converges after optimization. The above objective function is expressed as:

[0038]

[0039] Among them PF n is the power flow collection of the power outage plan after optimization by the intelligent agent, PF·V is the per-unit value of the node voltage in the current power flow collection (the reference node voltage of the current network topology is 1), PF·P loss is the active power loss of the transmission line in the current power flow collection, and PF success is the power flow convergence flag of each day in the power flow collection (a value of 1 indicates convergence, and a value of 0 indicates non-convergence).

[0040] Example: Assume that the maximum number of training episodes for the reinforcement learning agent in this example is 1200, and the maximum number of steps that can be taken in each training episode is 50.

[0041] In one embodiment of the present invention, it should be noted that in step S2, the specific implementation method of forming the initial maintenance plan is: the maintenance days of the corresponding equipment submitted by the subordinate units should be an array E equp =[Day1,Day2,…,Day n ](Day n ,n=1,2,……is the equipment maintenance date of the month, and the subscript equp is the current equipment in the Gantt chart G n In the example of the present invention, Figure 2 As shown in the figure, the power outage plan equipment submitted in the current month has been marked in the topology diagram. The maintenance plans submitted by all subordinate units will appear in the form of a collection E = {E equp1 ,E equp2 ,…E equpn}, all submitted maintenance plans are mapped to the Gantt chart to form the initial plan G0. Figure 2 It can be seen that in this example, the number of devices under maintenance this month is 5, and the corresponding device numbers in the topology file are 2, 10, 28, 31, and 44. The collection of devices under maintenance this month and the array corresponding to each maintenance device are:

[0042]

[0043] It can be seen that all devices are in the same period of maintenance, there is a serious conflict needs to be re-planned. By E can be determined that a device in the current month all possible actions have 55, so the action array a of a single device is:

[0044] a2 = [1, 2, …, 27, 0, -27, …, -2, -1] (6)

[0045] Where the positive n represents the overall plan of the current device down n days, negative n represents the overall plan of the current device up n days, and 0 represents no adjustment to the current plan;

[0046] The action set A is: A = {a2, a 10 , a 28 , a 31 , a 32} (7).

[0047] Based on the above technical concept, it needs to be explained that after forming the initial maintenance plan and before performing the initial power flow set calculation, a Q table needs to be initialized to store the expected return value of each power grid operating state and maintenance action pair, to provide an initial point for the agent, so as to continuously update and optimize through learning in the subsequent training process. The process is as follows: first, according to the number of devices to be shut down for maintenance in the month, and combining the N-k principle in the power system, the state space dimension of the Q table is determined, which reflects all possible state combinations of the power grid considering the possibility of k devices being shut down at the same time; second, define the action space, i.e. the maintenance actions that the agent can take in each state; third, create a two-dimensional array as the Q table, with the number of rows corresponding to the dimension of the state space and the number of columns corresponding to the dimension of the action space; finally, initialize all entries in the Q table to zero or a small decimal value close to zero, which indicates that at the beginning of training, the agent lacks understanding of the expected return value of any state-action pair, and needs to gradually learn and adjust through subsequent reinforcement learning process.

[0048] In an embodiment of the present application, it needs to be explained that the specific implementation of evaluating the operating state under the initial maintenance plan of the power grid in S3 step is: modifying the corresponding topology file according to the Gantt chart G0, and using the built-in power flow solver in matlab to calculate the monthly power flow set PF0 = {pf1, pf2, …, pf row} of the initial outage plan (if there are multiple devices to be shut down in a day, it is also considered as a day of outage plan); Wherein pf nThe power grid power flow on the nth maintenance day of the month is determined according to the judgment rule C to determine the state S1 corresponding to the current Gantt chart G0. S1=C{G0}=5 (8), the dimension of the Q table for storing Q learning is determined according to the number of outage maintenance equipment in the month and the N-k principle in the power system, and the current Q table should be a table of 5 rows and 275 columns, wherein the row represents the state S, corresponding to the maximum number of outage equipment k in the N-k principle in the month, and the column represents the action index of all equipment. All values of the Q table are initialized to -1000.

[0049] As an embodiment of the application, the initial operation state of the power grid is the operation state S1 of the power grid when the initial outage maintenance plan is implemented, and the operation state S1 is determined by the state judgment rule C, S1=C{G0}, wherein G0 is the current Gantt chart. In S4, the maintenance action is selected by the epsilon-greedy method, and the Gantt chart and the topology file are updated, the reward value reflecting the operation state of the power grid and the quality of the maintenance plan is calculated, and the Q value is updated by using the Bellman equation to optimize the maintenance plan and improve the safety and efficiency of the operation of the power grid.

[0050] In specific implementation, the double-loop nested process of the intelligent agent training is entered, the operation state S1 is taken as the starting point of the intelligent agent training process, after the maintenance action is selected by the epsilon-greedy method and the Gantt chart and the topology file are updated, the operation state S2 of the power grid after optimization is also determined by the state judgment rule C to reflect the influence of the maintenance action on the operation state of the power grid, and a more optimal voltage level, lower network loss and higher power flow convergence are realized. Specifically, only one meaningful action a is selected from the action set A according to the epsilon-greedy method in the current step equpn,k , modify E m {……,E equpn}, modify the Gantt chart to G m according to the selected action, modify the topology file for calculation according to G m , still use the built-in power flow solver in matlab to calculate the outage power flow plan set PF m ={pf1,pf2,…,pf row} after the action, determine the state S2 according to the judgment rule C. Set the action reward value mapping function f(PF);

[0051]

[0052] PF m is the outage plan power flow set after the action of the intelligent agent, PF m ·V is the per-unit value of the node voltage in the current power flow set (with the reference node voltage of the current network topology being 1), PF m ·P lossPF m success is the power flow convergence flag in each day of the power flow set (value 1 represents convergence, value 0 represents non-convergence);

[0053] According to the power flow set PF0, PF m and the action value mapping rule, the instantaneous reward value obtained after the current action is calculated;

[0054] reward m =f(PF m )-f(PF0)(10)

[0055] S2 is overlaid to S1, and the Q value is calculated by using the Bellman equation;

[0056] Q(S1,a equpn,k )=Q(S1,a equpn,k )+α[reward m +γmax a， Q(S2,a′)-Q(S1,a equpn,k )] (11)

[0057] In the formula, Q(S1,a equpn,k ) is the column corresponding to the action index in the Q table S1 row a equpn,k after the current action is performed, α is the learning rate, γ is the discount factor, and max a′ Q(S2,a′) is the maximum Q value in the meaningful action set a′ in the Q table S2 row.

[0058] Based on the above technical concept, the training process of the agent also needs to be result-verified in the double-loop nested process of agent training, and whether the training process is completed or not depends on whether the preset condition is reached. Specifically, the agent will continue to perform the training loop until any of the following conditions is met: the maximum number of training rounds is reached or the maximum number of steps in any training round is reached, wherein,

[0059] If the agent performs the maximum number of steps in any training round but does not meet the optimization target or does not reach the expected performance indicator, the current training process is considered incomplete and needs to continue; otherwise,

[0060] If the agent completes all preset steps and meets the optimization target or performance indicator in any training round, the current training process is considered complete, and after the training is completed, the agent outputs the two-dimensional Gantt chart G * corresponding to the final power outage repair plan and the sum of reward values of all steps in each training round, which serves as a basis for subsequent evaluation of training effect, that is, the Gantt chart G *And the sum of the reward of all steps of each episode is used as the training effect test.

[0061]

[0062] It can be understood that the above result test ensures the effectiveness of the outage maintenance plan optimization arrangement method to meet the safety and efficiency requirements of power grid operation.

[0063] As an embodiment of the present application, in order to better introduce the present application, the following will be described in detail through a specific simulation example:

[0064] As shown in Figure 2 , in the IEEE39 node topology, the current state judgment rules (2), (3), and the objective function (4) are set. The equipment to be maintained in the month is: 2, 10, 28, 31, 44, and the maintenance plan time period submitted in the month is (5). The Gantt chart G0 can be determined according to the above conditions as shown in Figure 3 , the action and action set are determined as (6), (7). According to (3) and G0, the state S1=5 is determined, and the Q table is initialized according to the above conditions as shown in Table 1. After the step (4), (5) cycle iteration, the final Q table is shown in Table 2, and the scheduling personnel can further learn or adjust according to the iteration Q table. The final outage plan Gantt chart G of the intelligent agent optimization is shown in * As shown in Figure 4 ; the final operation effect sum-reward is shown in Figure 5 , it can be seen that the training trend converges finally and upward.

[0065] Table 1 initialization Q table

[0066]

[0067]

[0068] Table 2 Q table after iteration

[0069]

[0070] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should all be within the protection scope of the present application.

Claims

1. A method for power grid outage maintenance scheduling optimization based on Q-learning, characterized in that: The method comprises the steps of: Generating a corresponding Gantt chart by obtaining a power grid original topology file, and establishing a state judgment rule and an objective function of current agent reinforcement learning; Integrating subordinate unit maintenance plan mapping into the Gantt chart to form an initial maintenance plan; Using a built-in power flow solver of matlab to calculate an initial power flow set to evaluate an initial operation state of the power grid; In the double-loop nested process of agent training, the maintenance action is selected by the epsilon-greedy method, and the Gantt chart and topology file are updated, and the Q value is updated by using the Bellman equation to optimize the maintenance plan and improve the safety and efficiency of the power grid operation; After the agent training is completed, the Gantt chart corresponding to the final outage maintenance plan is output to ensure the efficiency and reliability of the power grid dispatching; In the double-loop nested process of the agent training, an action reward value mapping function is set , the reward value is calculated according to the initial power flow set , the power-off plan power flow set and the action value mapping rule reward , the running state is covered to the running state , and the Q value is calculated by using the Bellman equation; Action reward value mapping function The formula is: wherein, PF m is the power flow collection after the action of the agent, is the per-unit value of the node voltage in the current power flow collection, is the active power loss of the transmission line in the current power flow collection, is the power flow convergence flag of each day in the power flow collection, with a value of 1 representing convergence and a value of 0 representing non-convergence; The reward value reward is calculated by the formula: ; The Q value formula calculated by the Bellman equation is: wherein, is the update to the Q-table after performing the current action row column corresponding to the action index, is the learning rate, is the discount factor, is the maximum Q-value in the set of meaningful actions in the Q-table row .

2. The power grid outage maintenance scheduling optimization method based on Q learning according to claim 1, characterized in that: After the initial maintenance plan is formed and before the initial power flow set is calculated, a Q table is initialized to store the expected reward value of each power grid operation state and maintenance action pair, providing a starting point for the agent to continuously update and optimize through learning in the subsequent training process, and the process is as follows: Firstly, according to the number of outage maintenance equipment in the current month, and combining the N-k principle in the power system, the state space dimension of the Q table is determined; secondly, the action space is defined, and a two-dimensional array is created as the Q table, with the number of rows corresponding to the dimension of the state space and the number of columns corresponding to the dimension of the action space; finally, all entries in the Q table are initialized to zero or a small decimal value close to zero, which is gradually learned and adjusted through subsequent reinforcement learning process.

3. The power grid outage maintenance scheduling optimization method based on Q learning according to claim 1, characterized in that: In the double-loop nested process of agent training, the agent will continue to perform the training loop until one of the following conditions is met: the maximum number of training rounds is reached or the maximum number of steps in any training round is reached, wherein, If the agent performs the maximum number of steps in any training round but does not meet the optimization target or does not reach the expected performance indicator, the current training process is considered incomplete and needs to continue; otherwise, If the agent completes all the preset steps and meets the optimization target or performance indicator in any training round, the current training process is considered complete, and after the training is completed, the agent outputs the two-dimensional Gantt chart corresponding to the final outage maintenance plan and the total reward value of all steps in each training round as a basis for subsequent evaluation of the training effect.

4. The power grid outage maintenance scheduling optimization method based on Q learning according to claim 1, characterized in that: In establishing the state judgment rule of the current intelligent agent reinforcement learning, firstly, the number of transmission lines branch and the number of generators gen are determined through the original power grid topology file, the total number of power grid repairable equipment col is calculated, and the data basis is provided for the state judgment rule, and then the single rule c1 is used to form the state judgment rule C, Wherein, the single rule c1 ensures that the number of power grid outage repair equipment in any given day does not exceed the total number of power grid repairable equipment col minus the maximum number of equipment in operation on the day , row is the number of days, is the number of equipment in operation on the day.

5. The power grid outage maintenance scheduling optimization method based on Q learning according to claim 1 or 4, characterized in that: On the basis of establishing the state judgment rule C of agent reinforcement learning, the objective function F is established by considering the minimum voltage deviation of the power grid nodes, the minimum active power loss of the power transmission line in the whole grid, and the non-convergence result of the power flow. PF = PF + PF n is the power flow collection after optimization by the agent, is the per-unit value of the node voltage in the current power flow collection, is the active power loss of the transmission line in the current power flow collection, is the power flow convergence flag of each day in the power flow collection, with a value of 1 representing convergence and a value of 0 representing non-convergence.

6. The power grid outage maintenance scheduling optimization method based on Q-learning according to claim 1 or 4, characterized in that: The initial operation state of the power grid is an operation state of the power grid when implementing an initial power-off maintenance plan The operation state is determined by a state judgment rule C, In the formula, is a current Gantt chart; and In the process of entering the double-loop nested process of agent training, the running state is taken as the starting point of the agent training process. After selecting the maintenance action through the epsilon-greedy method and updating the Gantt chart and the topology file, the state determination rule C is used to determine the running state of the power grid after optimization , reflecting the influence of the maintenance action on the running state of the power grid, achieving better voltage level, lower network loss, and higher power flow convergence.

7. The power grid outage maintenance scheduling optimization method based on Q-learning according to claim 6, characterized in that: Determine the operating status The specific process is: First, from the action collection Select a meaningful action based on the epsilon-greedy method , modify the maintenance plan of subordinate units ; Secondly, modify the Gantt chart according to the selected action , according to the Gantt chart Modify the topology file used for calculation; again, use the built-in power flow solver in MATLAB to calculate the power outage plan power flow collection after the action Finally, according to the judgment rules Determine the current state , among which, action collection It is determined based on the maintenance plans submitted by subordinate units and the operating conditions of the power grid.

Images