Power distribution network maintenance plan setting method and system based on multi-target reinforcement learning

Abstract

The invention provides a power distribution network maintenance plan setting method and system based on multi-objective reinforcement learning. Comprising the following steps: carrying out merging and mutual exclusion judgment on initial plans by utilizing a search method; a maintenance plan adjustment mathematical model including multiple targets such as power distribution network maintenance plan economy and task load is established, and meanwhile possible construction period duration fluctuation is considered; converting the mathematical model into a constrained Markov decision process, defining a maintenance state, a maintenance action, a maintenance plan arrangement reward function and a constraint space, and training and solving a plan arrangement agent by using noise depth Q reinforcement learning; the validity of the method is verified by using a certain maintenance plan data. Through a search method and multi-target noise depth Q reinforcement learning, automatic combination and plan scheduling of the power distribution network maintenance plan can be carried out, the rationality and economy of the maintenance plan are improved, and the intelligence of power distribution network decision making is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a method and system for setting a maintenance plan for a distribution network based on multi-objective reinforcement learning, which is particularly suitable for intelligent scheduling of maintenance plans with potentially fluctuating construction periods. Background Art

[0002] The maintenance of distribution networks is a key and basic link to ensure the reliable operation of power equipment and support the safe and stable operation of the system. A reasonable maintenance plan can not only improve the reliability of power supply and ensure the stable operation of the system, but also further shorten the area and duration of power outages and improve the overall economic efficiency of maintenance work.

[0003] At present, my country mainly adopts the method of planned maintenance. This periodic and regular maintenance method has greatly improved the system operation level. However, with the construction and development of new power systems, the "weather-dependent" distributed renewable energy power generation, "unpredictable" disasters and accidents, and changes in construction schedules caused by construction technology have made the maintenance arrangements of distribution networks face more prominent and complex challenges, and are prone to deviate from pre-established maintenance plans. For example, in 2018, the live-fire connection operation unit determined by a power supply company through a framework tender could not temporarily arrange personnel for operation and could not meet the user's requirements for time limits. Therefore, for the maintenance plan formulation department, it is urgent to study the intelligent scheduling method of distribution network maintenance plans containing uncertain decisions.

[0004] In addition, the power grid company lacks an information platform for maintenance work, which makes it difficult for various planning departments to obtain support, and it is difficult to reasonably arrange and adjust projects such as simultaneous maintenance and non-simultaneous maintenance. At the same time, the arrangement of maintenance plans is completed manually. In practice, there are problems such as complex data, lack of constraints, and inefficient processes faced by manual submission, manual arrangement, and manual coordination, which are difficult to adapt to the needs of actual business scenario development. In recent years, with the increasing size of distribution network maintenance business, the above problems have become more serious. For this reason, it is urgent to explore more intelligent maintenance plan arrangement methods. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for setting a maintenance plan for a distribution network based on multi-objective reinforcement learning, which efficiently and intelligently arranges maintenance plans, optimizes maintenance resource allocation, and improves system operation reliability while taking into account fluctuations in construction periods.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] The distribution network maintenance plan setting method based on multi-objective reinforcement learning includes:

[0008] Perform merging and mutual exclusion operations on the acquired original maintenance plan;

[0009] By setting the objective function and constraints, a mathematical model for setting the monthly maintenance plan of the distribution network is constructed; the constraints include the initial maintenance plan with merging operations and mutually exclusive operations;

[0010] The mathematical model is modeled by CMDP by defining the state space, action space, reward function, transfer probability, and discount factor of the mathematical model;

[0011] The mathematical model after CMDP modeling is solved by using the noise deep Q reinforcement learning algorithm to obtain the target model for setting the distribution network maintenance plan.

[0012] Use the test data to test the target model;

[0013] The tested target model is used to set a maintenance plan for the distribution network, and the maintenance plan for the distribution network is used for the maintenance work of the distribution network.

[0014] Preferably, the rules for the merge operation include:

[0015] When two maintenance plans involve exactly the same equipment, the equipment involved should be uniformly arranged to be maintained in the same period;

[0016] For the same substation, if the maintenance plan of the high-voltage main transformer and the connected busbar and peripheral equipment is within the same month, the maintenance of the substation should be carried out in the same period;

[0017] When the maintenance of two devices involved in the maintenance of distribution network equipment will cause power outage at the same load point, the two devices should be scheduled for maintenance in the same time period;

[0018] The rules for mutually exclusive operations include:

[0019] If two 220kV lines serve as the incoming power supply for a substation, avoid arranging maintenance of the two lines at the same time period;

[0020] In the same substation, avoid arranging maintenance of two buses with the same voltage level at the same time;

[0021] Avoid arranging two main transformers for maintenance at the same time in the same substation and voltage level;

[0022] For equipment that may cause electrical islands, the maintenance time should be staggered.

[0023] Preferably, the objective function includes the overall economy of the maintenance plan, the balance of the daily task volume of the maintenance plan, and the relative adjustment amount of the initial plan.

[0024] The process of setting the objective function includes:

[0025] Determine the distribution network maintenance plan based on the health status of the equipment, power outage losses and fault losses;

[0026] Based on cost-effectiveness, resource allocation, and practicality, the economic cost of the maintenance plan, the relative adjustment of the initial maintenance plan, and the balance of the daily task volume of the maintenance plan are respectively corresponded, and the formula is used to calculate the maintenance plan.

[0027] min F1,F2,F3

[0028] Construct the objective function; where F1 is the economic cost of the monthly maintenance plan, F2 is the relative adjustment of the initial plan, and F3 is the variance of the monthly task volume;

[0029] For the monthly maintenance plan of the distribution network, the formula

[0030]

[0031] Calculate the economic cost as the daily cost C of the scheduled maintenance plan i The sum of; where C i is the cost of the i-th day of the month, and T is the total number of days;

[0032] Pass-through

[0033]

[0034] The variance of daily task volume is used to represent the balance of daily task volume of the monthly maintenance plan; where x i is the task amount for the i-th day, is the average monthly task volume, X total The total workload planned for monthly maintenance;

[0035] For N maintenance plans of the distribution network, the formula

[0036]

[0037] Calculate the relative adjustment of the initial plan; where: is the scheduled start time of the nth plan, Article n plans the start time of the original plan;

[0038] Constraints include non-simultaneous maintenance constraints, simultaneous maintenance constraints, date immutability constraints, maintenance sequence constraints, and daily task volume constraints. The process of setting constraints specifically includes:

[0039] Pass-through

[0040] x i,A ·x i,B =0

[0041] Calculate the constraint that cannot be repaired simultaneously; where x i,A is the maintenance task of plan A on day i, x i,B The maintenance task for plan B on day i;

[0042] Pass-through

[0043] t A,begin =t B,begin

[0044] Calculate the simultaneous maintenance constraints; where t A,begin is the start time of plan A; t B,begin The starting time of Plan B;

[0045] Pass-through

[0046]

[0047] The calculation date cannot be changed; where t A,begin The scheduled start time for Plan A; The starting time before the A plan is arranged;

[0048] Pass-through

[0049] t A,begin >t B,begin +x B

[0050] Calculate the maintenance sequence constraints; where t A,begin is the start time of plan A; t B,begin is the start time of plan B, x B This is the construction period of Plan B.

[0051] Pass-through

[0052] x i ≤x max

[0053] Calculate the daily task volume constraint; where x max The maximum daily task volume.

[0054] Preferably, the process of constructing the mathematical model also includes setting a cumulative probability distribution function of the construction period fluctuation, specifically including:

[0055] Combined with the specific construction type, historical weather patterns and construction environment characteristics, according to historical data analysis and probability statistics, the formula

[0056] F(d)=P(T delay ≤d)

[0057] The cumulative probability distribution model of maintenance plan time extension is constructed, and the cumulative probability distribution function is obtained; where F(d) is the probability that the extension time is less than or equal to a specific number of days, T delay is the time difference between the actual completion time of the maintenance plan and the completion time of the initial maintenance plan, and d is the specific extension time of the maintenance plan;

[0058] According to the probability distribution of delay time,

[0059]

[0060] Calculate the duration value with duration fluctuation; where, is the uncertain duration of the nth maintenance plan, is the original maintenance plan duration of the nth maintenance plan, △p n The extension time for the nth plan.

[0061] Preferably, by defining the state space, action space, state transition probability, discount factor, reward function, and constraint conditions of the mathematical model, the process of performing CMDP modeling on the mathematical model specifically includes:

[0062] Pass-through

[0063]

[0064] Define the state space;

[0065] Pass-through

[0066] a=(t 1,begin ,t 2,begin ,…,t n,begin ),t n,begin ≤t max

[0067] Define the action space;

[0068] The state transition probability and discount factor are defined by interacting with the maintenance plan orchestration environment.

[0069] Pass-through

[0070]

[0071] Define the reward function; where R is the reward and punishment function for the orchestration agent, To arrange the reward function for the nth maintenance plan, is the penalty function for scheduling the nth maintenance plan;

[0072] Pass-through

[0073]

[0074] Obtain reward values ​​based on multiple objective functions; where: λ1 is the scaling factor of the economic cost function, λ2 is the scaling factor of the daily task volume balance function, and λ3 is the scaling factor of the initial plan relative adjustment function.

[0075] Preferably, the penalty value is given by

[0076]

[0077] Calculated; where P task is the penalty for exceeding the daily task limit, P conflict is the penalty for violating the simultaneous maintenance constraint, P infeasible is the penalty for violating the non-simultaneous maintenance constraint, P immutable is the penalty for violating the immutable constraint, P order is the penalty for violating the maintenance sequence constraint, and λ4~λ8 are the scaling coefficients of the constraint penalty function.

[0078] Preferably, the process of solving the mathematical model after CMDP modeling by using the noise deep Q reinforcement learning algorithm and obtaining the target model for setting the distribution network maintenance plan specifically includes:

[0079] Determine the noise deep Q reinforcement learning algorithm by defining the actor network, target network, loss function, and experience replay;

[0080] Pass-through

[0081]

[0082] Define the loss function of noisy deep Q reinforcement learning; where θ is the parameter of the Q network, θ - is the target Q network weight, Q(s,a;θ) is the action value function estimated by the Q network based on θ under the current state and action, r is the immediate reward obtained after executing action a under state s, s' is the next state after executing the action, γ is the discount factor, and max a′ Q target (s′,a′;θ - ) is the maximum Q value given by the target Q network for all possible actions a' in the next state s', and E is the mathematical expectation; ∈ and ∈ - are the noise terms of the Q network and the target network respectively, ∈ is the noise that obeys the normal distribution to enhance the exploratory nature of the network;

[0083] Construct a replay storage area, store the four-tuple data sampled from the environment each time into the replay buffer, and perform defined experience replay;

[0084] The four-tuple data includes state, action, reward and next state.

[0085] In a second aspect, the present invention provides a distribution network maintenance plan setting system based on multi-objective reinforcement learning, comprising:

[0086] Analytical computing modules for:

[0087] Perform merging and mutual exclusion operations on the acquired initial maintenance plan;

[0088] By setting the objective function and constraints, a mathematical model for setting the monthly maintenance plan of the distribution network is constructed; the constraints include the initial maintenance plan with merging operations and mutually exclusive operations;

[0089] The mathematical model is modeled by CMDP by defining the state space, action space and reward function of the mathematical model;

[0090] The mathematical model after CMDP modeling is solved by using the noise deep Q reinforcement learning algorithm to obtain the target model for setting the distribution network maintenance plan.

[0091] The test module is used to test the target model using test data;

[0092] The maintenance plan output module is used to: use the target model after the test to set the maintenance plan of the distribution network and output the maintenance plan.

[0093] Preferably, the objective function set by the analysis and calculation module includes the balance of the daily task volume of the maintenance plan, the relative adjustment amount of the initial maintenance plan and the economic cost of the maintenance power outage;

[0094] The process of analyzing the objective function set by the calculation module includes:

[0095] Determine the maintenance plan based on the health status of the equipment, power outage losses and failure losses;

[0096] Based on cost-effectiveness, resource allocation, and practicality, the economic cost of the maintenance plan, the relative adjustment of the initial maintenance plan, and the balance of the daily task volume of the maintenance plan are respectively corresponded, and the formula is used to calculate the maintenance plan.

[0097] min F1,F2,F3

[0098] Construct the objective function; where F1 is the monthly economic cost, F2 is the relative adjustment of the initial plan, and F3 is the variance of the monthly task volume;

[0099] For the monthly maintenance plan of the distribution network, the formula

[0100]

[0101] Calculate the economic cost as the daily cost C of the scheduled maintenance plan iThe sum of; where C i is the cost on the i-th day of this month, T is the total number of days;

[0102] Pass-through

[0103]

[0104] The variance of daily task volume is used to represent the balance of daily task volume of the monthly maintenance plan; where x i is the task amount for the i-th day, is the average monthly task volume, X total The total workload planned for monthly maintenance;

[0105] For N maintenance plans of the distribution network, the formula

[0106]

[0107] Calculate the relative adjustment of the initial plan; where: is the scheduled start time of the nth plan, Article n plans the start time of the original plan;

[0108] Constraints include non-simultaneous maintenance constraints, simultaneous maintenance constraints, date immutability constraints, maintenance sequence constraints, and daily task volume constraints. The process of setting constraints specifically includes:

[0109] Pass-through

[0110] x i,A ·x i,B =0

[0111] Calculate the constraint that cannot be repaired simultaneously; where x i,A is the maintenance task of plan A on day i, x i,B The maintenance task for plan B on day i;

[0112] Pass-through

[0113] t A,begin =t B,begin

[0114] Calculate the simultaneous maintenance constraints; where t A,begin is the start time of plan A; t B,begin The starting time of Plan B;

[0115] Pass-through

[0116]

[0117] The calculation date cannot be changed; where t A,begin The scheduled start time for Plan A; The starting time before the A plan is arranged;

[0118] Pass-through

[0119] t A,begin >t B,begin +x B

[0120] Calculate the maintenance sequence constraints; where t A,begin is the start time of plan A; t B,begin is the start time of plan B, x B This is the construction period of Plan B.

[0121] Pass-through

[0122] x i ≤x max

[0123] Calculate the daily task volume constraint; where x max The maximum daily task volume.

[0124] Preferably, the process of the test module using the test data to test the target model includes:

[0125] Acquire multiple groups of historical maintenance plans and input them into the analysis target model in sequence, and use the output of the target model to reversely update the target model;

[0126] The maintenance plan output module uses the tested target model to set the monthly maintenance plan for the distribution network, and the process of outputting the monthly maintenance plan includes:

[0127] Obtain target data and input it into the target model after testing to calculate the monthly maintenance plan of the distribution network;

[0128] Visualize and / or print out the monthly maintenance plan for the distribution network.

[0129] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention provides a method for setting a distribution network maintenance plan based on a search method and multi-objective reinforcement learning. It includes: using a search method to carry out the merging and mutual exclusion judgment of the initial plan; establishing a mathematical model for adjusting the maintenance plan including multiple objectives such as the economy of the distribution network maintenance plan and the amount of tasks, while taking into account possible fluctuations in the duration of the construction period; converting the mathematical model into a constrained Markov decision process, defining the maintenance status, maintenance action, maintenance plan scheduling reward function and constraint space, and using a noise deep Q reinforcement learning agent to train and solve the plan scheduling agent; using a certain maintenance plan data to verify the effectiveness of the proposed method. Through the search method and multi-objective noise deep Q reinforcement learning, the present invention can automatically merge and schedule the distribution network maintenance plan, improve the rationality and economy of the maintenance plan, and enhance the intelligence of the distribution network decision.

[0130] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0132] Figure 1 A flow chart of a method for setting a distribution network maintenance plan based on multi-objective reinforcement learning provided by the present invention;

[0133] Figure 2 A plan comparison diagram before and after adjustment without considering construction period fluctuations for the distribution network maintenance plan setting method based on multi-objective reinforcement learning provided by the present invention;

[0134] Figure 3 A comparison chart of daily task volume before and after adjustment without considering construction period fluctuations for the distribution network maintenance plan setting method based on multi-objective reinforcement learning provided by the present invention;

[0135] Figure 4 A comparison chart of the plans before and after considering the construction period fluctuation adjustment of the distribution network maintenance plan setting method based on multi-objective reinforcement learning provided by the present invention;

[0136] Figure 5 A schematic diagram showing the comparison of daily task volumes before and after adjustment considering construction period fluctuations in the method for setting a distribution network maintenance plan based on multi-objective reinforcement learning provided by the present invention;

[0137] Figure 6 A logical block diagram of a distribution network maintenance plan setting system based on multi-objective reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0138] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0139] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0140] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0141] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0142] The present invention provides a method for merging and arranging distribution network maintenance plans based on multi-objective reinforcement learning. The technical solution of the present invention includes the following concepts:

[0143] (1) Distribution network refers to the power network system that transmits electricity from the transmission system to the end user, usually including distribution transformers, distribution lines, distribution cabinets and distribution equipment. The main function of the distribution network is to convert high-voltage electricity into low-voltage electricity suitable for users and reliably transmit it to the user terminal.

[0144] (2) Maintenance plan refers to the plan for regular maintenance and inspection of various equipment in the distribution network in order to ensure the safe and stable operation of power equipment. Its contents include inspection items, inspection time, inspection methods, and arrangements for inspection personnel.

[0145] (3) Deep reinforcement learning is a technology that combines deep learning and reinforcement learning. It uses deep neural networks to approximate policy functions or value functions, thereby improving the decision-making ability of intelligent agents in complex environments. Its core idea is to continuously optimize strategies through interaction with the environment, so that the intelligent agent can choose the best action in a given state and achieve the goal of maximizing cumulative rewards.

[0146] (4) Schedule fluctuation refers to the phenomenon that the actual execution time of the maintenance plan is inconsistent with the scheduled time due to unforeseen factors (such as weather, equipment failure, human resource shortage, etc.) during the maintenance process. This increases the complexity and difficulty of maintenance planning.

[0147] (5) Multi-objective optimization is a method to optimize multiple conflicting objective functions simultaneously. In the scheduling of distribution network maintenance plans, it is usually necessary to consider multiple objectives such as the balance of daily maintenance plan tasks, the relative adjustment of the initial maintenance plan, and the economic cost of maintenance outages. By balancing these objectives, the optimal solution can be found.

[0148] (6) Constrained Markov decision process is a mathematical model used to describe the process of an intelligent agent making decisions in a random environment. CMDP includes state space, action space, state transition probability and reward function, and constraints. In the distribution network maintenance plan, CMDP is used to transform complex planning problems into sequential decision problems.

[0149] (7) Noise deep Q reinforcement learning is a deep reinforcement learning algorithm based on Q learning, which approximates the Q value function by using a deep neural network. Noise deep Q reinforcement learning can handle high-dimensional state spaces and is widely used in games, robot control and other fields. In the present invention, noise deep Q reinforcement learning is used to optimize maintenance planning.

[0150] In some prior art:

[0151] Chinese invention patent specification CN118153846A discloses a distribution network line fault maintenance method based on an auxiliary maintenance mechanism, which belongs to the field of distribution network fault maintenance, including: generating a repair work order based on the fault information of the distribution network line and uploading it to the dispatching center; the dispatching center analyzes the repair work order to determine the fault situation, and constructs a maintenance password work order, a tool retrieval work order and a personnel matching work order based on the fault situation; configuring a toolbox based on the tool retrieval work order and generating an extraction key for the corresponding toolbox based on the personnel matching work order; obtaining the toolbox based on the extraction key and executing the maintenance plan based on the maintenance password work order; determining whether to trigger the auxiliary maintenance mechanism based on the execution of the maintenance plan step by step, and re-formulating and executing the maintenance plan based on the expert consultant library in the auxiliary maintenance mechanism.

[0152] Chinese invention patent specification CN117709727A discloses a new distribution network operation risk assessment and active maintenance method based on data twins. The distribution network data is monitored by constructing a data twin model, and the data twin model is associated with the topological structure of the distribution network and the distribution network equipment information; based on the distribution network data, the target distribution network data related to the operation risk is first screened out, and then the target distribution network data is used to realize the operation risk assessment; then, based on the operation risk assessment results, the corresponding active maintenance strategy is given, so that relevant personnel can maintain the distribution network based on the active maintenance strategy.

[0153] Chinese invention patent specification CN117077933A discloses a method for intelligently compiling a distribution network power outage plan based on big data. Basic data is established based on the SqlServer database, and a web page interface is established based on the local area network system with a B / S architecture to realize the priority level of maintenance plan decision-making. A genetic algorithm is used to solve a variety of constraints, and the optimized power outage time is used as the input condition for load transfer path optimization. The radial characteristics of the power outage area network are used to solve the load transfer path optimization problem and obtain a transfer plan. The genetic algorithm is used for optimization with the goal of minimizing power outage load, the minimum number of switches, and the minimum network loss, and finally a power outage maintenance plan with the minimum power sales loss and the minimum total cost is obtained; a topological search method for electrical wiring diagrams is constructed.

[0154] Chinese invention patent specification CN115936332A discloses a method and system for dispatching and repairing a low-voltage distribution network without power outages, including: receiving a repair task and obtaining location information; formulating a route plan with the shortest time to reach the repair site, wherein the vehicles include a repair vehicle and a power generation vehicle; the selected vehicles and repair personnel rush to the repair site according to the formulated plan; connecting the power generation vehicle to the distribution network and disconnecting the mains at the same time, and the load is supplied with power by the power generation vehicle; the repair personnel repair the distribution network, and after the repair is completed, disconnect the power generation vehicle and connect the mains at the same time.

[0155] Chinese invention patent specification CN117875930A relates to a method for studying the state maintenance strategy of a source distribution network based on LCC theory. The improved Prim algorithm is introduced, and the breadth-first algorithm is used to search the undirected graph of the distribution network to search for the power circle. In order to solve the problem of overlapping power circles, sub-power circles are introduced and expanded and merged. The final power circle is all possible power supply paths of the DG; the particle swarm algorithm improved based on quantum mechanics theory is used to solve the maintenance decision; under the premise of considering the purchase cost of distributed power sources and environmental benefits, the maintenance time and maintenance method of the distribution network equipment are optimized with the goal of minimizing the maintenance cost and failure cost, and the maintenance plan of each equipment to be repaired during its entire life cycle is obtained to achieve the goal of minimizing the LCC of the equipment.

[0156] Chinese invention patent specification CN116644541A discloses a method for dynamic network reconstruction of a post-disaster distribution network taking into account fault maintenance, including a method for allocating maintenance materials, a method for formulating a fault maintenance plan, and a method for dynamic network reconstruction of a distribution network. Specifically, according to the fault situation of the distribution network, combined with the distance between the material warehouse and the fault location, with the goal of minimizing the material transportation time, maintenance materials are allocated to each fault; the fault maintenance plan formulation method, combined with the fault situation in the post-disaster distribution network, with the goal of minimizing the maintenance time, dispatches maintenance personnel and formulates a maintenance plan; the distribution network dynamic network reconstruction method, the distribution network updates the line switch status in combination with the maintenance plan, thereby realizing the dynamic network reconstruction of the distribution network and dynamically adjusting the network topology of the distribution network.

[0157] In their work practice, the applicant found that the main problems existing in the existing technology are: the arrangement of distribution network maintenance plans and the generation of method plans involve many factors such as power grid risks, source-load balance, and power supply reliability. Manually compiled maintenance plans and method adjustment plans face problems such as complex data analysis, low compilation efficiency, and great influence of subjective experience.

[0158] The present invention aims to construct an intelligent entity for maintenance planning and mode plan generation that takes into account multiple objective factors, so as to realize the intelligent and automatic adjustment of distribution network maintenance plan and operation mode. It comprehensively considers multiple objective factors such as operation rules, safety risks, maintenance task volume, etc., to generate a more comprehensive and flexible maintenance plan.

[0159] See also Figure 1 The present invention provides a method for setting a monthly maintenance plan for a distribution network based on multi-objective reinforcement learning, comprising the following steps:

[0160] Merge and mutually exclude the initial maintenance plans reported by each equipment operation and maintenance unit;

[0161] By setting the objective function and constraint conditions, a mathematical model for setting the monthly maintenance plan of the distribution network is constructed; the constraint conditions include the initial maintenance plan with the combined operation and the mutually exclusive operation;

[0162] By defining the state space, action space, reward function, transfer probability, and discount factor of the mathematical model, CMDP modeling is performed on the mathematical model;

[0163] By using a noise deep Q reinforcement learning algorithm, the mathematical model after CMDP modeling is solved to obtain a target model for setting a distribution network maintenance plan;

[0164] Testing the target model using test data;

[0165] The tested target model is used to set a maintenance plan for the distribution network, and the maintenance plan for the distribution network is used for maintenance work of the distribution network, mainly for monthly maintenance.

[0166] In a preferred embodiment provided by the present invention, the above-mentioned merge operation rules include:

[0167] (1) When two maintenance plans involve exactly the same equipment, these equipment should be scheduled for maintenance in the same time period. (2) For the same substation, if the maintenance plans for the high-voltage main transformer and its connected busbar and surrounding equipment are within the same month, the maintenance of these related equipment should be carried out in the same time period. (3) When the maintenance of distribution network equipment involves two equipment that will cause power outages at the same load point, these equipment should be scheduled for maintenance in the same time period to avoid repeated power outages.

[0168] The rules for the above mutually exclusive operations include:

[0169] (1) If two 220kV lines serve as the incoming power supply for a substation, the two lines should not be scheduled for maintenance at the same time to ensure the power supply safety of the substation. (2) In the same substation, two busbars of the same voltage level should not be scheduled for maintenance at the same time to prevent the risk of power outage in the entire station. (3) In the same substation and at the same voltage level, two main transformers should not be scheduled for maintenance at the same time. (4) For equipment that may cause electrical islands, their maintenance time should be staggered to avoid unnecessary power outages and ensure the safety and stability of the system.

[0170] After the merging and mutual exclusion operations, the data of the initial maintenance plan in matrix form is obtained and integrated into the subsequent process of setting constraints.

[0171] In a preferred embodiment provided by the present invention, the second step specifically includes the following execution process:

[0172] Determine the monthly maintenance plan based on the health status of the equipment, power outage losses, and failure losses.

[0173] The design is based on cost-effectiveness, resource allocation, and practicality, and the objective function is determined by the economic cost of the maintenance plan, the relative adjustment amount of the initial maintenance plan, and the balance of the daily task volume of the maintenance plan. Including:

[0174] min F1,F2,F3

[0175] Where F1 is the economic cost of the monthly maintenance plan, F2 is the relative adjustment of the initial plan, and F3 is the variance of the monthly task volume.

[0176] For the monthly maintenance plan, the total economic cost is the sum of the daily costs of the scheduled plan, that is:

[0177]

[0178] In the formula, C i is the cost on the i-th day of the month, and T is the total number of days.

[0179] The balance of daily tasks in the monthly maintenance plan is expressed by the variance of the monthly tasks:

[0180]

[0181] In the formula, x i is the task amount for the i-th day, is the average monthly task volume, X total Total workload planned for monthly maintenance

[0182] For N maintenance plans of the distribution network, the relative total adjustment of the initial plan is

[0183]

[0184] In the formula, is the scheduled start time of the nth plan, Article n plans the start time of the original plan;

[0185] The monthly maintenance plan constraints are determined based on the constraints of non-simultaneous maintenance, simultaneous maintenance, date immutability, maintenance sequence, and daily task volume. Including:

[0186] The constraints that cannot be repaired simultaneously are:

[0187] x i,A ·x i,B =0

[0188] In the formula, x i,A is the maintenance task of plan A on day i, xi,B is the maintenance task of plan B on day i.

[0189] At the same time, the maintenance constraints are:

[0190] t A,begin =t B,begin

[0191] Where, t A,begin is the start time of plan A; t B,begin The start time of Plan B.

[0192] The date cannot be changed:

[0193]

[0194] Where, t SA is the start time of plan A; t OA The original start time of Plan A.

[0195] The maintenance order constraints are:

[0196] t A,begin >t B,begin +x B

[0197] Where, t A,begin is the start time of plan A; t B,begin is the start time of plan B, x B This is the construction period of Plan B.

[0198] Daily task volume constraints:

[0199] x i ≤x max

[0200] In the formula, x max The maximum daily task volume.

[0201] In some feasible embodiments, the process of setting the mathematical model further includes: combining the specific construction type, historical weather patterns and construction environment characteristics, and constructing a cumulative probability distribution model of the maintenance time extension according to historical data analysis and probability statistics methods, and obtaining a cumulative probability distribution function as follows:

[0202] F(d)=P(T delay ≤d)

[0203] Where F(d) is the probability that the delay time is less than or equal to a certain number of days, T delay is the time difference between the actual completion time of the maintenance plan and the completion time of the initial maintenance plan, and d is the specific extension time of the maintenance plan.

[0204] According to the above delay time probability distribution, the final construction period is:

[0205]

[0206] In the formula, is the uncertain duration of the nth maintenance plan, is the original maintenance plan duration of the nth maintenance plan, △p n The extension time for the nth plan.

[0207] In the preferred embodiment provided by the present invention, the third step

[0208] The specific execution process includes the following:

[0209] Convert the mathematical model into a constrained Markov decision process and define the state space, action space and reward function, including:

[0210] Define the state space:

[0211]

[0212] Define the action space:

[0213] a=(t 1,begin ,t 2,begin ,…,t n,begin ),t n,begin ≤t max

[0214] Define the reward function. The reward function for the agent should be divided into two parts. One is the objective function, which is composed of the balance of the planned daily task volume, the relative adjustment of the initial plan, and the calculation of the economic cost. The other part is the constraint penalty, which is mainly the penalty for the agent violating the constraint. Add the reward function part and the constraint penalty part to get the reward function:

[0215]

[0216] Where: R(n) is the reward function obtained after the nth plan is arranged, R t (n) is the objective function reward obtained after the nth plan is arranged, R P (n) is the constraint penalty obtained after the nth plan is compiled.

[0217] On this basis, scaling by a certain proportion can obtain the objective function part as follows:

[0218] R t (n)=-λ var ·σ 2 (n)-λ t T(n)-λm ·M(n)

[0219] Where: var is the scaling factor of the planned daily task volume balance function, λ t is the scaling factor of the initial plan relative adjustment function, λ m is the scaling factor of the economic cost function.

[0220] The constraint penalty is:

[0221]

[0222] Where: P task is the penalty for exceeding the daily task limit, P conflict is the penalty for violating the simultaneous maintenance constraint, P infeasible is the penalty for violating the non-simultaneous maintenance constraint, P immutable is the penalty for violating the immutable constraint, P order is the penalty for violating the maintenance sequence constraint, and λ4~λ8 are the scaling coefficients of the constraint penalty function.

[0223] In a preferred embodiment provided by the present invention, the fourth step specifically includes the following execution process:

[0224] The maintenance scheduling problem is solved using the noisy deep Q reinforcement learning algorithm, including:

[0225] Define the actor network, target network, loss function, and experience replay part to determine the noisy deep Q reinforcement learning algorithm.

[0226] Define the loss function for noisy deep Q reinforcement learning:

[0227] L(θ+∈)=E[(r+γmax d Q target (s′,a′;θ - +∈ - )-Q(s,a;θ+∈)) 2 ]

[0228] Where θ is the parameter of the Q network, θ - is the target Q network weight, Q(s,a;θ) is the action value function estimated by the Q network based on θ under the current state and action, r is the immediate reward obtained after executing action a under state s, s' is the next state after executing the action, γ is the discount factor, and max a′ Q target (s′,a′;θ - ) is the maximum Q value given by the target Q network for all possible actions a' in the next state s'. ∈ and ∈ -are the noise terms of the Q network and the target network respectively, ∈ is the noise that obeys the normal distribution to enhance the exploratory nature of the network, and E is the mathematical expectation, that is, the average value.

[0229] Define experience replay, build a replay storage area, and store the four-tuple data (state, action, reward, next state) sampled from the environment each time into the replay buffer.

[0230] In some feasible embodiments, the process of testing the target model includes:

[0231] The algorithm is programmed in Python and tested on a computer with an AMD Ryzen 7 6800H@3.20GHz CPU, an NVIDlA GeForceRTX 3070Ti Laptop GPU, 32GB RAM, and Windows 11 operating system.

[0232] In some feasible embodiments, in order to verify the effectiveness of the proposed method, the examples are divided into four schemes to calculate the multi-objective function values:

[0233] Solution 1: Initial maintenance plan without considering construction period fluctuations;

[0234] Solution 2: Maintenance scheduling plan without considering construction period fluctuations;

[0235] Option 3: Initial maintenance plan considering construction period fluctuations;

[0236] Solution 4: Plan after maintenance scheduling taking into account fluctuations in construction period.

[0237] (1) Scheduling results without considering construction period fluctuations

[0238] Figure 2 This is a Gantt chart comparing the scheduling results before and after not considering the fluctuation of the construction period. Under the scheduling of reinforcement learning, the No. 5 and No. 6 plans that should be overhauled at the same time will start maintenance on the 16th. The original No. 11 plan conflicted with the No. 12 plan, but after the reinforcement learning scheduling, they no longer conflicted. In addition, the conflicting plans No. 7 and No. 8 were also scheduled for maintenance at different times. At the same time, reinforcement learning also minimizes the adjustment time of each plan to reduce the impact on other work of the power grid. According to the comparison of daily task volume before and after the adjustment (such as Figure 3 ) It can be seen that the overall planning schedule has become more balanced. The balance of the planned daily task volume before and after the adjustment is calculated: the balance of the planned daily task volume before the adjustment is 0.676, and the balance of the planned daily task volume after the adjustment is 0.225. This result shows that the imbalance of the daily task volume after the reinforcement learning adjustment is significantly lower than that of the initial plan, and the scheduling of the maintenance plan is more reasonable.

[0239] (2) Scheduling results considering construction period fluctuations

[0240] Figure 4 This is a Gantt chart comparing the training schedule before and after considering the fluctuation of the construction period. The same as before is that the No. 5 and No. 6 plans that should be overhauled at the same time still start the overhaul on the 16th, and the conflicting No. 11, No. 12 and No. 7, No. 8 plans no longer conflict. However, compared with the case where uncertainty is not considered ( Figure 2 )The difference is: since Plan 8 has an uncertain duration, if Plan 7 is still scheduled for maintenance after Plan 8, there will be a possibility that conflicting plans will be maintained at the same time. Therefore, the agent chooses to have Plan 7 maintained in advance and Plan 8 delayed, thus solving the problem of fluctuations in maintenance duration.

[0241] According to the comparison of daily task volume before and after adjustment (such as Figure 5 ) It can be seen that, considering the fluctuation of the construction period, the overall planning schedule has also become more balanced. The balance of the planned daily task volume before and after the adjustment is calculated: the balance of the planned daily task volume before the adjustment is 0.733, and the balance of the planned daily task volume after the adjustment is 0.345. This shows that the uncertainty scheduling plan solved by the noisy deep Q reinforcement learning algorithm has a significantly lower daily task volume imbalance than the initial plan, and the scheduling result is more reasonable.

[0242] In a second aspect, the present invention provides a distribution network maintenance plan setting system based on multi-objective reinforcement learning for performing distribution network maintenance plan scheduling by implementing the above method, such as Figure 6 As shown, it includes:

[0243] The analysis and calculation module 601 is used for:

[0244] Merge and mutually exclude the initial maintenance plans reported by each equipment operation and maintenance unit;

[0245] By setting the objective function and constraint conditions, a mathematical model for setting the monthly maintenance plan of the distribution network is constructed; the constraint conditions include the initial maintenance plan with the combined operation and the mutually exclusive operation;

[0246] By defining the state space, action space, reward function, transfer probability, and discount factor of the mathematical model, CMDP modeling is performed on the mathematical model;

[0247] By using a noise deep Q reinforcement learning algorithm, the mathematical model after CMDP modeling is solved to obtain a target model for setting a distribution network maintenance plan;

[0248] A testing module 602 is used to test the target model using test data;

[0249] The maintenance plan output module 603 is used to set a monthly maintenance plan for the distribution network using the tested target model and output the monthly maintenance plan.

[0250] In some preferred embodiments, the process of the testing module 602 using the test data to test the target model includes:

[0251] A plurality of historical maintenance plans are obtained and sequentially input into the analysis target model, and the target model is reversely updated using the output of the target model. The method of reversely updating the target model can be set according to the prior art, which will not be described in detail here.

[0252] The maintenance plan output module 603 uses the tested target model to set the monthly maintenance plan of the distribution network, and the process of outputting the monthly maintenance plan includes:

[0253] Obtaining target data and inputting it into the target model after testing, and calculating and obtaining a monthly maintenance plan for the distribution network;

[0254] The monthly maintenance plan of the distribution network is outputted visually and / or printed out. For example, a display device with a touch screen and a printer can be provided.

[0255] In summary, the present invention provides a method and system for setting a distribution network maintenance plan based on multi-objective reinforcement learning, which effectively solves the problem of model solving difficulties caused by fluctuations in maintenance schedules. The results of the example show that the model can not only optimize the allocation of tasks and achieve a balance in tasks, but also reduce economic costs while ensuring the safety of power grid operation, while minimizing the relative adjustment of the initial plan. This proves that the model and method proposed in this article can better realize the adaptive scheduling of maintenance schedule fluctuations, and provide theoretical and technical support for scheduling departments and personnel to arrange actual power grid maintenance plans.

[0256] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0257] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0258] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0259] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A distribution network maintenance plan setting method based on multi-objective reinforcement learning, characterized in that: include: Perform merging and mutual exclusion operations on the acquired original maintenance plan; By setting the objective function and constraint conditions, a mathematical model for setting the monthly maintenance plan of the distribution network is constructed; the constraint conditions include the initial maintenance plan with the combined operation and the mutually exclusive operation; By defining the state space, action space, reward function, transfer probability, and discount factor of the mathematical model, CMDP modeling is performed on the mathematical model; By using a noise deep Q reinforcement learning algorithm, the mathematical model after CMDP modeling is solved to obtain a target model for setting a distribution network maintenance plan; Testing the target model using test data; The tested target model is used to set a maintenance plan for the distribution network, and the maintenance plan for the distribution network is used for maintenance work of the distribution network.

2. The method according to claim 1, characterized in that The rules of the merge operation include: When two maintenance plans involve exactly the same equipment, the equipment involved should be uniformly arranged to be maintained in the same period; For the same substation, if the maintenance plan of the high-voltage main transformer and the connected busbar and peripheral equipment is within the same month, the maintenance of the substation should be carried out in the same period; When the maintenance of two devices involved in the maintenance of distribution network equipment will cause power outage at the same load point, the two devices should be scheduled for maintenance in the same time period; The rules of the mutually exclusive operation include: If two 220kV lines serve as the incoming power supply for a substation, avoid arranging maintenance of the two lines at the same time period; In the same substation, avoid arranging maintenance of two buses with the same voltage level at the same time; Avoid arranging two main transformers for maintenance at the same time in the same substation and voltage level; For equipment that may cause electrical islands, the maintenance time should be staggered.

3. The method according to claim 2, characterized in that The objective function includes the overall economy of the maintenance plan, the balance of the daily task volume of the maintenance plan, and the relative adjustment amount of the initial plan. The process of setting the objective function includes: Determine the distribution network maintenance plan based on the health status of the equipment, power outage losses and fault losses; Based on cost-effectiveness, resource allocation, and practicality, the economic cost of the maintenance plan, the relative adjustment of the initial maintenance plan, and the balance of the daily task volume of the maintenance plan are respectively corresponded, and the formula is used to calculate the maintenance plan. min F1,F2,F3 Construct the objective function; where F1 is the economic cost of the monthly maintenance plan, F2 is the relative adjustment of the initial plan, and F3 is the variance of the monthly task volume; For the monthly maintenance plan of the distribution network, the formula Calculate the economic cost as the daily cost C of the scheduled maintenance plan i The sum of; where C i is the cost of the i-th day of the month, and T is the total number of days; Pass-through The variance of daily task volume is used to represent the balance of daily task volume of the monthly maintenance plan; where x i is the task amount for the i-th day, is the average monthly task volume, X total The total workload planned for monthly maintenance; For N maintenance plans of the distribution network, the formula Calculate the relative adjustment of the initial plan; where: is the scheduled start time of the nth plan, Article n plans the start time of the original plan; The constraint conditions include non-simultaneous maintenance constraint, simultaneous maintenance constraint, date non-changeable constraint, maintenance sequence constraint, and daily task quantity constraint. The process of setting the constraint conditions specifically includes: Pass-through x i,A ·x i,B =0 Calculate the constraint that cannot be repaired simultaneously; where x i,A is the maintenance task of plan A on day i, x i,B The maintenance task for plan B on day i; Pass-through t A,begin =t B,begin Calculate the simultaneous maintenance constraints; where t A,begin is the start time of plan A; t B,begin The starting time of Plan B; Pass-through The calculation date cannot be changed; where t A,begin The scheduled start time for Plan A; The starting time before the A plan is arranged; Pass-through t A,begin >t B,begin +x B Calculate the maintenance sequence constraints; where t A,begin is the start time of plan A; t B,begin is the start time of plan B, x B This is the construction period of Plan B. Pass-through x i ≤x max Calculate the daily task volume constraint; where x max The maximum daily task volume.

4. The method according to claim 3, characterized in that The process of constructing the mathematical model also includes setting a cumulative probability distribution function of the construction period fluctuation, specifically including: Combined with the specific construction type, historical weather patterns and construction environment characteristics, according to historical data analysis and probability statistics, the formula F(d)=P(T delay ≤d) Construct a cumulative probability distribution model for maintenance plan time extension and obtain the cumulative probability distribution function; where F(d) is the probability that the extension time is less than or equal to a specific number of days, T delay is the time difference between the actual completion time of the maintenance plan and the completion time of the initial maintenance plan, and d is the specific extension time of the maintenance plan; According to the delay time probability distribution, through the formula Calculate the duration value with duration fluctuation; where, is the uncertain duration of the nth maintenance plan, is the original maintenance plan duration of the nth maintenance plan, △p n The extension time for the nth plan.

5. The method according to claim 4, characterized in that The process of performing CMDP modeling on the mathematical model by defining the state space, action space, state transition probability, discount factor, reward function, and constraint conditions of the mathematical model specifically includes: Pass-through Define the state space; Pass-through a=(t 1,begin ,t 2,begin ,…,t n,begin ),t n,begin ≤t max Define the action space; The state transition probability and discount factor are defined by interacting with the maintenance plan orchestration environment. Pass-through Define the reward function; where R is the reward and punishment function for the orchestration agent, To arrange the reward function for the nth maintenance plan, is the penalty function for scheduling the nth maintenance plan; Pass-through Obtain reward values ​​based on multiple objective functions; where: λ1 is the scaling factor of the economic cost function, λ2 is the scaling factor of the daily task volume balance function, and λ3 is the scaling factor of the initial plan relative adjustment function.

6. The method according to claim 5, characterized in that The penalty value is expressed as Calculated; where P task is the penalty for exceeding the daily task limit, P conflict is the penalty for violating the simultaneous maintenance constraint, P infeasible is the penalty for violating the non-simultaneous maintenance constraint, P immutable is the penalty for violating the immutable constraint, P order is the penalty for violating the maintenance sequence constraint, and λ4~λ8 are the scaling coefficients of the constraint penalty function.

7. The method according to claim 6, characterized in that The process of solving the mathematical model after CMDP modeling by using the noise deep Q reinforcement learning algorithm to obtain the target model for setting the distribution network maintenance plan specifically includes: The noisy deep Q reinforcement learning algorithm is determined by defining the actor network, target network, loss function, and experience replay.

Images