Primary and secondary synchronization planning optimization method for distribution network based on reinforcement learning

Abstract

The present application relates to a kind of based on reinforcement learning primary-secondary synchronous planning optimization method of distribution network, comprising the following steps: S1: the original planning data of target area distribution network is obtained, and preprocessing is carried out, obtains standardization planning data set;S2: the digital twin environment of primary-secondary coupling distribution network is built;S3: according to the operation index output by digital twin environment, the state of reinforcement learning intelligent agent in planning stage is obtained, and graph embedding feature is introduced, and joint state is constructed;S4: establish hierarchical reinforcement learning model;S5: the joint planning action is input into digital twin environment, executes joint planning action and calculates multi-objective reward value;S6: state, joint planning action, reward value, next state and constraint violation information are used as training sample, and optimization model is trained and primary-secondary synchronous planning result is output.The present application effectively improves the efficiency and reliability of primary-secondary synchronous planning of distribution network.

Description

Technical Field

[0001] This invention relates to the field of power distribution planning, and in particular to a method for optimizing primary and secondary synchronization planning of power distribution networks based on reinforcement learning. Background Technology

[0002] With the large-scale integration of distributed photovoltaic, wind power, energy storage, electric vehicles, and flexible loads into distribution networks, the power flow direction, load level, and fault current level of distribution networks exhibit significant fluctuations and uncertainties. Traditional distribution network planning models, primarily based on unidirectional power supply, are no longer adequate to meet operational requirements such as multi-source integration, bidirectional power flow, active control, and rapid fault recovery. Therefore, it is necessary to expand, upgrade, and optimize the configuration of primary equipment such as substations, distribution transformers, feeders, tie lines, switchgear, and reactive power compensation equipment.

[0003] Meanwhile, secondary systems such as distribution automation, relay protection, communication networks, and edge computing are crucial foundations for achieving fault location, isolation, power restoration, status monitoring, and distributed resource collaborative control. The configuration of secondary equipment typically depends on the primary network topology and switch placement, while the latency, bandwidth, and reliability of communication links directly affect the actual effectiveness of protection actions, remote control, and feeder self-healing functions. When the primary network is expanded, restructured, or distributed power supply access changes, the original secondary terminal locations, communication networks, and protection settings may no longer be suitable.

[0004] In existing technologies, primary equipment planning and secondary equipment planning are typically implemented independently or sequentially. This involves first determining the primary network structure and equipment capacity, and then configuring automation terminals, communication links, and protection devices. This approach makes it difficult to comprehensively consider primary equipment construction costs, secondary system construction costs, communication resource accessibility, protection coordination, power supply reliability, and operational risks in the early planning stages. This can easily lead to problems in subsequent construction, such as insufficient communication coverage, unavailability of remote control, inadequate protection selectivity, or repeated modifications. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a reinforcement learning-based optimization method for primary and secondary synchronization planning in distribution networks, which effectively improves the efficiency and reliability of primary and secondary synchronization planning in distribution networks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The reinforcement learning-based method for primary and secondary synchronization planning and optimization of distribution networks includes the following steps: S1: Obtain the original planning data of the power distribution network in the target area, and preprocess it to obtain a standardized planning dataset; S2: Based on standardized planning datasets, construct a digital twin environment for distribution networks that is coupled in primary and secondary dimensions; S3: Based on the operational metrics output by the digital twin environment, obtain the state of the reinforcement learning agent during the planning phase, and introduce graph embedding features to construct a joint state; S4: Establish a hierarchical reinforcement learning model, including an upper-layer primary planning agent and a lower-layer secondary collaborative agent; the upper-layer primary planning agent receives the joint state and outputs the primary equipment planning action; the lower-layer secondary collaborative agent receives the joint state and the primary equipment planning action and outputs the secondary equipment planning action. S5: Input the joint planning action into the digital twin environment, execute the joint planning action, and calculate the multi-objective reward value; S6: Use the state, joint planning action, reward value, next state and constraint violation information as training samples to train the optimization model and output the results of primary and secondary synchronous planning.

[0007] Furthermore, the original planning data includes primary equipment data, secondary equipment data, load forecast data, distributed power source access forecast data, energy storage access forecast data, geospatial data, equipment economic data, communication resource data, and operational constraint data.

[0008] Further, the preprocessing shown includes performing missing data completion, anomaly correction, time scale unification, spatial coordinate matching, parameter normalization, and topology association processing on the original planning data to form a standardized planning dataset containing primary network topology, secondary communication topology, equipment attributes, and prediction scenario parameters.

[0009] Furthermore, the digital twin environment includes a primary electrical network simulation model, a secondary communication network simulation model, a protection and control simulation model, and an economic evaluation model. The primary electrical network simulation model is used to calculate node voltage, branch power flow, line load factor, transformer load factor, network loss, short-circuit current, and power supply capacity based on the distribution network topology, line parameters, equipment capacity, load level, and distributed power output. The secondary communication network simulation model is used to calculate the communication path, transmission delay, bandwidth occupancy, packet loss rate, communication availability, and redundancy between the automation terminal and the edge node or distribution automation master station. The protection and control simulation model is used to determine whether the protection device operates correctly, whether the fault is effectively isolated, and whether the power supply can be quickly restored based on the fault location, fault type, fault current, switch status, and communication status.

[0010] Furthermore, based on the operational indicators output by the digital twin environment, the state of the reinforcement learning agent during the planning phase is obtained, and graph embedding features are introduced to construct a joint state, specifically as follows: The states include electrical state, communication state, protection state, economic state, and uncertainty state; substations, feeders, distribution transformers, switches, and distributed power sources in the primary electrical network are taken as first-type nodes, and protection devices, automation terminals, communication nodes, edge nodes, and master stations in the secondary network are taken as second-type nodes, and physical connection relationships, communication connection relationships, and control association relationships are taken as different types of edges; feature extraction is performed on this heterogeneous graph through a graph neural network to obtain graph embedding features that can characterize the topological association relationships between the primary and secondary networks; the graph embedding features are concatenated with the electrical state, communication state, protection state, economic state, and uncertainty state to obtain the joint state.

[0011] Furthermore, primary equipment planning actions include the construction or expansion of substations, the construction or capacity increase of distribution transformers, the construction or renovation of feeders, the construction of tie lines, the configuration of sectionalizing switches, the configuration of reactive power compensation equipment, the configuration of energy storage equipment, and the adjustment of distributed power supply access capacity; secondary equipment planning actions include the configuration of relay protection devices, the configuration of feeder automation terminals, the configuration of fault indicators, the selection of communication methods, the construction of communication links, the configuration of redundant communication paths, the deployment of edge computing nodes, the selection of protection setting ranges, and the selection of fault self-healing control strategies; before generating actions, action masks are generated based on equipment capacity boundaries, planning budgets, line corridor conditions, station conditions, terminal power supply conditions, communication coverage capabilities, communication bandwidth conditions, protection adaptation relationships, and the minimum construction period of equipment. The upper-level primary planning agent outputs feasible primary equipment planning actions under the constraint of action mask; the lower-level secondary collaborative agent outputs matching secondary equipment planning actions based on the grid construction results corresponding to the primary equipment planning actions; the two types of actions are combined to form a joint planning action.

[0012] Furthermore, the joint planning actions are input into the digital twin environment, executed, and multi-objective reward values ​​are calculated. Specifically, the primary network topology, primary equipment capacity, switch placement, secondary terminal layout, communication links, protection configuration, and control strategies for the current planning stage are updated. Subsequently, multi-scenario simulations are performed in the updated distribution network model. Based on the simulation results, the state for the next planning stage is generated, and reward values ​​are calculated.

[0013] Furthermore, the state, joint planning action, reward value, next state, and constraint violation information are used as training samples to train the optimization model and output the results of primary and secondary simultaneous planning, as follows: Training samples are stored in the experience replay pool, and sample priority is determined based on timing difference error, power supply reliability risk, communication failure risk, and protection mismatch risk. Constraint reinforcement learning algorithm is used to update the policy parameters of the upper-layer primary planning agent and the lower-layer secondary collaborative agent, so that the model can improve the cumulative reward while satisfying the constraints of voltage, current, capacity, communication delay, communication reliability, protection action time, protection selectivity and investment budget. During the training process, when the cumulative reward is greater than the preset value, the constraint violation rate is lower than the preset threshold, and the comprehensive technical and economic indicators under different prediction scenarios meet the requirements, the trained primary and secondary synchronous planning optimization model is obtained. Different load growth paths, distributed power access paths, energy storage development paths, communication failure paths, and equipment failure paths are input into the trained model to generate multiple candidate planning schemes. Then, the candidate planning schemes are comprehensively evaluated, and the scheme that meets the hard constraints and has the highest overall benefit is selected as the final output.

[0014] The distribution network primary and secondary synchronization planning and optimization system based on reinforcement learning includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the distribution network primary and secondary synchronization planning and optimization method based on reinforcement learning as described above.

[0015] A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0016] The present invention has the following beneficial effects: 1. This invention incorporates the primary network structure, secondary automation equipment, communication network and protection control strategy of the distribution network into a unified reinforcement learning decision framework, and links the expansion of primary equipment with secondary terminals, communication links and protection configuration actions. It can identify problems such as communication unreachability, protection mismatch or insufficient automation coverage in advance during the planning stage, thereby improving the consistency of primary and secondary equipment configuration and the feasibility of engineering. 2. This invention utilizes a primary and secondary coupled digital twin environment to jointly simulate indicators such as power flow, voltage, equipment load, fault isolation, communication latency, link reliability, and protection actions; and processes complex topologies and multi-scale planning decisions through heterogeneous graph feature extraction and hierarchical reinforcement learning. 3. This solution uses action masking, constraint reinforcement learning, and multi-objective reward mechanisms to enable the planning model to control the investment cost throughout the entire life cycle while taking into account power supply reliability, voltage quality, communication availability, protection correct operation rate, and fault recovery speed. Ultimately, it forms a phased implementation scheme for simultaneous primary and secondary construction, reducing the risk of repeated modifications and equipment idleness in the later stages. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 In this embodiment, a reinforcement learning-based method for primary and secondary synchronization planning and optimization of a power distribution network is provided, comprising the following steps: S1: Obtain the original planning data of the power distribution network in the target area, and preprocess it to obtain a standardized planning dataset; S2: Based on standardized planning datasets, construct a digital twin environment for distribution networks that is coupled in primary and secondary dimensions; S3: Based on the operational metrics output by the digital twin environment, obtain the state of the reinforcement learning agent during the planning phase, and introduce graph embedding features to construct a joint state; S4: Establish a hierarchical reinforcement learning model, including an upper-layer primary planning agent and a lower-layer secondary collaborative agent; the upper-layer primary planning agent receives the joint state and outputs the primary equipment planning action; the lower-layer secondary collaborative agent receives the joint state and the primary equipment planning action and outputs the secondary equipment planning action. S5: Input the joint planning action into the digital twin environment, execute the joint planning action, and calculate the multi-objective reward value; S6: Use the state, joint planning action, reward value, next state and constraint violation information as training samples to train the optimization model and output the results of primary and secondary synchronous planning.

[0019] In this embodiment, the original planning data includes primary equipment data, secondary equipment data, load forecast data, distributed power source access forecast data, energy storage access forecast data, geospatial data, equipment economic data, communication resource data, and operational constraint data. Primary equipment data includes the location, capacity, connection relationships, service life, and failure rate of substations, distribution transformers, feeders, tie lines, switches, reactive power compensation devices, and energy storage devices. Secondary equipment data includes the location, model, communication method, bandwidth, latency, and availability of relay protection devices, feeder automation terminals, fault indicators, communication terminals, edge computing nodes, and distribution automation master stations. Communication resource data includes fiber optic channels, wireless base stations, communication towers, station building resources, and existing communication links.

[0020] In this embodiment, the preprocessing includes performing missing data completion, anomaly correction, time scale unification, spatial coordinate matching, parameter normalization, and topology association processing on the original planning data to form a standardized planning dataset containing primary network topology, secondary communication topology, equipment attributes, and predicted scenario parameters.

[0021] Specifically, in this embodiment, the collected raw planning data undergoes integrity verification and missing data completion processing. For missing fields in primary equipment ledgers, secondary equipment ledgers, load records, distributed power output records, communication link records, and fault records, corresponding completion methods are used based on data type: for static attributes such as equipment model, rated capacity, and installation location, completion can be based on similar equipment ledgers, design drawings, or parameters of adjacent equipment; for time-series data such as load, output, communication latency, and bandwidth usage, completion can be based on data from adjacent time periods, historical data under similar weather conditions, or prediction model output. Subsequently, abnormal data is identified and corrected. For example, capacity values ​​exceeding the rated capacity range of equipment, voltage or power flow values ​​exceeding the physically feasible range, obviously abnormal communication latency, and duplicate equipment numbers are screened. Data confirmed to be due to collection errors is corrected, removed, or marked to obtain basic data whose data quality meets the requirements of planning calculations. The verified data undergoes time-scale unification, spatial coordinate matching, and parameter normalization. Data from different sources is mapped to a unified planning period and sampling interval. For example, minute-level load and distributed power output data are aggregated into typical daily, typical time period, or annual planning scenario data according to preset rules. Equipment commissioning time, renovation time, and life cycle are mapped to the corresponding planning stage. Based on geographical coordinates, station building number, tower number, line section number, and feeder information of the equipment, spatial location matching is performed on substations, distribution transformers, lines, switches, automation terminals, protection devices, communication nodes, and communication links to identify the actual installation, power supply, communication, and control relationships between primary and secondary equipment. For parameters with different dimensions, such as capacity, impedance, cost, delay, reliability, load, and output, they are converted into a unified numerical range according to preset normalization rules to reduce the impact of different dimensional parameters on the state input of subsequent reinforcement learning models.

[0022] Finally, topology association processing is performed on the preprocessed data to establish primary network topology, secondary communication topology, and primary-secondary cross-layer association relationships. The primary network topology is generated based on the electrical connections between substations, busbars, distribution transformers, feeders, lines, tie lines, switches, distributed power sources, and energy storage devices. The secondary communication topology is generated based on the connections between automation terminals, protection devices, communication access equipment, communication nodes, edge computing nodes, master stations, and communication links. Furthermore, associations are established between switches and their corresponding monitoring and control terminals, protection devices and protected primary equipment, automation terminals and communication links, and communication nodes and edge nodes or master stations, forming a primary-secondary coupled topology. The aforementioned topology structure, equipment attributes, cost parameters, reliability parameters, and load and distributed power source prediction scenarios are collectively encapsulated into a standardized planning dataset.

[0023] In this embodiment, the digital twin environment includes a primary electrical network simulation model, a secondary communication network simulation model, a protection and control simulation model, and an economic evaluation model. The primary electrical network simulation model is used to calculate node voltage, branch power flow, line load factor, transformer load factor, network loss, short-circuit current, and power supply capacity based on the distribution network topology, line parameters, equipment capacity, load level, and distributed power output. The secondary communication network simulation model is used to calculate the communication path, transmission delay, bandwidth occupancy, packet loss rate, communication availability, and redundancy between the automation terminal and the edge node or distribution automation master station. The protection and control simulation model is used to determine whether the protection device operates correctly, whether the fault is effectively isolated, and whether the power supply can be quickly restored based on the fault location, fault type, fault current, switch status, and communication status.

[0024] The system establishes a correspondence between switches, transformers, and distributed power supply access points in the primary electrical network and protection devices, monitoring and control terminals, and communication links in the secondary system. After adjustments to the primary network topology, the digital twin environment synchronously updates communication link connectivity, protection action logic, and fault handling capabilities. When a communication link fails or protection configuration is mismatched, the corresponding results are fed back to the reliability assessment of the primary network. The system outputs electrical operation indicators, communication indicators, protection indicators, and economic indicators for each simulation scenario.

[0025] In this embodiment, based on the operational metrics output by the digital twin environment, the state of the reinforcement learning agent during the planning phase is obtained, and graph embedding features are introduced to construct a joint state, specifically as follows: The states include electrical state, communication state, protection state, economic state, and uncertainty state; the electrical state includes the current network topology, node voltage deviation, line load factor, distribution transformer load factor, power supply radius, network loss rate, reserve capacity, and distributed power supply penetration rate; the communication state includes terminal coverage, communication latency, bandwidth utilization, link availability, communication path redundancy, and edge node computing load; the protection state includes protection coverage, protection action accuracy, protection selectivity, fault isolation time, and power restoration time; the economic state includes invested costs, available budget, and so on. The system anticipates maintenance costs, power outage losses, and equipment replacement costs. Uncertainties include load forecasting errors, distributed power output fluctuations, equipment failure probabilities, and communication link failure probabilities. Substations, feeders, distribution transformers, switches, and distributed power sources in the primary electrical network are classified as first-class nodes, while protection devices, automation terminals, communication nodes, edge nodes, and master stations in the secondary network are classified as second-class nodes. Physical connections, communication connections, and control relationships are classified as different types of edges. A graph neural network is used to extract features from this heterogeneous graph, resulting in graph embedding features that characterize the topological relationships between the primary and secondary networks. These graph embedding features are then concatenated with electrical states, communication states, protection states, economic states, and uncertainties to obtain a joint state.

[0026] In this embodiment, primary equipment planning actions include the construction or expansion of substations, the construction or capacity increase of distribution transformers, the construction or renovation of feeders, the construction of tie lines, the configuration of sectionalizing switches, the configuration of reactive power compensation equipment, the configuration of energy storage equipment, and the adjustment of distributed power supply access capacity; secondary equipment planning actions include the configuration of relay protection devices, the configuration of feeder automation terminals, the configuration of fault indicators, the selection of communication methods, the construction of communication links, the configuration of redundant communication paths, the deployment of edge computing nodes, the selection of protection setting ranges, and the selection of fault self-healing control strategies. Before generating actions, an action mask is generated based on equipment capacity boundaries, planning budget, line corridor conditions, station conditions, terminal power supply conditions, communication coverage capabilities, communication bandwidth conditions, protection adaptation relationships, and minimum equipment construction period. The action mask is used to exclude planned actions that are not feasible to construct, cannot be accessed, or cannot meet security constraints. In this embodiment, the upper-layer primary planning agent outputs feasible primary equipment planning actions under the action mask constraint; the lower-layer secondary collaborative agent outputs matching secondary equipment planning actions based on the grid construction results corresponding to the primary equipment planning actions; the two types of actions are combined to form a joint planning action.

[0027] Specifically, in this embodiment, A hierarchical reinforcement learning model is established, comprising an upper-level primary planning agent and a lower-level secondary collaborative agent. The upper-level primary planning agent generates primary equipment planning actions based on the joint state of the primary and secondary distribution networks, while the lower-level secondary collaborative agent generates secondary equipment planning actions that match the primary network structure based on the joint state and the primary equipment planning actions, thereby forming a joint planning decision for the coordinated optimization of primary equipment construction and secondary equipment configuration.

[0028] In the planning phase t, the preprocessed joint state is represented as: ; in, It indicates the primary electrical status, including network topology, node voltage, line load factor, transformer load factor, equipment capacity margin, network loss rate, and distributed power supply access level; It indicates the status of secondary communication, including communication link availability, transmission latency, bandwidth utilization, communication path redundancy, and edge node load rate; It indicates the protection and control status, including protection coverage, protection action accuracy, fault isolation time, and power restoration time; It indicates the economic status, including invested costs, remaining budget, operation and maintenance costs, and losses due to power outages; This represents a state of uncertainty, including load forecasting errors, distributed power generation output fluctuations, equipment failure probabilities, and communication failure probabilities; g t This represents the primary and secondary coupled topological features extracted through a heterogeneous graph neural network.

[0029] Upper-level planning agents use joint state s t As input, and utilize policy networks Output the primary equipment planning action: ; Where θ1 represents the policy network parameters of the upper-layer primary planning agent. The primary device planning action is expressed as: ; in, This indicates the construction, expansion, or capacity adjustment of a substation. This indicates the construction, capacity expansion, or renovation of distribution transformers; This indicates the construction and renovation of feeder lines, tie lines, or cable lines; Indicates the configuration and operation of the sectionalizing switch, tie switch, or recloser; This indicates the location and capacity adjustment actions of distributed power sources; Indicates the configuration action of the energy storage device; This indicates the configuration action of the reactive power compensation equipment; To improve the feasibility of upper-level planned actions, a primary equipment action mask is constructed based on the current state. The action mask is generated based on the equipment capacity limit, annual investment budget, line corridor conditions, station construction conditions, access capacity limitations, and construction sequence during the planning phase. The upper-level primary planning agent selects actions only from the set of possible actions, and its probability distribution is expressed as: ; If a single equipment planning action satisfies the constraints, then If the constraints are not met, then The feasible primary device planning actions obtained after filtering by action mask are output to the lower-level secondary collaborative intelligent agent.

[0030] The lower-level secondary collaborative agents are in joint state s t And feasible primary equipment planning actions As input, utilize the conditional policy network Output secondary equipment planning actions: ; Where θ2 represents the policy network parameters of the lower-level secondary collaborative agent; The secondary equipment planning action is represented as follows: ; in, This indicates the type, quantity, installation location, configuration, and operation of the relay protection device; This indicates the configuration actions of the feeder automation terminal, measurement and control terminal, and fault indicator; This indicates the action of selecting a communication mode: fiber optic communication, wireless private network, 5G communication, or carrier communication. Indicates communication link routing, redundant links, and bandwidth allocation actions; This indicates the deployment of resources for edge computing nodes or power distribution automation master stations; This indicates the protection setting range and the configuration of protection coordination parameters. This indicates the configuration actions of feeder self-healing, fault isolation, and power restoration control strategies. For the lower-level secondary collaborative intelligent agent, construct a secondary device action mask that is associated with the planning actions of the upper-level primary device. The secondary equipment action mask is determined at least based on the primary equipment installation location, primary equipment type, switch controllability conditions, terminal power supply conditions, communication resource coverage, communication bandwidth, end-to-end delay, protection action time limit, and protection coordination relationship. The effective strategy for the lower-level secondary collaborative intelligent agent is: ; Therefore, the lower-level secondary collaborative intelligent agent generates secondary equipment planning actions that are compatible with the primary network construction actions. For example, when the upper-level intelligent agent configures a remotely controllable sectionalizing switch on a feeder, the lower-level intelligent agent configures a monitoring and control terminal, protection device, and communication link that meets the time delay requirements at the corresponding switch location; when the upper-level intelligent agent increases the capacity of distributed power supply access, the lower-level intelligent agent adjusts the protection directional elements, communication redundancy paths, and islanding detection and control strategies accordingly.

[0031] Combining primary equipment planning actions with secondary equipment planning actions yields joint planning actions: ; Joint planning actions Input a primary- or secondary-coupled digital twin environment to obtain the next state s t+1 Stage rewards r t and constraint information c t The reward value is constructed as follows: ; Among them, C t This represents the primary and secondary equipment investment costs, modification costs, and operation and maintenance costs; L t Indicates network loss and power outage loss; R t Indicates power supply reliability; Q t Indicates voltage quality index; Ut Indicates metrics such as communication reliability, communication coverage, and low-latency service capabilities; P t Indicates the protection correct operation rate, protection selectivity, and fault isolation efficiency; Φ t This represents the penalty value for constraint violation; α, β, γ, δ, ε, ζ, and η are the corresponding weight coefficients. The penalty value for constraint violation can be composed of electrical constraints, communication constraints, protection constraints, and investment constraints. ; in, This refers to violations of electrical constraints caused by node voltage exceeding limits, line overload, and transformer overload. This refers to the violation of communication constraints caused by communication delay exceeding limits, insufficient bandwidth, and unavailability of communication links; To protect against violations of protection constraints caused by failure to activate, malfunction, or failure of protection coordination; λ1 represents the economic constraint violation caused by planned investment exceeding the budget; λ2, λ3, and λ4 are the penalty coefficients corresponding to various constraints.

[0032] The upper-layer primary planning agent and the lower-layer secondary collaborative agent are trained with the goal of maximizing long-term cumulative reward. The optimization objective of the upper-layer agent is: ; The optimization objective of the lower-level agent is: ; Where T is the total number of stages in the planning period, and γ is the discount factor. By alternately or jointly updating the upper-level strategy network parameters θ1 and the lower-level strategy network parameters θ2, the upper-level agent learns the long-term primary network construction strategy, and the lower-level agent learns the secondary equipment, communication, and protection coordination configuration strategy that matches the evolution process of the primary network. Finally, the output is a distribution network primary and secondary synchronous planning scheme that meets the constraints of electrical safety, communication reliability, protection coordination, and economy.

[0033] In this embodiment, joint planning actions are input into the digital twin environment, executed, and multi-objective reward values ​​are calculated. Specifically, the primary network topology, primary equipment capacity, switch placement, secondary terminal layout, communication links, protection configuration, and control strategies for the current planning stage are updated. Subsequently, multi-scenario simulations are performed in the updated distribution network model. These scenarios include peak load scenarios, low load scenarios, typical load scenarios, high distributed power output scenarios, low distributed power output scenarios, feeder fault scenarios, communication link fault scenarios, protection failure scenarios, and protection maloperation scenarios. For each scenario, power flow results, power supply reliability, voltage qualification rate, equipment overload rate, communication latency, communication availability, fault isolation time, power supply recovery time, protection correct operation rate, and equipment investment and maintenance costs are calculated. Based on the simulation results, the state for the next planning stage is generated, and the reward value r is calculated. t .

[0034] In this embodiment, the state, joint planning action, reward value, next state, and constraint violation information are used as training samples to train the optimization model and output the results of primary and secondary synchronous planning, as follows: Training samples are stored in the experience replay pool, and sample priority is determined based on timing difference error, power supply reliability risk, communication failure risk, and protection mismatch risk. Constraint reinforcement learning algorithm is used to update the policy parameters of the upper-layer primary planning agent and the lower-layer secondary collaborative agent, so that the model can improve the cumulative reward while satisfying the constraints of voltage, current, capacity, communication delay, communication reliability, protection action time, protection selectivity and investment budget. During the training process, when the cumulative reward is greater than the preset value, the constraint violation rate is lower than the preset threshold, and the comprehensive technical and economic indicators under different prediction scenarios meet the requirements, the trained primary and secondary synchronous planning optimization model is obtained. Different load growth paths, distributed power source access paths, energy storage development paths, communication failure paths, and equipment failure paths are input into the trained model to generate multiple candidate planning schemes. These candidate schemes are then comprehensively evaluated, and the scheme that meets hard constraints and offers the highest overall benefit is selected as the final output. The final output includes: construction schemes for substations, distribution transformers, feeders, tie lines, switches, energy storage, and reactive power compensation equipment at each planning stage; configuration schemes for protection devices, feeder automation terminals, fault indicators, communication equipment, and edge computing nodes; communication link paths and redundancy configuration schemes; protection settings and feeder self-healing control schemes; phased investment plans; and corresponding predictions for power supply reliability, voltage qualification rate, communication availability, fault isolation time, and total life cycle cost.

[0035] The distribution network primary and secondary synchronization planning and optimization system based on reinforcement learning includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the distribution network primary and secondary synchronization planning and optimization method based on reinforcement learning as described above.

[0036] A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0037] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0038] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0039] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0040] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0041] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A reinforcement learning-based method for primary and secondary synchronization planning and optimization in power distribution networks, characterized in that, Includes the following steps: S1: Obtain the original planning data of the power distribution network in the target area, and preprocess it to obtain a standardized planning dataset; S2: Based on standardized planning datasets, construct a digital twin environment for distribution networks that is coupled in primary and secondary dimensions; S3: Based on the operational metrics output by the digital twin environment, obtain the state of the reinforcement learning agent during the planning phase, and introduce graph embedding features to construct a joint state; S4: Establish a hierarchical reinforcement learning model, including an upper-layer primary planning agent and a lower-layer secondary collaborative agent; the upper-layer primary planning agent receives the joint state and outputs the primary equipment planning action; the lower-layer secondary collaborative agent receives the joint state and the primary equipment planning action and outputs the secondary equipment planning action. S5: Input the joint planning action into the digital twin environment, execute the joint planning action, and calculate the multi-objective reward value; S6: Use the state, joint planning action, reward value, next state and constraint violation information as training samples to train the optimization model and output the results of primary and secondary synchronous planning.

2. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 1, characterized in that, The original planning data includes primary equipment data, secondary equipment data, load forecast data, distributed power source access forecast data, energy storage access forecast data, geospatial data, equipment economic data, communication resource data, and operational constraint data.

3. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 2, characterized in that, The preprocessing described includes performing missing data completion, anomaly correction, time scale unification, spatial coordinate matching, parameter normalization, and topology association processing on the original planning data to form a standardized planning dataset containing primary network topology, secondary communication topology, equipment attributes, and prediction scenario parameters.

4. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 2, characterized in that, The digital twin environment includes a primary electrical network simulation model, a secondary communication network simulation model, a protection and control simulation model, and an economic evaluation model. The primary electrical network simulation model is used to calculate node voltage, branch power flow, line load factor, transformer load factor, network loss, short-circuit current, and power supply capacity based on the distribution network topology, line parameters, equipment capacity, load level, and distributed power output. The secondary communication network simulation model is used to calculate the communication path, transmission delay, bandwidth occupancy, packet loss rate, communication availability, and redundancy between the automation terminal and the edge node or the distribution automation master station. The protection and control simulation model is used to determine whether the protection device operates correctly, whether the fault is effectively isolated, and whether the power supply can be quickly restored, based on the fault location, fault type, fault current, switch status, and communication status.

5. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 1, characterized in that, The process involves obtaining the state of the reinforcement learning agent during the planning phase based on the operational indicators output by the digital twin environment, and then introducing graph embedding features to construct a joint state, as follows: The states include electrical state, communication state, protection state, economic state, and uncertainty state; substations, feeders, distribution transformers, switches, and distributed power sources in the primary electrical network are designated as first-type nodes, and protection devices, automation terminals, communication nodes, edge nodes, and master stations in the secondary network are designated as second-type nodes; physical connections, communication connections, and control relationships are designated as different types of edges; feature extraction is performed on this heterogeneous graph using a graph neural network to obtain graph embedding features that can characterize the topological relationships between the primary and secondary networks; the graph embedding features are then concatenated with the electrical state, communication state, protection state, economic state, and uncertainty state to obtain the joint state.

6. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 1, characterized in that, The primary equipment planning actions include the construction or expansion of substations, the construction or capacity increase of distribution transformers, the construction or renovation of feeders, the construction of tie lines, the configuration of sectionalizing switches, the configuration of reactive power compensation equipment, the configuration of energy storage equipment, and the adjustment of distributed power supply access capacity; the secondary equipment planning actions include the configuration of relay protection devices, the configuration of feeder automation terminals, the configuration of fault indicators, the selection of communication methods, the construction of communication links, the configuration of redundant communication paths, the deployment of edge computing nodes, the selection of protection setting ranges, and the selection of fault self-healing control strategies; before generating actions, an action mask is generated based on equipment capacity boundaries, planning budget, line corridor conditions, station conditions, terminal power supply conditions, communication coverage capabilities, communication bandwidth conditions, protection adaptation relationships, and the minimum construction period of the equipment; The upper-level primary planning agent outputs feasible primary equipment planning actions under the constraint of action mask; the lower-level secondary collaborative agent outputs matching secondary equipment planning actions based on the grid construction results corresponding to the primary equipment planning actions; the two types of actions are combined to form a joint planning action.

7. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 1, characterized in that, The process of inputting joint planning actions into the digital twin environment, executing joint planning actions, and calculating multi-objective reward values ​​is as follows: updating the primary network topology, primary equipment capacity, switch placement, secondary terminal layout, communication links, protection configuration, and control strategies for the current planning stage; then performing multi-scenario simulations in the updated distribution network model; generating the state for the next planning stage based on the simulation results, and calculating reward values.

8. The reinforcement learning-based primary and secondary synchronization planning and optimization method for distribution networks according to claim 7, characterized in that, The process involves using the state, joint planning action, reward value, next state, and constraint violation information as training samples to train the optimization model and output the results of primary and secondary synchronous planning, as detailed below: Training samples are stored in the experience replay pool, and sample priority is determined based on timing difference error, power supply reliability risk, communication failure risk, and protection mismatch risk. Constraint reinforcement learning algorithm is used to update the policy parameters of the upper-level primary planning agent and the lower-level secondary collaborative agent, so that the model can improve the cumulative reward while satisfying the constraints of voltage, current, capacity, communication delay, communication reliability, protection action time, protection selectivity and investment budget. During training, when the cumulative reward is greater than the preset value, the constraint violation rate is lower than the preset threshold, and the comprehensive technical and economic indicators under different prediction scenarios meet the requirements, the trained primary and secondary synchronous planning optimization model is obtained. Different load growth paths, distributed power access paths, energy storage development paths, communication failure paths, and equipment failure paths are input into the trained model to generate multiple candidate planning schemes. Then, the candidate planning schemes are comprehensively evaluated, and the scheme that meets the hard constraints and has the highest overall benefit is selected as the final output.

9. A distribution network primary and secondary synchronization planning and optimization system based on reinforcement learning, characterized in that, It includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the reinforcement learning-based distribution network primary and secondary synchronization planning optimization method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-8.

Images