A power distribution network hierarchical adaptive optimization method and device based on PPO reinforcement learning

Abstract

The application discloses a power distribution network hierarchical adaptive optimization method and device based on PPO reinforcement learning, obtains multi-dimensional heterogeneous distributed resource operation data, and constructs resource capability features; a power distribution network hierarchical adaptive optimization architecture is constructed, a task layer identifies weak points of the power distribution network based on power grid operation state data, converts the weak points into task segment features, and calculates a task influence domain; a coalition layer filters candidate dynamic complementary coalitions from the resource capability features in the task influence domain according to a preset mode; the task layer uses a PPO reinforcement learning decision model to optimize the candidate dynamic complementary coalitions to output target dynamic complementary coalitions, and issues executable task envelopes corresponding to the target dynamic complementary coalitions to a unit layer; the unit layer calls a local strategy template, converts the executable task envelopes into device native commands and executes the same, and in response to the fact that a comprehensive execution deviation or a burst event quantitative index exceeds a corresponding threshold value, local reconstruction is triggered, otherwise the unit layer is self-corrected.

Description

Technical Field

[0001] This application relates to a hierarchical adaptive optimization method and device for distribution networks based on PPO reinforcement learning, belonging to the field of substation operation and maintenance technology. Background Technology

[0002] With the large-scale integration of distributed photovoltaics, distributed energy storage, electric vehicles, flexible adjustable loads, microgrids, and inverter-type reactive power regulation devices, the dispatching targets of the distribution network have shifted from relatively homogeneous load units to a mixed resource group with strong heterogeneity, strong randomness, and multiple constraint coupling characteristics. Different resources exhibit significant differences in terms of regulation direction, dynamic response delay, duration, recovery time, available capacity, user constraints, communication reliability, and historical reliability.

[0003] The development of artificial intelligence technology has propelled reinforcement learning (RL) to become a crucial technique for solving complex decision-making problems. Through the interaction between the agent and the environment, and driven by reward signals, RL optimizes policies without relying on precise mathematical models. It can achieve optimal or near-optimal decisions in uncertain environments, providing a new technical path for addressing these problems. Common reinforcement learning algorithms include value function-based deep Q-networks (DQNs) and policy gradient-based actor-critic methods. However, DQN algorithms are primarily suitable for discrete action spaces and struggle with the continuous adjustment interval and resource combination decision-making problems discussed in this application. While traditional policy gradient methods can handle continuous action spaces, they are prone to problems such as excessively large policy update amplitudes and unstable convergence during training.

[0004] Current distribution network dispatch optimization methods still have many shortcomings and are difficult to adapt to the dispatch requirements of the aforementioned heterogeneous resources. Most existing distribution network dispatch optimization methods still adopt a unified power variable modeling approach, approximating heterogeneous resources as continuously adjustable quantities that can be substituted for each other. While this approach facilitates the construction of mathematical models, it easily overlooks the essential differences between resources in terms of "whether they can be adjusted, when to adjust, how long they can be adjusted, and whether there is a rebound after adjustment." This leads to deviations or even failures in the execution of upper-level dispatch results at the lower level. Especially when the distribution network has weak links such as node voltage exceeding limits, branch overload, rapid net load ramp-up, reverse power flow, and insufficient local reserves, the unified variable model often fails to quickly organize resources that are truly effective for the weak points and cannot timely mitigate operational risks.

[0005] On the other hand, the scheduling architecture lacks adaptability. Existing methods mostly rely on centralized optimization across the entire network, using multiple objective functions to balance the economy, security, and power quality of the distribution network. This architecture is prone to problems such as model dimensionality expansion, frequent recalculations, and excessively long solution times when there are a large number of devices, frequent scenario fluctuations, and dynamic topology changes. At the same time, most hierarchical scheduling methods use fixed voltage level hierarchies or fixed geographical partitioning patterns, failing to adaptively adjust the scheduling boundaries according to changes in specific weak points in the distribution network. This results in resources with adjustment capabilities and the ability to influence weak points not being included in the coordination scope in a timely manner, reducing the targeting and effectiveness of scheduling.

[0006] Furthermore, existing methods often directly distribute scheduling results as equipment setpoints, lacking intermediate resource alliance acceptance and local template conversion mechanisms. When energy storage status approaches its limit, electric vehicle users temporarily disconnect from the grid, flexible load comfort is limited, or communication quality deteriorates, the direct setpoint approach is prone to execution deviations, control oscillations, and post-event rebounds, affecting the stability of the distribution network operation.

[0007] The collaborative scheduling problem of heterogeneous distributed resources in the power distribution network is characterized by high state space dimension, uncertain resource response, and strong temporal coupling of the adjustment process. Traditional optimization methods are difficult to simultaneously take into account real-time performance and optimality. Therefore, it is necessary to introduce reinforcement learning methods to model and solve the scheduling process.

[0008] In summary, existing scheduling optimization methods cannot effectively adapt to the scheduling requirements of heterogeneous distributed resources. There is an urgent need to propose a hierarchical adaptive optimization method that does not rely on the simple superposition of multiple objective functions, but starts from the weak points of distribution network operation, can accurately characterize the inherent differences of heterogeneous distributed resources, dynamically organize effective resources, and has the ability to quickly reconfigure local resources. Summary of the Invention

[0009] Objective: In view of at least one of the above-mentioned technical problems, this application provides a hierarchical adaptive optimization method and device for distribution networks based on PPO reinforcement learning.

[0010] The technical solution adopted in this application is:

[0011] Firstly, this application provides a hierarchical adaptive optimization method for distribution networks based on PPO reinforcement learning, including:

[0012] Step S1: Obtain multidimensional heterogeneous distributed resource operation data, and construct resource capability characteristics based on the multidimensional heterogeneous distributed resource operation data;

[0013] Step S2: Construct a hierarchical adaptive optimization architecture for the distribution network consisting of a task layer, an alliance layer, and a unit layer. The task layer identifies the weak points of the distribution network based on the power grid operation status data, and converts each weak point into a task segment feature that matches the resource capability feature dimension. The task influence domain is then calculated based on the task segment feature.

[0014] Step S3: Within the mission's influence domain, the alliance layer selects candidate dynamic complementary alliances from resource capability characteristics according to a preset method.

[0015] Step S4: The task layer uses the PPO reinforcement learning decision model to optimize the candidate dynamic complementary alliances and output the target dynamic complementary alliance, and then sends the executable task envelope corresponding to the target dynamic complementary alliance to the unit layer.

[0016] Step S5: The unit layer calls the corresponding local policy template according to the resource type, converts the received executable task envelope into a device native command, and executes it.

[0017] In some embodiments, the method further includes: step S6, obtaining the actual response result and the burst event quantification index during device execution, and calculating the comprehensive execution deviation based on the deviation between the actual response result and the executable task envelope; triggering local reconstruction when the comprehensive execution deviation or the burst event quantification index exceeds the corresponding preset threshold, otherwise self-correcting by the unit layer.

[0018] In a second aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0019] Thirdly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0020] Beneficial Effects: The distribution network hierarchical adaptive optimization method and device based on PPO reinforcement learning provided in this application have the following advantages: It constructs a closed-loop scheduling system covering the entire process, from accurate characterization of heterogeneous resource capabilities and quantitative identification of operational weaknesses, to dynamic hierarchical architecture construction, complementary resource alliance aggregation, interval-based task envelope execution, and event-triggered local reconstruction. This achieves task-driven, dynamic adaptation, and precise execution of distribution network scheduling, improving the distribution network's collaborative scheduling capability for heterogeneous distributed resources. It solves the problems of difficulty in unified scheduling of heterogeneous resources, poor adaptability of fixed hierarchies, and insufficient instruction executability, ultimately improving the real-time performance, stability, and economy of distribution network operation. Its core idea is: instead of first uniformly solving the objective function of stacking heterogeneous resources across the entire network, it starts with identifying distribution network weaknesses, transforming the problem into a matching problem between task segments and resource capabilities. By constructing multi-dimensional quantitative resource capability characteristics, a unified quantitative representation of heterogeneous resource capabilities is achieved, laying the foundation for precise aggregation of resources and tasks. Using the task influence domain as the scheduling boundary of the alliance layer, the scheduling range is dynamically determined through weighted calculations of electrical sensitivity, electrical distance, and historical validity updated over time. Only resources that have a practical adjustment effect on weak points are included in the coordination range, avoiding the dimensional expansion of the model in centralized optimization across the entire network and improving the real-time performance of scheduling. Through the complementary combination of dynamic complementary alliances and the interval-based distribution of executable task envelopes, the unit layer is given the flexibility of local adjustment. Combined with local policy templates designed for different resources, the precise conversion of interval-based task requirements to native device commands is achieved, solving the problem of execution deviation in traditional fixed-setpoint instructions. An event-triggered local reconstruction mechanism is adopted, so that when a sudden situation occurs, alliance reconstruction and task envelope updates are performed only within the affected area, without requiring a full network recalculation. This significantly reduces the consumption of computing and communication resources and improves the system's response speed. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a hierarchical adaptive optimization method for distribution networks based on PPO reinforcement learning according to an embodiment of this application.

[0022] Figure 2 This is a schematic diagram illustrating the principle of a method according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a hierarchical adaptive optimization architecture according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the PPO reinforcement learning decision-making process according to one embodiment of this application. Detailed Implementation

[0025] The present application will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and should not be used to limit the scope of protection of the present application.

[0026] In the description of this application, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0027] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0028] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0029] Example 1: This example provides a hierarchical adaptive optimization method for distribution networks based on PPO reinforcement learning, such as... Figure 1 As shown, it includes:

[0030] Step S1: Obtain multidimensional heterogeneous distributed resource operation data, and construct resource capability characteristics based on the multidimensional heterogeneous distributed resource operation data;

[0031] Step S2: Construct a hierarchical adaptive optimization architecture for the distribution network consisting of a task layer, an alliance layer, and a unit layer. The task layer identifies the weak points of the distribution network based on the power grid operation status data, and converts each weak point into a task segment feature that matches the resource capability feature dimension. The task influence domain is then calculated based on the task segment feature.

[0032] Step S3: Within the mission's influence domain, the alliance layer selects candidate dynamic complementary alliances from resource capability characteristics according to a preset method.

[0033] Step S4: The task layer uses the PPO reinforcement learning decision model to optimize the candidate dynamic complementary alliances and output the target dynamic complementary alliance, and then sends the executable task envelope corresponding to the target dynamic complementary alliance to the unit layer.

[0034] Step S5: The unit layer calls the corresponding local policy template according to the resource type, converts the received executable task envelope into a device native command, and executes it.

[0035] Step S6: Obtain the actual response results and emergency event quantification indicators during the device execution process, and calculate the comprehensive execution deviation based on the deviation between the actual response results and the executable task envelope; when the comprehensive execution deviation or emergency event quantification indicator exceeds the corresponding preset threshold, trigger local reconstruction, otherwise self-correction is performed by the unit layer.

[0036] Furthermore, the historical reliability, dynamic response latency, sustainability, and bounce characteristics in the resource capability features are updated based on the execution results for use in the construction of dynamic complementary alliances in the next scheduling cycle.

[0037] The following is a detailed explanation of each step.

[0038] Step S1: Obtain multidimensional heterogeneous distributed resource operation data, and construct resource capability characteristics based on the multidimensional heterogeneous distributed resource operation data.

[0039] The heterogeneous distributed resources obtained in this application embodiment include at least distributed power sources, energy storage, electric vehicle aggregates, flexible loads, and reactive power regulation devices; furthermore, in this embodiment, the heterogeneous distributed resource operation data includes resource access location, power output / consumption status, energy remaining, actual response delay, user constraints, communication delay, packet loss rate, etc.

[0040] Preferably, based on the above-mentioned heterogeneous distributed resource operation data, resource capability characteristics are constructed for each heterogeneous distributed resource. The resource's adjustment direction set, dynamic response latency, sustainability, recovery time, available capacity, user constraints, communication reliability, historical credibility, and other multi-dimensional data are quantified into a feature matrix. This enables a precise and unified representation of the heterogeneous resource adjustment capability, breaks through the limitations of traditional single power boundary description, and lays the foundation for accurate matching of resources and tasks in the future.

[0041] In some embodiments, the resource capability characteristics encompass resource type, access location, and available adjustment service types, specifically including adjustment direction set, dynamic response latency, persistence capability, recovery time, available capacity, user constraints, communication reliability, and historical reliability. Represented as:

[0042] ,

[0043] in, For a moment Resource i's resource capability characteristics , , , , , , , Representing time respectively The adjustment direction set, dynamic response latency, persistence capability, recovery time, available capacity, user constraints, communication reliability, and historical reliability of resource i.

[0044] Step S2: Construct a hierarchical adaptive optimization architecture for the distribution network, consisting of a task layer, an alliance layer, and a unit layer. The task layer identifies the weak points of the distribution network based on the power grid operation status data, converts each weak point into a task segment feature that matches the resource capability feature dimension, and calculates the task influence domain based on the task segment feature.

[0045] Specifically, in this embodiment, the power grid operation status data includes distribution network topology, switch status, power flow status, node voltage, branch load rate, weather forecast, user behavior forecast, and communication status. It should be noted that the power grid operation status data described in this application includes historical operation status data and short-term forecast data for a future period obtained based on historical operation status data.

[0046] Based on the current operating status and short-term forecast results of the distribution network, the system calculates core operating indicators such as node voltage, branch load rate, net load change rate, reverse power flow, and reserve margin through a dual approach of operating indicator over-limit judgment criteria and trend quantitative analysis. This identifies weak points in the distribution network, including voltage weaknesses, overload weaknesses, net load ramp-up weaknesses, reverse power flow weaknesses, and insufficient reserve weaknesses.

[0047] More specifically, in this embodiment, the first is defined as... A weak point at all times Risk intensity for:

[0048] ;

[0049] in, Indicates items exceeding the limit; Indicates a deteriorating trend; This indicates a significant increase in uncertainty. , , Let be the weight coefficient, and satisfy... .

[0050] The "Exceeding Limit Risk" indicator represents the normalized degree of exceeding the limit after an operational indicator deviates from the safe reference value. The formula is as follows:

[0051] ;

[0052] in, ; The target operating parameters are, for example, node voltage and branch load rate in the current operating state of the distribution network. Indicates a reference safety value; This indicates the allowable safety deviation zone.

[0053] The trend deterioration indicator further considers whether the risk is accelerating and is used to constrain variables. The rate of change and acceleration are used to ensure a smooth trajectory and avoid drastic fluctuations. The calculation formula is as follows:

[0054] ;

[0055] in, For variables The absolute value of the first derivative, For variables The absolute value of the second derivative; , , which is the weighting coefficient used to balance the penalty intensity of first-order changes and second-order changes.

[0056] The uncertainty amplification index is used to characterize the level of uncertainty in the future operating state. The greater the predicted volatility, the more likely the system is to rapidly transition from a boundary safe state to an unstable state. The formula is as follows:

[0057] ;

[0058] in, Representing the future time domain Internal prediction sequence of operating indicators, This represents the VAR function.

[0059] By risk intensity To identify various weak points in the power distribution network.

[0060] Each vulnerability is transformed into a standardized task fragment that matches the resource capability characteristics. The control requirements of the vulnerability are broken down into standardized parameters such as task location, task time window, power demand range, adjustment direction, effective time limit, duration, safety boundary set, and rollback order. This transforms the complex optimization problem of the entire network into a localized task problem oriented towards the vulnerability, greatly reducing the solution complexity of the scheduling problem, while achieving dimensional matching between tasks and resources.

[0061] In some embodiments, the task segment features include the task's location of action, task time window, power requirement range, adjustment direction, effective time limit, duration, required minimum capacity, and set of safety boundaries corresponding to each vulnerability. In this embodiment, the task segment features are represented as follows:

[0062] ,

[0063] in, For a moment The task segment features corresponding to the kth weakness. The location where the task is applied; For task time windows; For the power demand range; To adjust the direction; For the effective period; Duration; For the minimum required capacity, Let k be the set of security boundaries, and k be the index of the vulnerability point or task.

[0064] Based on task segment characteristics, the task influence domain is dynamically generated according to the electrical sensitivity, electrical distance, and historical validity updates over time between the vulnerability and each resource. In this embodiment, the task influence domain is represented as follows:

[0065] ;

[0066] ;

[0067] in, For the current moment The task's influence domain For the future time domain, For the time in the rolling time domain Electrical sensitivity; Electrical distance; This is the distance attenuation coefficient; This is the distance decay kernel function; This is a historical validity item that updates over time; The current availability gate function for the resource; , , As weight;

[0068] in, This is an indicator function; it takes the value 1 if the condition is met, and 0 otherwise. Representing resources At any moment The set of adjustment directions, Indicates the first Adjusting the direction of each task; For resources At any moment Dynamic response latency, For the first The task at time The effective period; For resources At any moment Available capacity For the first The task at time Minimum required capacity.

[0069] It should be noted that the hierarchical adaptive optimization architecture of the distribution network constructed in this application, consisting of task layer, alliance layer, and unit layer, dynamically determines the functions and scheduling boundaries of each layer based on task fragment characteristics, thereby realizing task-driven dynamic hierarchical structure and breaking through the limitations of traditional fixed hierarchical structure.

[0070] The task layer is the top-level scheduling layer, responsible for quantitative identification of weak points in the distribution network, generation of standardized task fragment features, calculation of task influence domain, selection of target dynamic complementary alliances, and allocation of task envelopes.

[0071] The alliance layer serves as the intermediate coordination layer, with its scheduling boundary defined by the task's influence domain. Within this domain, candidate dynamic complementary alliances are formed through a two-level quantitative screening and complementary quantitative combination process. This process identifies resource sets that have a practical regulatory effect on weak points, and resource coordination occurs only within these sets, avoiding the participation of ineffective resources in scheduling. It should be noted that the scheduling boundary of the alliance layer is not a fixed voltage level or geographical boundary, but rather determined by the task's influence domain corresponding to the characteristics of the task segments.

[0072] The task influence domain is dynamically generated over time based on the electrical sensitivity, electrical distance, and historical validity between the weak points and various resources, thereby achieving dynamic layering driven by the task. The task influence domain is the cumulative influence integral over the rolling time domain, taking into account both the physical sensitivity relationship between resources and target points, as well as spatial attenuation and historical execution performance, thus forming a task influence domain boundary that changes dynamically over time.

[0073] The unit layer is the underlying execution layer. It uses a single resource or a resource aggregate as the scheduling unit and is responsible for calling the local policy module, converting the intervalized task envelope into the device's native command, executing scheduling instructions, and collecting actual response data.

[0074] Step S3: Within the task's influence domain, the alliance layer selects candidate dynamic complementary alliances from resource capability characteristics according to a preset method.

[0075] In step S3, within the task's influence domain, candidate dynamic complementary alliances are formed through two-level quantitative screening and complementary quantitative combination to achieve efficient collaboration of heterogeneous resources. Specifically, this includes:

[0076] Within the task's influence domain, a first screening is performed based on the basic matching degree (adjustment direction, effective time limit) between resource capability characteristics and task segment characteristics. A second screening is then performed based on the matching degree of operational boundaries (duration, available capacity, user constraints). Finally, the screened resources are divided into fast-response short-duration resources and slow-response long-duration resources according to dynamic response latency and sustainability. Complementarity metrics are used to form resource combinations, retaining resource combinations with a comprehensive credibility higher than a threshold, forming candidate dynamic complementary alliances. Fast-response resources are used to quickly suppress weaknesses, while slow-response resources are used to continuously adjust weaknesses, improving the stability and effectiveness of regulation.

[0077] More specifically, in this embodiment, the basic matching degree between resource capability characteristics and task fragment characteristics is firstly screened based on the task's adjustment direction (set of resource adjustment directions) and the effective time limit (resource dynamic response delay); the matching degree of the running boundary is secondly screened based on the duration of the task and the resource's continuous capability, recovery time, available capacity, and user constraints.

[0078] Candidate dynamic complementary alliances combine local resource constraints and energy boundaries to complete task feasibility verification and return the alliance-level executable task envelope and execution credibility to the task layer.

[0079] Step S4: The task layer uses the PPO reinforcement learning decision model to optimize the candidate dynamic complementary alliances and output the target dynamic complementary alliance, and then sends the executable task envelope corresponding to the target dynamic complementary alliance to the unit layer.

[0080] To achieve intelligent optimization of the dynamic complementary alliance, the power distribution network scheduling process is modeled as a sequential decision problem in reinforcement learning, which first constructs a state space and action space that reflect the system's operating state.

[0081] The state space, serving as input to the reinforcement learning policy network, is used to characterize comprehensive information about the distribution network's operating environment and task requirements at the current scheduling moment. It is composed of resource capability characteristics, task segment characteristics, and grid operating status data. In this embodiment, the system state is represented as:

[0082] , Indicates time The system status; For a moment Resource capabilities characteristics For a moment The task segment features corresponding to the kth weakness. For a moment The power grid's operating status.

[0083] Specifically, resource capability features are used to characterize the responsiveness and operational constraints of various resources; task segment features are used to describe the adjustment needs corresponding to weak points, including the location of action, power demand range, time constraints, and adjustment direction; and grid operation status data are used to reflect real-time operating conditions such as node voltage, branch load rate, and power flow distribution. By fusing the above multi-dimensional state information, the complexity and uncertainty of the current scheduling problem can be comprehensively reflected, providing accurate decision-making basis for reinforcement learning models, thereby improving the adaptability and accuracy of dynamic coalition selection.

[0084] The task layer establishes a multi-dimensional evaluation criterion with task coverage, execution reliability, and disturbance level as the core. The comprehensive evaluation value is calculated based on the task coverage, execution reliability, and disturbance level of each candidate dynamic complementary alliance. The comprehensive value of each candidate dynamic complementary alliance and the current system state are input into the PPO reinforcement learning decision model to generate the target dynamic complementary alliance.

[0085] The comprehensive evaluation value is calculated based on the task coverage, execution reliability, and disturbance level of each candidate dynamic complementary alliance, including:

[0086] ;

[0087] in, For comprehensive evaluation, For task coverage, To ensure credibility, Scoring for disturbances; , and These are the weighting coefficients.

[0088] In some embodiments, the task coverage is used to characterize the degree to which the executable task envelope returned by the candidate dynamic complementary alliance satisfies the adjustment task segment, and is calculated based on the active adjustment interval, reactive adjustment interval, effective time limit, holding time, adjustment direction and safety boundary.

[0089] In some embodiments, the execution reliability is used to characterize the reliability of the candidate dynamic complementary alliance in completing the executable task envelope, and is calculated based on the historical execution reliability, communication availability, response latency, actual fulfillment ratio, continuous fulfillment capability, and rebound degree after adjustment of resources within the alliance.

[0090] In some embodiments, the disturbance level is used to characterize the degree to which the candidate dynamic complementary alliance has a secondary impact on the distribution network operation status and the resource status itself after performing the adjustment task, and is determined based on the proximity of the resource energy boundary, the strength of user constraints, communication quality, predicted mutations, topology changes and adjustment rebound risk.

[0091] Action space As the output of the strategy network, it is used to generate decisions on the selection and structural adjustment of target dynamic complementary alliances. It not only includes selecting target dynamic complementary alliances from candidate dynamic complementary alliances, but also covers the dynamic adjustment process of alliance structure, such as increasing or decreasing resources, optimizing resource combination methods, and redistributing adjustment capabilities.

[0092] The PPO policy optimization mechanism introduces the Proximal Policy Optimization (PPO) algorithm into the dynamic complementary alliance selection process to constrain and optimize the policy update process, thereby avoiding drastic fluctuations in the policy during training and ensuring the smooth evolution of the alliance selection policy.

[0093] PPO Strategy Optimization Function as follows:

[0094] ,

[0095] in, Represents the expectation function, As the current strategy, This is the old strategy. For the dominant function, These are the trimming parameters.

[0096] The decision-making process based on the PPO algorithm includes the following steps: initializing the parameters of the policy network and value network, and constructing a distribution network scheduling environment model; under the current policy, the policy network outputs actions based on the comprehensive evaluation value of each candidate dynamic complementary alliance and the current system state, selecting or adjusting the alliance structure from the candidate dynamic complementary alliances; executing the actions to obtain a new system state and corresponding reward value, and forming state transition samples; calculating the advantage function based on the sampled data to measure the performance improvement of the current policy relative to the benchmark policy; updating the policy network using the PPO pruning objective function to improve the expected return under the condition of limiting the policy update range; repeatedly executing the adjustment of the alliance structure to update the policy network until the policy converges or reaches the preset training rounds; finally, during online scheduling, the trained policy model is used to make real-time optimization decisions for candidate dynamic complementary alliances.

[0097] The Proximal Policy Optimization (PPO) algorithm introduces a pruning mechanism during policy updates to constrain the deviation between the old and new policies, thereby preventing drastic policy fluctuations and ensuring the stability of the training process. Furthermore, the PPO algorithm eliminates the need for complex second-order optimization calculations, offering advantages such as simple implementation, high computational efficiency, and high sample utilization. Compared to Trust Region Policy Optimization (TRPO), PPO significantly reduces computational complexity while maintaining policy optimization stability, making it more suitable for the real-time-critical distribution network scheduling scenario described in this application.

[0098] By embedding the PPO reinforcement learning algorithm into the dynamic complementary alliance selection process, the traditional rule-based selection is transformed into adaptive decision-making based on policy learning, enabling the alliance construction process to have online learning and dynamic optimization capabilities.

[0099] S43. The task layer distributes the executable task envelope corresponding to the target dynamic complementary alliance to the unit layer.

[0100] In some embodiments, the executable task envelope includes an active power adjustment range, a reactive power adjustment range, a start time limit, a hold time, a ramp level, a reserve margin, and a fallback priority; the task layer sends the task envelope to the unit layer instead of fixed equipment settings.

[0101] Step S5: The unit layer calls the corresponding local policy template according to the resource type, converts the received executable task envelope into a device native command, and executes it.

[0102] In some embodiments, the local strategy template includes at least one of the following: energy storage strategy template, inverter-type distributed power generation strategy template, electric vehicle aggregation strategy template, flexible load strategy template, and reactive power regulation equipment strategy template.

[0103] Each template is an envelope-instruction conversion rule set designed for different resource physical characteristics and operational constraints. It is used to accurately convert the intervalized task envelope into the device's native commands that the device can directly execute and issue them for execution. At the same time, it collects data such as the device's actual active power response, actual reactive power response, and actual response delay in real time, providing a basis for subsequent deviation monitoring and feature updates.

[0104] Step S6: Obtain the actual response results and emergency event quantification indicators during the device execution process, and calculate the comprehensive execution deviation based on the deviation between the actual response results and the executable task envelope; when the comprehensive execution deviation or emergency event quantification indicator exceeds the corresponding preset threshold, trigger local reconstruction, otherwise self-correction is performed by the unit layer.

[0105] In some embodiments, an emergency includes at least one of the following: topology switching caused by a switching action, sudden changes in source-load prediction, energy storage state approaching the boundary, changes in user response intention, and deterioration in communication quality.

[0106] During the execution of dispatch instructions by the equipment, the deviation between the actual response result and the executable task envelope is continuously monitored to calculate the comprehensive execution deviation. At the same time, sudden events such as distribution network topology switching, sudden changes in source-load prediction, energy storage status approaching the boundary, changes in user response willingness, and communication quality degradation are monitored. All monitoring indicators are defined using quantitative criteria.

[0107] In this embodiment, the actual response results include the actual active response, the actual reactive response, and the actual response delay;

[0108] The overall execution deviation is calculated based on the deviation between the actual response result and the executable task envelope, and is expressed as follows:

[0109] ,

[0110] ,

[0111] in, To comprehensively account for execution deviations, , , These are active power deviation, reactive power deviation, and time deviation, respectively. , , The weights for active power deviation, reactive power deviation, and time deviation are respectively determined, and the weights of each dimension are dynamically adjusted according to the type of weakness to accurately judge the degree of execution deviation. , The actual active power response and actual reactive power response of the corresponding alliance or equipment for the task segment; , The active and reactive targets are used as references for the task envelope requirements; , The normalized benchmark for the reference quantity; This refers to the actual response time. The task requires a specific timeframe for taking effect; To prevent tiny positive numbers with a denominator of zero.

[0112] When the overall execution deviation If the quantitative indicators of a sudden event exceed the preset threshold, a full network recalculation will not be performed. Instead, a local reconstruction will be triggered (re-selecting resources, combining alliances, and generating task envelopes only within the affected task fragment features and their corresponding task influence domains), which will significantly reduce the consumption of computing and communication resources. If the deviation does not exceed the threshold, the unit layer will implement local self-correction within the scope of local policy template rules, fine-tuning the device execution instructions, suppressing system control oscillations, and improving operational stability.

[0113] Example 2: Based on Example 1, this example provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Example 1.

[0114] Example 3: Based on Example 1, this example provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in Example 1.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A hierarchical adaptive optimization method for distribution networks based on PPO reinforcement learning, characterized in that, include: Step S1: Obtain multidimensional heterogeneous distributed resource operation data, and construct resource capability characteristics based on the multidimensional heterogeneous distributed resource operation data; Step S2: Construct a hierarchical adaptive optimization architecture for the distribution network consisting of a task layer, an alliance layer, and a unit layer. The task layer identifies the weak points of the distribution network based on the power grid operation status data, and converts each weak point into a task segment feature that matches the resource capability feature dimension. The task influence domain is then calculated based on the task segment feature. Step S3: Within the mission's influence domain, the alliance layer selects candidate dynamic complementary alliances from resource capability characteristics according to a preset method. Step S4: The task layer uses the PPO reinforcement learning decision model to optimize the candidate dynamic complementary alliances and output the target dynamic complementary alliance, and then sends the executable task envelope corresponding to the target dynamic complementary alliance to the unit layer. Step S5: The unit layer calls the corresponding local policy template according to the resource type, converts the received executable task envelope into a device native command, and executes it.

2. The method according to claim 1, characterized in that, It also includes: step S6, obtaining the actual response results and emergency event quantification indicators during the device execution process, and calculating the comprehensive execution deviation based on the deviation between the actual response results and the executable task envelope; when the comprehensive execution deviation or emergency event quantification indicator exceeds the corresponding preset threshold, triggering local reconstruction, otherwise self-correcting by the unit layer.

3. The method according to claim 1, characterized in that, The resource capability characteristics cover resource type, access location, and available adjustment service types, specifically including adjustment direction set, dynamic response latency, persistence capability, recovery time, available capacity, user constraints, communication reliability, and historical reliability, which are used to characterize the adjustment capability and execution feasibility of resources participating in dynamic complementary alliances. And / or, the weak points include any of the following: voltage weak points, overload weak points, net load ramping weak points, reverse power flow weak points, and insufficient reserve weak points; And / or, the task segment features include the task action location, task time window, power demand range, adjustment direction, effective time limit, duration, required minimum capacity, and safety boundary set corresponding to each vulnerability.

4. The method according to claim 1, characterized in that, The task influence domain is calculated based on task fragment features, including: Based on the characteristics of task segments, the current availability of each resource is determined. Based on the current availability of each resource, the task influence domain is dynamically generated according to the electrical sensitivity, electrical distance, and historical validity update items between the vulnerability and each resource over time, as shown below: ； ； in, For the current moment The task's influence domain For the future time domain, For the time in the rolling time domain Electrical sensitivity; Electrical distance; This is the distance attenuation coefficient; This is the distance decay kernel function; This is a historical validity item that updates over time; The current availability gate function for the resource; , , As weight; This is an indicator function; it takes the value 1 if the condition is met, and 0 otherwise. Representing resources At any moment The set of adjustment directions, Indicates the first Adjusting the direction of each task; For resources At any moment Dynamic response latency, For the first The task at time The effective period; For resources At any moment Available capacity For the first The task at time Minimum required capacity.

5. The method according to claim 1, characterized in that, Within the mission's influence domain, the alliance layer selects candidate dynamic complementary alliances from resource capability characteristics according to a preset method, including: a preset method of two-level quantitative screening and complementary quantitative combination, specifically including: Within the task's influence domain, a first screening is performed based on the basic matching degree between resource capability characteristics and task segment characteristics. A second screening is performed based on the matching degree of the operational boundary. Finally, the screened resources are divided into fast-response short-duration resources and slow-response long-duration resources according to dynamic response latency and sustainability. Complementarity is achieved through complementarity quantification to form resource combinations. Resource combinations with a comprehensive credibility higher than the threshold are retained to form candidate dynamic complementary alliances.

6. The method according to claim 1, characterized in that, The task layer utilizes the PPO reinforcement learning decision model to optimize candidate dynamic complementary alliances and output the target dynamic complementary alliance, including: The task layer calculates a comprehensive evaluation value based on the task coverage, execution reliability, and perturbation level of each candidate dynamic complementary alliance; the comprehensive value of each candidate dynamic complementary alliance and the current system state are input into the PPO reinforcement learning decision model to generate the target dynamic complementary alliance; The comprehensive evaluation value is calculated based on the task coverage, execution reliability, and disturbance level of each candidate dynamic complementary alliance, including: ； in, For comprehensive evaluation, For task coverage, To ensure credibility, Scoring for disturbances; , and These are the weighting coefficients; The system state is represented as follows: ， in, Indicates time The system status; For a moment Resource capabilities characteristics For a moment The task segment features corresponding to the kth weakness. For a moment Power grid operation status data.

7. The method according to claim 1, characterized in that, The executable task envelope includes active power adjustment range, reactive power adjustment range, start time limit, hold time, ramp level, reserve margin and fallback priority; the task layer sends the task envelope to the unit layer instead of fixed equipment settings. And / or, the local strategy template includes at least one of the following: energy storage strategy template, inverter-type distributed power generation strategy template, electric vehicle aggregation strategy template, flexible load strategy template, and reactive power regulation equipment strategy template.

8. The method according to claim 2, characterized in that, The actual response results include the actual active power response, the actual reactive power response, and the actual response delay. The overall execution deviation is calculated based on the deviation between the actual response result and the executable task envelope, and is expressed as follows: ， ， in, To comprehensively account for execution deviations, , , These are active power deviation, reactive power deviation, and time deviation, respectively. , , These are the weights for active power deviation, reactive power deviation, and time deviation, respectively. , The actual active power response and actual reactive power response of the corresponding alliance or equipment for the task segment; , The active and reactive targets are used as references for the task envelope requirements; , The normalized benchmark for the reference quantity; This refers to the actual response time. The task requires a specific timeframe for taking effect; To prevent the use of tiny positive numbers with a denominator of zero; And / or, the sudden event includes at least one of the following: topology switching caused by switching action, sudden change in source load prediction, energy storage state approaching the boundary, change in user response intention, and deterioration in communication quality; And / or, triggering local reconstruction means: returning to execute S1 to S4, and only performing local reconstruction on the affected task fragment features and corresponding dynamic complementary alliances.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 8.

Images