A micro-grid optimization scheduling method, device, equipment and medium

Abstract

The application discloses a kind of micro-grid optimization scheduling method, device, equipment and medium, it is related to micro-grid dispatching technical field, the physical operation mechanism of micro-grid, time sequence logic and safety constraint are encoded as the transition trigger condition and the library of Petri net is updated rule, to build TCPN model, through multi-scenario Monte Carlo simulation generates covering extensive working condition, strict physical safety, contains complete state-action-reward trajectory High-quality offline data set, solve the problem of insufficient data coverage from data source;Then, double conservative Q-learning algorithm DC-CQL is used as the core of strategy learning, and the two complementary paths of explicit punishment out-of-distribution action and implicit pessimistic value estimation are used to suppress distribution deviation error, to build a conservative constraint for the whole link from current value estimation to target value calculation, and to solve the defects at data level and algorithm level, to promote the practical engineering landing of reinforcement learning in power system.

Description

Technical Field

[0001] This invention relates to the field of microgrid dispatching technology, and in particular to a microgrid optimization dispatching method, apparatus, equipment and medium. Background Technology

[0002] With the deepening of the global sustainable energy transition, microgrids, as an important platform for integrating distributed power sources, flexible loads and energy storage systems, aim to achieve comprehensive optimization in terms of multi-dimensional indicators such as operating economy and power supply reliability by rationally formulating the output plans of each controllable unit. This scheduling problem is essentially a complex sequential decision problem in a high-dimensional continuous state space.

[0003] Current optimization scheduling schemes mainly rely on deep reinforcement learning mechanisms, enabling agents to automatically learn scheduling strategies through interactive trial and error. However, in microgrids, which are power infrastructures with extremely high operational safety requirements, online trial and error of Markov decision processes may lead to safety accidents. Therefore, the mainstream approach at present is offline reinforcement learning.

[0004] Currently, the mainstream approach to offline reinforcement learning typically employs the CQL algorithm, which reduces the Q-value of out-of-distribution actions by adding a regularization term to the commentator network's loss function. However, the regularization term in the CQL algorithm only applies to the current Q-value output of the commentator network, employing a single-dimensional conservative strategy. When facing the distribution shift problem in microgrid offline datasets, the distribution shift error passes through two key nodes in the offline reinforcement learning value learning chain: the first node is the commentator network's static value estimate output for the current state-out-of-distribution action pair, and the second node is the dynamic propagation of the next state objective value in the Bellman objective calculation during temporal difference updates. This single conservative strategy can only act on one node, while the error will continuously infiltrate through the other node, gradually accumulating and amplifying during value iteration. This results in the convergence strategy's conservatism being significantly lower than designed, ultimately making efficient microgrid optimization and scheduling difficult. Summary of the Invention

[0005] This invention provides a microgrid optimization scheduling method, apparatus, equipment, and medium, which can solve the problems existing in the prior art.

[0006] This invention provides a microgrid optimized scheduling method, comprising the following steps: Obtain an offline scheduling trajectory dataset of a microgrid system during its historical operation. Each frame sample in the dataset contains the system state, action space, reward, and next state. Based on the actor-critic reinforcement learning framework, a scheduling policy network model is constructed on the basis of the conservative Q-learning algorithm CQL. The model includes an actor network, a target actor network, an ensemble critic network, and a target critic network. The scheduling policy network model is trained based on an offline scheduling trajectory dataset. In each training round, the ensemble critic network samples out-of-distribution actions from the action space through a random action sampler. It extracts the difference between the Q-values ​​of in-distribution and out-of-distribution actions to suppress the value estimation of out-of-distribution actions and impose an explicit value penalty. At the same time, the target actor network generates the target action according to the next state. Multiple independent Q-networks in the target critic network calculate the target Q-value, and the minimum value is taken as the final target value for implicit estimation. The update loss of the ensemble critic network and the actor network are calculated separately to update their parameters in reverse until the loss function of the ensemble critic network converges to a preset threshold, thus obtaining the trained scheduling policy network model. The current system state of the microgrid is input into the trained scheduling strategy network model, and the optimal scheduling action is obtained through forward computation to achieve optimized scheduling of the microgrid.

[0007] Preferably, the acquisition of the offline scheduling trajectory dataset includes: Based on the physical topology and operating mechanism of the microgrid system, a time-colored Petri net TCPN model for microgrids is constructed. The TCPN model adopts a two-layer architecture of top-level model and bottom-level model and embeds physical security constraints. The top-level model is composed of an initialization module, a traditional scheduling module, and a demand response module. The initialization module is used to generate global time tokens and initialize photovoltaic output, wind power output, load demand, and electricity price information. The traditional scheduling module is used to aggregate new energy output, generate dynamic electricity prices, execute energy storage charging and discharging control, and encapsulate state transition data. The demand response module drives the transfer and restoration operations of transferable loads based on electricity price thresholds. The physical security constraints include system power balance constraints, unit output limiting constraints, and energy storage charge state boundary constraints. With only the initialization module and the traditional scheduling module enabled, the probability distribution of dynamic electricity prices within the microgrid is simulated through Monte Carlo experiments. The density peak clustering algorithm is used to extract low-to-medium electricity price thresholds and medium-to-high electricity price thresholds. The two electricity price thresholds are then embedded into the demand response module to form a complete TCPN model. A complete TCPN model is run based on parameter information of the microgrid system during its historical operation to simulate the dynamic scheduling process of the microgrid system under preset constraints and random factors. The state-action-reward-next state quadruple data generated during the scheduling process is stored in a database. Through repeated simulations in multiple scenarios, an offline scheduling trajectory dataset covering multiple operating conditions is generated.

[0008] Preferably, the formation of the complete TCPN model includes: Under the condition that only the initialization module and the traditional scheduling module are enabled, but the demand response module is not enabled, the TCPN model is repeatedly run under various new energy output and load fluctuation scenarios. The probability distribution generation process of dynamic electricity price within the microgrid is simulated through Monte Carlo experiments. Electricity price data samples generated from the Monte Carlo experiment were collected, and density peak clustering algorithm was used to perform cluster analysis on the electricity price data samples. By calculating the local density and distance characteristics of each electricity price data sample point, low electricity price cluster centers, medium electricity price cluster centers and high electricity price cluster centers formed based on preset thresholds in the electricity price distribution were identified. Based on low-electricity-price cluster centers, medium-electricity-price cluster centers, and high-electricity-price cluster centers, a first threshold for distinguishing between low and medium-electricity prices and a second threshold for distinguishing between medium and high-electricity prices are extracted. The first and second thresholds are then embedded into the transition triggering conditions of the demand response module to form a complete TCPN model.

[0009] Preferably, the simulation generates an offline scheduling trajectory dataset covering multiple operating conditions, including: In each simulation, the complete TCPN model takes the current microgrid system operating state as input. The traditional scheduling module and the demand response module jointly decide the current scheduling action based on the current electricity price and system state. After the scheduling action is executed, it triggers the system state to transition to the next time step. At the same time, the instant reward value corresponding to this state transition is calculated according to the pre-built reward function, forming a quadruple sample including the current state, the selected action, the instant reward, and the next state. By setting multiple random scenarios and multiple initial conditions, multiple batches of simulated scheduling are repeatedly executed to generate a quadruplet sample set covering multiple operating conditions. Based on the quadruplet sample set, an offline scheduling trajectory dataset covering multiple operating conditions is formed.

[0010] Preferably, the actor network is used to output scheduling actions based on the current input state of the microgrid system; the target actor network, as a delayed update copy of the actor network, is used to provide target actions when calculating the target value. The integrated critic network consists of multiple Q networks with identical structures but independently initialized parameters, operating in parallel. Each Q network receives state and action as input and outputs a corresponding action value estimate. The target critic network is a delayed-updated copy of each Q network in the integrated critic network, used to provide a value benchmark for target value calculation.

[0011] Preferably, the working mechanism of the explicit value penalty and implicit estimation includes: In explicit value penalty, the scheduling policy network model calls a random action sampler to independently sample a set of out-of-distribution action candidates from the action space of the offline scheduling trajectory dataset in each round of training. The integrated critic network calculates the Q-values ​​corresponding to the in-distribution actions in the offline scheduling trajectory dataset and the randomly sampled out-of-distribution actions in the current state, respectively, to calculate the explicit penalty loss. In implicit estimation, during each training round of the scheduling policy network model, the target actor network generates the target action based on the next state, and all target critic networks estimate the value of the combination of the next state and the target action. The minimum value among multiple estimates is selected as the final target value for updating the current Q network to calculate the implicit estimation loss.

[0012] Preferably, the step of calculating the update losses of the integrated critic network and actor network respectively to update their parameters in reverse includes: Obtain the mean squared error loss between the estimate of the current state-action pair and the target value of each Q network in the integrated critic network, and sum the mean squared error loss and the explicit penalty loss using preset weight coefficients to obtain the update loss of the critic network. The mean squared error loss drives the ensemble critic network to continuously improve the value estimation accuracy of each Q network for in-distribution samples in the offline dataset, and the explicit penalty loss drives the ensemble critic network to suppress the value estimation of out-of-distribution actions in order to update the parameters of the critic network through gradient backpropagation. The update loss of the actor network is an implicit estimation loss. The minimum value among the value estimates of the combination of the next state and the target action of all Q networks in the target commentator network is used as the evaluation index. The parameter update direction of the actor network is configured to maximize this minimum value so as to update the parameters of the actor network through gradient backpropagation.

[0013] This invention also provides a microgrid optimized scheduling device, comprising: The data module is used to acquire offline scheduling trajectory datasets of the microgrid system during its historical operation. Each frame sample in the dataset contains system state, action space, reward, and next state. The model building module is used to construct a scheduling policy network model based on the actor-critic reinforcement learning framework and the conservative Q-learning algorithm CQL. The model includes an actor network, a target actor network, an ensemble critic network, and a target critic network. The model training module is used to train the scheduling policy network model based on the offline scheduling trajectory dataset. In each round of training, the ensemble critic network samples out-of-distribution actions from the action space through a random action sampler, extracts the difference between the Q values ​​of in-distribution and out-of-distribution actions to suppress the value estimation of out-of-distribution actions and impose an explicit value penalty. At the same time, the target actor network generates the target action according to the next state. Multiple independent Q networks in the target critic network calculate the target Q value, and take the minimum value as the final target value for implicit estimation. The update loss of the ensemble critic network and the actor network are calculated separately to update their parameters in reverse until the loss function of the ensemble critic network converges to a preset threshold, and the trained scheduling policy network model is obtained. The scheduling module is used to input the current system state of the microgrid into the trained scheduling strategy network model, and obtain the optimal scheduling action through forward calculation to achieve optimized scheduling of the microgrid.

[0014] This invention also provides an electronic device, including a memory and a processor; The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the steps of the microgrid optimization scheduling method described above.

[0015] This invention also provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of a microgrid optimization scheduling method as described above.

[0016] This invention provides a microgrid optimized scheduling method, apparatus, equipment, and medium, which have the following advantages compared with the prior art: This invention introduces a dual conservative mechanism based on the classic conservative Q-learning algorithm (CQL). ​​The explicit penalty mechanism works by using a regularization term in the loss function to force the ensemble commentator network to systematically suppress Q-value estimates for out-of-distribution actions while maintaining reasonable Q-value estimates for in-distribution actions. This directly suppresses the tendency for out-of-distribution actions to be assigned unreasonably high values. The implicit pessimistic estimation mechanism works by recognizing that when multiple Q-networks disagree on the value estimates of the same state-action pair, the smallest estimate is least likely to exceed the true value, thus having the lowest risk of extrapolation error. Therefore, through explicit penalty and implicit constraint, dual conservative barriers are applied at the value estimation output and target value calculation ends, respectively. Even if the explicit penalty fails to completely eliminate the overestimation in the current Q-value, the implicit constraint can still cut off the signal from entering the next iteration at the target end. This dual conservative mechanism effectively blocks the continuous accumulation of distribution offset errors in value iteration, making the strategy convergence point closer to the true optimal value rather than the deviation direction, ultimately achieving efficient microgrid optimization scheduling.

[0017] Furthermore, this invention constructs a time-colored Petri net TCPN model for microgrids, using the TCPN model as a data generation engine. The physical operation mechanism, timing logic, and all security constraints of the microgrid are encoded into transition triggering conditions and place update rules for the Petri net. Through a multi-scenario Monte Carlo simulation system, a high-quality offline dataset is generated that covers a wide range of operating conditions, has strict physical security, and contains complete state-action-reward trajectories. This addresses the problem of insufficient coverage of existing historical datasets due to data privacy constraints and limited operating conditions at the data source. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall functional architecture of the TCPN model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the top-level microgrid TCPN model architecture provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the underlying TCPN model architecture of the initialization module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the underlying TCPN model of a traditional scheduling module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the overall execution framework provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the overall algorithm structure provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] The microgrid scheduling problem is essentially a high-dimensional stochastic optimization problem. Its core difficulty lies in the strong coupling between multidimensional constraints and uncertainties on both the source and load sides. Traditional optimization methods based on physical models, including mixed integer linear programming (MILP) and stochastic programming (SP), face significant limitations: these methods rely heavily on accurate system models and have high computational complexity. When dealing with strongly coupled and highly dynamic real-time scheduling scenarios, they are difficult to simultaneously meet the requirements of reliability, optimality, and timeliness in decision-making.

[0021] The aforementioned challenges have driven the development of data-driven approaches. Among them, learning-based control frameworks, as a cutting-edge approach to solving high-dimensional stochastic optimization problems, can automatically learn complex mappings and decision-making strategies from data without the need for precise physical models. Among various learning paradigms, reinforcement learning (RL) has received widespread attention in the fields of scheduling and energy management because it is naturally applicable to sequential decision-making problems.

[0022] Early research mainly employed tabular reinforcement learning methods, such as Monte Carlo-based generalized policy iteration and multi-agent Q-learning. However, these methods present a fundamental trade-off between modeling accuracy and computational feasibility. Breakthroughs in deep reinforcement learning (DRL) have utilized function approximators to process continuous spaces, and advanced algorithms such as DDPG, PPO, and SAC have been widely applied to microgrid energy management. However, most of these methods belong to the online reinforcement learning paradigm, whose core mechanism is that agents learn policies through real-time trial-and-error interactions with the environment.

[0023] While online learning can achieve satisfactory performance, its inherent drawback lies in requiring agents to explore and occasionally perform risky actions. In the field of power systems, where operational safety is paramount, this interactive learning mechanism is often considered unacceptable due to the associated operational risks and potential costs. Although simulation environments can replace physical systems, they often suffer from insufficient fidelity or excessive computational costs. Therefore, the inherent safety and cost issues of online trial-and-error learning have become the main bottlenecks restricting the practical deployment of reinforcement learning in critical systems such as microgrids.

[0024] To mitigate the risks and costs associated with online interaction, the Offline Reinforcement Learning (OfflineRL) paradigm has emerged. This paradigm learns policies solely from pre-collected static datasets, completely avoiding real-time exploration. Theoretically, this method has the potential to prevent trial-and-error, but its performance is highly dependent on the quality and coverage of the offline dataset.

[0025] In microgrid dispatching scenarios, existing historical datasets often lack sufficient coverage due to data privacy constraints and limited operating conditions, failing to adequately represent extreme operating conditions or potentially dangerous states. Agents trained on such incomplete datasets may make unreliable or even unsafe decisions when encountering unseen states after deployment.

[0026] Petri nets, as a formal modeling tool for discrete event dynamic systems, have been successfully applied in behavioral modeling and performance analysis of energy systems such as microgrids. This model-driven simulation method can systematically generate datasets with embedded operational constraints and covering diverse states, providing a promising approach to solving the problem of generating high-quality training data for offline reinforcement learning.

[0027] However, existing research has the following shortcomings: 1. Petri nets are mainly used for system analysis or simple integration with online learning, and their potential as a dedicated generator of secure constraints and high-fidelity synthetic data has not been fully explored.

[0028] 2. Petri nets' modeling capabilities are not deeply coupled with the policy optimization requirements of offline reinforcement learning, and a complete "modeling-data-policy" closed-loop framework has not yet been formed.

[0029] 3. Existing offline reinforcement learning algorithms (such as BCQ, TD3+BC, IQL, CQL, etc.) have shortcomings in dealing with distribution offset problems. They either have difficulty effectively suppressing value overestimation or are not robust to suboptimal trajectories in the dataset, and cannot achieve balanced and excellent comprehensive performance in the multi-objective optimization of microgrid scheduling.

[0030] To address the shortcomings of current solutions, this invention proposes a microgrid optimization scheduling method combining time-colored Petri nets and offline reinforcement learning, specifically including: I. Construction of the TCPN model for microgrids.

[0031] This invention constructs a Timed Colored Petri Net (TCPN) model for microgrids, and its overall functional architecture is as follows: Figure 1 As shown, the model integrates the actual operating mechanism and constraints of the microgrid to ensure accurate simulation of the microgrid's operating characteristics. To meet the benchmark requirements of reinforcement learning, a Markov decision process modeling framework is further embedded. The model takes time series data as input and generates a sample data pool containing electricity price distribution and offline datasets required for offline reinforcement learning training after TCPN model computation.

[0032] 1.1 Structure of the microgrid TCPN model.

[0033] The model adopts a top-bottom two-layer architecture, with the top layer structure as follows: Figure 2 As shown, the top-level model is constructed by three core modules that work together to build the microgrid scheduling framework: the initialization module, the traditional scheduling module, and the demand response module. The GlobalTime of the storage place serves as the initial storage place and drives the orderly scheduling simulation of the model by maintaining a global time token. After the scheduling process is completed, the Transitions of the terminating storage place will store the processed transition sequence to provide input data for the subsequent training phase. It should be noted that the double rectangle symbol in the initialization module and the traditional scheduling module represents the replacement transition, which encapsulates the underlying sub-model that ensures the stable operation of the model.

[0034] 1.1.1 Initialize the module.

[0035] The underlying TCPN model of the initialization module is as follows: Figure 3As shown; when transition T1 is triggered, the global time token in the place GlobalTime will be propagated to places INT1-5, forming a local time base that controls the orderly triggering of subsequent transitions; after transitions T2, T3, T4, T5 and T10 are triggered, their output places PV, WT, Price, Load and TR will obtain the initial information to participate in the scheduling.

[0036] In transition T2, the integrated formulas (1)-(4) will generate time on the place PV after the transition is triggered. t The output of the photovoltaic array below.

[0037] (1) (2) (3) (4) in: The output power of photovoltaics, S t This is the basic curve for photovoltaic output power. V t This is the random correction factor for irradiation. For abnormal Bernoulli variables, The rated power of the photovoltaic system. τ After the integrated formulas (5)-(10) in the fault state output coefficient transition T3 are triggered, the output status of the wind turbine at time t will be generated on the warehouse WT.

[0038] (5) (6) (7) (8) (9) (10) in: The output power of the fan. C t This is the basic curve for the output power of the wind turbine. F t For random fluctuations, G t Low probability gain term, B For abnormal Bernoulli variables, ε Tiny perturbations, and These are the maximum and minimum output power of the fan, respectively.

[0039] Transition T4 controls the time-of-use pricing set by the upper-level distribution network for the microgrid; in transition T5, the integrated formulas (11)-(13) will generate the load status at time t after triggering the load on the storage area; the average load level on weekends is greater than that on weekdays, and the average load level on weekends is greater than that on weekdays. d = 1, working day d =0.

[0040] (11) (12) (13) in: Let t be the load level of the microgrid. For the amplitude limiting function, For load deterministic reference components, The random disturbance term follows a truncated normal distribution. L min and L max These represent the maximum and minimum load levels, respectively.

[0041] 1.1.2 Demand Response Module.

[0042] Demand response mechanisms can be divided into two categories: incentive-based and price-driven. Incentive-based methods guide users to reduce load during specific periods through direct compensation, while price-driven methods rely on dynamic pricing strategies to regulate consumption behavior and have higher operational flexibility. Therefore, this invention designs a dynamic pricing strategy as shown in formula (14). In scenarios where renewable energy output is insufficient and energy storage capacity is limited, the dynamic electricity price within the microgrid guides users to implement load transfer, thereby alleviating the system's power supply pressure.

[0043] To achieve orderly management of transferred loads, a load buffer pool was constructed to store loads awaiting recovery, such as... Figure 2 As shown in the Ltr of the central storage; the capacity status of the load buffer pool and the storage amount of transferred load are dynamically calculated by formula (15) and formula (16); considering the actual physical limitations, the proportion of transferable load is constrained by the maximum transfer rate defined by formula (17), which is initialized when the transition T10 in the initialization module is triggered; when the transition T8 in the module is triggered, the user performs load transfer or recovery operation according to formula (14); the system transferred load and the actual load are represented by formula (18) and formula (19) respectively.

[0044] (14) (15) (16) (17) (18) (19) in: p renew Output power for new energy sources p renew = p pv + p wt , The upper limit of the transfer ratio, To transfer the load, u t Transfer enable indicator function, For microgrid dynamic electricity pricing, For the load storage pool capacity, SOT Transfer load capacity status, This is the actual load. c t For the proportion of load transfer, c r To restore the load ratio.

[0045] 1.1.3 Traditional scheduling module.

[0046] The underlying model of a traditional scheduling module is as follows: Figure 4 As shown; when transition T6 is triggered, the output power of photovoltaic and wind turbines is aggregated to generate the total power of new energy and stored in the Renew storage; then transition T7 is triggered, and a dynamic electricity price signal is generated based on this power value and input to the demand response module to coordinate user load transfer behavior; when the demand response module transmits the actual load data to the replacement transition Termination, transition T9 in the underlying model is triggered to activate the load smoothing function of the energy storage system; at this time, the remaining energy storage capacity and charge status are updated in real time according to formula (20) and formula (21); finally, transitions T11-T14 encapsulate the state transition data of the scheduling process into standard tuple form ( s t , a t , r t , s t+1 The training sample pool for offline reinforcement learning is constructed by storing the samples in the Transitions. The net load size after energy storage and smoothing is shown in formula (22).

[0047] (20) (twenty one) (twenty two) in: For energy storage t The capacity at time +1 For energy storage t Capacity of time, c b The energy storage and discharge ratio, SOC t The state of charge of energy storage. For energy storage capacity, This refers to the net load size.

[0048] In the early stages of reinforcement learning training, agents often generate action policies that violate system safety constraints. As the training progresses, the real-time reward mechanism gradually guides the policy to converge to the safe action space. However, offline reinforcement learning, due to the lack of real-time feedback and the inherent overestimation bias of Q-learning, is prone to induce agents to perform actions that endanger system safety. To this end, this invention strictly embeds physical safety constraints in the TCPN modeling process, including system power balance, unit output limits, and hard boundaries of energy storage, as shown in formulas (23) to (28). This mechanism ensures that the generated dataset has physical executability, fundamentally eliminating the pollution of training data by dangerous actions.

[0049] (twenty three) (twenty four) (25) (26) (27) (28) 1.2 Markov Decision Process.

[0050] In Markov decision-making processes, sequential decision problems are solved using quadruples (...). S , A , R , S’ )express, S Let be a set of states, where A For action sets, R ( S , A , S’ The reward function is denoted as ; microgrid scheduling is essentially a high-dimensional stochastic sequential decision problem, and its core elements can be mapped as follows within the Markov decision process framework: State Space: The state space is defined as a seven-tuple, containing the current time, renewable energy output power, load demand, microgrid dynamic electricity price, load buffer capacity status, energy storage system state of charge, and net load; represented as: (29) Action Space: The action space is defined as a triplet, including load transfer ratio, load recovery ratio, and energy storage discharge ratio, expressed as: (30) Reward function: As the core guiding mechanism of reinforcement learning, its design quality directly determines the performance limit of the agent's policy; in the optimization scheduling problem, the system objective function and the reinforcement learning reward function are functionally equivalent, and both continuously drive the control policy to evolve toward the desired goal by quantitatively evaluating the system state; this invention establishes a mathematical mapping from the scheduling objective to the reward function to ensure the direct correspondence between policy optimization and the scheduling objective, and the specific implementation is given by formula (31)-formula (34).

[0051] (31) (32) (33) (34) In formula (31), the reward function r It consists of three weighted components, among which n 1, n 2, n 3 represents the weighting coefficient; Characterizing the user's electricity purchase cost; the user's actual load after load transfer / restoration. Power is supplied by a microgrid, and users need to pay according to dynamic electricity prices. The payment fee is considered as the cost of electricity, and this cost is the first part of the reward function, representing a core economic indicator. This is a load transfer penalty. Users are subject to penalties based on dynamic electricity pricing. Perform load transfer / restoration; when dynamic electricity prices are in effect. When the dynamic electricity price is high, users will perform load shifting. If the dynamic electricity price remains high for a long time, the load storage pool capacity will surge, and the load waiting to be restored will increase. Once the dynamic electricity price decreases, the large amount of restored load will increase the pressure on the microgrid system. If the proportion of restored load is reduced, too much load will not be restored, which will cause user dissatisfaction. Therefore, the second part of the reward function is a load shifting penalty term, which is used to constrain excessive shifting operations to maintain user satisfaction. This represents the net load penalty. The independent operation of a microgrid is an extremely important indicator. When the main grid fails, the microgrid needs to operate independently to maintain power supply to local loads; therefore, a net load penalty is set. In this microgrid, after users purchase electricity, the microgrid needs to assume the responsibility of supplying power to them. Besides uncontrollable energy sources like solar and wind turbines, the peak-shaving and valley-filling function of energy storage is a key link in maintaining system power balance and achieving independent operation of the microgrid. Energy storage minimizes the net load by suppressing load curve fluctuations after wind and solar power supply, thus reducing the main grid's power support to the microgrid. By reducing the net load after energy storage smoothing, the dependence on the main grid is reduced, improving microgrid independence and indirectly increasing the microgrid's utilization rate of new energy sources, achieving the "dual carbon" goal.

[0052] 1.3 Model execution mechanism.

[0053] To ensure accurate modeling and data-driven operation of the microgrid dispatching process, the TCPN model proposed in this invention is implemented through two sequential stages: the first stage establishes a data-driven electricity price threshold through Monte Carlo simulation, and the second stage uses the complete TCPN model to generate operation trajectory data for offline reinforcement learning.

[0054] 1.3.1 Monte Carlo experiment.

[0055] To accurately characterize the electricity price-based load response behavior in the demand response module, it is necessary to define key thresholds that distinguish between high, medium, and low electricity price levels. The dynamic electricity price within the microgrid is only affected by the output of new energy sources, the state of charge of energy storage, and the time-of-use pricing of the main grid. Therefore, in this stage, under the action of the initialization module and the traditional scheduling module, the electricity price generation process is simulated through a large-scale Monte Carlo experiment to analyze the probability distribution characteristics of electricity prices. Subsequently, the density peak clustering algorithm is used to process the data. By calculating the local density and distance characteristics of sample points, the cluster centers of low, medium, and high electricity prices are identified, and the key thresholds that distinguish between low-medium and medium-high electricity prices are extracted accordingly. These thresholds are then embedded into the demand response module to finally complete the construction of the complete TCPN model.

[0056] 1.3.2 Simulated scheduling.

[0057] Based on the constructed complete TCPN model, a multi-scenario scheduling simulation of microgrids is performed. This process systematically simulates the dynamic operation of microgrids under complex constraints and random factors, generating trajectory data covering rich state-action-transition relationships, providing a high-quality dataset for subsequent offline reinforcement learning training.

[0058] II. Overall Framework Design.

[0059] After establishing the TCPN model and execution mechanism, this invention integrates the overall architecture and method of the proposed timed colored Petri net and offline reinforcement learning ensemble framework. First, the structure and execution flow of the framework are explained, and then the dual conservative Q-learning algorithm used by the framework is described in detail.

[0060] 2.1 Framework Structure.

[0061] The microgrid optimization framework proposed in this invention is as follows: Figure 5 As shown, the framework comprises five core implementation steps. It is optimized for grid-connected microgrid systems, encompassing photovoltaic (PV) power generation, wind power generation, battery energy storage systems, and price-driven demand response, and achieves power interaction with the main grid through a point of common coupling. The specific implementation process is as follows:

[0062] 1) System Modeling①: Based on Petri net theory, a timed colored Petri net dynamic model of the microgrid is constructed on the CPN TOOLS platform, including an initialization module, a traditional scheduling module, and a demand response module. In the demand response module, transferable loads are transferred and scheduled according to the internal electricity price of the microgrid. To accurately simulate the user load transfer behavior, a dynamic electricity price threshold parameter is preset in the module.

[0063] 2) Electricity Price Clustering Analysis and Model Integration ②③④: Under the premise of only considering the role of the initialization module and the traditional scheduling module in the timed colored Petri net model, the probability distribution characteristics of the electricity price inside the microgrid are obtained through Monte Carlo experiments. The density peak clustering algorithm is used to extract the optimal threshold parameter from the distribution and embed it into the demand response module to complete the construction of the complete colored Petri net model.

[0064] 3) Offline data generation ⑤: Run a complete high-level timed colored Petri net model to simulate the entire microgrid scheduling process; the standardized operational data is stored in the storage state transition of the traditional scheduling module, and the microgrid offline operational dataset covering multiple operational scenarios is generated by repeatedly simulating the scheduling process.

[0065] 4) Offline Training⑥: Mini-batch sampling is performed on the generated offline dataset to train the offline reinforcement learning algorithm; the algorithm learns the optimal scheduling policy by iteratively updating the parameters of the evaluation network and the policy network. The specific algorithm details will be elaborated in a later section.

[0066] 5) Deployment and online operation of intelligent agents ⑦⑧: The trained policy network is used as a microgrid scheduling intelligent agent and directly applied to the optimized operation of the actual microgrid system, realizing a complete closed loop from simulation training to actual application.

[0067] 2.2 Double Conservative Q-Learning Algorithm.

[0068] In the aforementioned optimization framework, the offline reinforcement learning training phase begins after offline dataset collection. The core challenge of offline reinforcement learning is the distribution shift problem, the most serious consequences of which are value overestimation and extrapolation error, ultimately causing the learned policy to fail in actual deployment. To address this issue, this invention proposes a dual conservative Q-learning algorithm (DC-CQL). ​​This algorithm introduces a dual conservative mechanism based on the classic conservative Q-learning algorithm (CQL). ​​On the one hand, it generates out-of-distribution actions through a random action sampler and applies explicit penalties; on the other hand, it constrains the value function update direction through the pessimistic estimation of the integrated target network. The proposed dual conservative Q-learning algorithm is based on an actor-critic framework, where the actor network adopts a master-slave dual-network architecture, and the critic network adopts an integrated dual-network architecture, as shown in the specific structure below. Figure 6 As shown.

[0069] 1) Actor Network: Input is the current state s The output is the strategy action. Its parameter updates are guided by value estimates provided by a network of critics.

[0070] 2) Target Actor Network: A delayed update copy of the actor network. Output stable motion This avoids the rapid fluctuations in online strategies affecting the calculation of target value.

[0071] 3) Critics Network: by N An independent Q network Q 1, ..., Q N Composition, input state and action ( s , a Output the corresponding action value estimate. Q i ( s , a (This can be achieved by integrating multiple networks to improve the robustness of value estimation.)

[0072] 4) Target Critics Network: A delayed-update copy corresponding one-to-one with each Q network in the critics network. ,..., The parameters are kept stable through soft updates, providing a benchmark for the calculation of the target value.

[0073] The core advantage of the dual conservative Q-learning algorithm lies in its dual conservative mechanism that combines explicit penalties with implicit constraints. This mechanism achieves complementary error suppression through collaborative design. The explicit mechanism is specifically designed to address the overestimation problem of out-of-distribution actions, while the implicit mechanism focuses on the stability of target value estimation. Together, they construct a conservative constraint on the entire chain from current value estimation to target value calculation.

[0074] 0) Explicit Conservative Term: The explicit conservative mechanism directly suppresses the overestimation of the value of out-of-distribution actions through random action sampling and penalty term design. Its core idea is to enhance the sensitivity of the value function to in-distribution actions while maintaining a conservative attitude towards out-of-distribution actions. Specific implementation: A random action sampler (RAS) is used, which is independent of the dataset. From the action space A Mid-sampling action In a continuous action space, actions sampled in this way have a high out-of-distribution probability; the ensemble critic network separately samples in-distribution actions. a and sampling action Calculate the Q value and suppress out-of-distribution actions using the explicit penalty term defined by formula (35). Value estimate.

[0075] (35) 6) Implicit Conservative Term: The implicit conservative mechanism achieves pessimistic value estimation by integrating the minimum operation of the target network. Its core idea is to quantify uncertainty by utilizing the differences between multi-network estimates. When multiple independent Q-networks disagree on the value estimates of the same action, the minimum value is taken as the target value. Its essence lies in adopting the most conservative consensus estimate. Specifically, the target actor network generates actions. Subsequently, each target commentator network independently calculated its value estimate. Ultimate goal value y It is generated by the minimum value operation as shown in formula (36).

[0076] (36) Therefore, the update loss function for the critic network and the actor network is defined as: (37) (38) The parameter update rules for the critic network and the actor network are as follows: (39) (40) in: and Each is a network update for critics and actors regarding their learning rates.

[0077] The target network's parameter updates employ a soft update strategy: (41) (42) in: τ This refers to the soft update rate.

[0078] This invention was verified through numerical simulation experiments, and the effects include: (1) Optimal overall performance.

[0079] On the standard dataset, the cumulative reward of the DC-CQL algorithm reached -4189.83, which is better than all the comparison algorithms such as BCQ (-4441.86), CQL (-4321.83), IQL (-4460.69) and TD3+BC (-5422.04), achieving the best overall scheduling performance. The comparison results are shown in Table 1.

[0080] Table 1. Comparison of DC-CQL algorithm with other algorithms on standard datasets. BCQ -4441.86 -1332.14 -2683.19 -426.52 CQL -4321.83 -1296.89 -2788.48 -236.45 IQL -4460.69 -1391.75 -3068.94 0.00 TD3+BC -5422.04 -1493.83 -2245.36 -1682.84 DC-CQL -4189.83 -1295.92 -2790.04 -103.87 (2) It has outstanding multi-objective equilibrium optimization capabilities.

[0081] DC-CQL maintains competitiveness in three dimensions: economy (r1=-1295.92), load transfer penalty (r2=-2790.04), and system autonomy (r3=-103.87), avoiding the imbalance problem of sacrificing overall performance due to single-dimensional optimization of algorithms such as TD3+BC and IQL. This balanced advantage stems from the synergistic effect of dual conservative mechanisms: explicit penalty for OOD actions and implicit integrated pessimistic estimation complement each other, effectively alleviating overestimation while promoting balanced optimization of multi-objective scheduling.

[0082] (3) The suboptimal data has strong anti-interference ability.

[0083] On the expanded dataset with 50% suboptimal data, DC-CQL achieved an Outperform Demonstrator Accuracy (SDA) of 78.42%, significantly higher than CQL's 36.56% and its own 48.06% on the standard dataset, as shown in Table 2. This indicates that DC-CQL can effectively suppress the interference of suboptimal data, while fully utilizing the expanded state diversity to enhance policy learning, and can maintain or even strengthen the learning effect when the dataset quality changes.

[0084] Table 2 Comparison on the expanded dataset with 50% suboptimal data injected. CQL 38.78% 36.56% IC-CQL 24.54% 39.80% DC-CQL 48.06% 78.42% (4) Dataset generation is efficient and secure.

[0085] The TCPN model generates 160 batches of scheduling data for 5000 steps. Each batch takes 87.8 seconds to compile and 28.9 seconds to execute, for a total computation time of 5.18 hours. The generated dataset is strictly embedded with physical safety constraints and contains no dangerous action samples. Offline training takes 8.6 hours, while online single-step inference takes only 0.2 milliseconds, meeting the requirements for real-time scheduling.

[0086] (5) Eliminate the risk of online trial and error.

[0087] The entire framework is trained entirely on offline data and requires no exploratory interaction with the environment after deployment. This fundamentally eliminates the trial-and-error costs and security risks of online reinforcement learning, providing a feasible path for the practical engineering application of reinforcement learning in safety-critical fields such as power systems.

[0088] The explicit penalty mechanism of this invention suppresses the current Q-value of out-of-distribution actions at the output of the commentator network, while the implicit constraint mechanism suppresses the optimistic component in the target value at the Bellman target calculation end through the minimum operation of the target commentator network. Therefore, even if the explicit penalty fails to completely eliminate the overestimation in the current Q-value due to a weak coefficient setting, the implicit constraint can still cut off the overestimation signal from entering the next iteration at the target end. Conversely, even if the implicit constraint produces a slight estimation deviation due to the lag in target network updates, the explicit penalty can still correct the current value in real time at the output end. The explicit penalty mechanism and the implicit constraint mechanism are redundant and mutually supportive, ensuring that no matter which node in the value chain the optimistic error infiltrates from, it will be suppressed by at least one conservative mechanism, fundamentally preventing the continuous accumulation and gradual amplification of the optimistic error in value iteration.

[0089] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A microgrid optimal scheduling method, characterized in that, Includes the following steps: Obtain an offline scheduling trajectory dataset of a microgrid system during its historical operation. Each frame sample in the dataset contains the system state, action space, reward, and next state. Based on the actor-critic reinforcement learning framework, a scheduling policy network model is constructed on the basis of the conservative Q-learning algorithm CQL. The model includes an actor network, a target actor network, an ensemble critic network, and a target critic network. The scheduling policy network model is trained based on an offline scheduling trajectory dataset. In each training round, the ensemble critic network samples out-of-distribution actions from the action space through a random action sampler. It extracts the difference between the Q-values ​​of in-distribution and out-of-distribution actions to suppress the value estimation of out-of-distribution actions and impose an explicit value penalty. At the same time, the target actor network generates the target action according to the next state. Multiple independent Q-networks in the target critic network calculate the target Q-value, and the minimum value is taken as the final target value for implicit estimation. The update loss of the ensemble critic network and the actor network are calculated separately to update their parameters in reverse until the loss function of the ensemble critic network converges to a preset threshold, thus obtaining the trained scheduling policy network model. The current system state of the microgrid is input into the trained scheduling strategy network model, and the optimal scheduling action is obtained through forward computation to achieve optimized scheduling of the microgrid.

2. The microgrid optimal scheduling method according to claim 1, characterized in that, The acquisition of the offline scheduling trajectory dataset includes: Based on the physical topology and operating mechanism of the microgrid system, a time-colored Petri net TCPN model for microgrids is constructed. The TCPN model adopts a two-layer architecture of top-level model and bottom-level model and embeds physical security constraints. The top-level model is composed of an initialization module, a traditional scheduling module, and a demand response module. The initialization module is used to generate global time tokens and initialize photovoltaic output, wind power output, load demand, and electricity price information. The traditional scheduling module is used to aggregate new energy output, generate dynamic electricity prices, execute energy storage charging and discharging control, and encapsulate state transition data. The demand response module drives the transfer and restoration operations of transferable loads based on electricity price thresholds. The physical security constraints include system power balance constraints, unit output limiting constraints, and energy storage charge state boundary constraints. With only the initialization module and the traditional scheduling module enabled, the probability distribution of dynamic electricity prices within the microgrid is simulated through Monte Carlo experiments. The density peak clustering algorithm is used to extract low-to-medium electricity price thresholds and medium-to-high electricity price thresholds. The two electricity price thresholds are then embedded into the demand response module to form a complete TCPN model. A complete TCPN model is run based on parameter information of the microgrid system during its historical operation to simulate the dynamic scheduling process of the microgrid system under preset constraints and random factors. The state-action-reward-next state quadruple data generated during the scheduling process is stored in a database. Through repeated simulations in multiple scenarios, an offline scheduling trajectory dataset covering multiple operating conditions is generated.

3. The microgrid optimization scheduling method according to claim 2, characterized in that, The formation of the complete TCPN model includes: Under the condition that only the initialization module and the traditional scheduling module are enabled, but the demand response module is not enabled, the TCPN model is repeatedly run under various new energy output and load fluctuation scenarios. The probability distribution generation process of dynamic electricity price within the microgrid is simulated through Monte Carlo experiments. Electricity price data samples generated from the Monte Carlo experiment were collected, and density peak clustering algorithm was used to perform cluster analysis on the electricity price data samples. By calculating the local density and distance characteristics of each electricity price data sample point, low electricity price cluster centers, medium electricity price cluster centers and high electricity price cluster centers formed based on preset thresholds in the electricity price distribution were identified. Based on low-electricity-price cluster centers, medium-electricity-price cluster centers, and high-electricity-price cluster centers, a first threshold for distinguishing between low and medium-electricity prices and a second threshold for distinguishing between medium and high-electricity prices are extracted. The first and second thresholds are then embedded into the transition triggering conditions of the demand response module to form a complete TCPN model.

4. The microgrid optimization scheduling method according to claim 3, characterized in that, The simulation generates an offline scheduling trajectory dataset covering multiple operating conditions, including: In each simulation, the complete TCPN model takes the current microgrid system operating state as input. The traditional scheduling module and the demand response module jointly decide the current scheduling action based on the current electricity price and system state. After the scheduling action is executed, it triggers the system state to transition to the next time step. At the same time, the instant reward value corresponding to this state transition is calculated according to the pre-built reward function, forming a quadruple sample including the current state, the selected action, the instant reward, and the next state. By setting multiple random scenarios and multiple initial conditions, multiple batches of simulated scheduling are repeatedly executed to generate a quadruplet sample set covering multiple operating conditions. Based on the quadruplet sample set, an offline scheduling trajectory dataset covering multiple operating conditions is formed.

5. The microgrid optimal scheduling method according to claim 1, characterized in that, The actor network is used to output scheduling actions based on the current input state of the microgrid system; the target actor network, as a delayed update copy of the actor network, is used to provide target actions when calculating target value. The integrated critic network consists of multiple Q networks with identical structures but independently initialized parameters, operating in parallel. Each Q network receives state and action as input and outputs a corresponding action value estimate. The target critic network is a delayed-updated copy of each Q network in the integrated critic network, used to provide a value benchmark for target value calculation.

6. The microgrid optimization scheduling method according to claim 5, characterized in that, The working mechanism of the explicit value penalty and implicit estimation includes: In explicit value penalty, the scheduling policy network model calls a random action sampler to independently sample a set of out-of-distribution action candidates from the action space of the offline scheduling trajectory dataset in each round of training. The integrated critic network calculates the Q-values ​​corresponding to the in-distribution actions in the offline scheduling trajectory dataset and the randomly sampled out-of-distribution actions in the current state, respectively, to calculate the explicit penalty loss. In implicit estimation, during each training round of the scheduling policy network model, the target actor network generates the target action based on the next state, and all target critic networks estimate the value of the combination of the next state and the target action. The minimum value among multiple estimates is selected as the final target value for updating the current Q network to calculate the implicit estimation loss.

7. A microgrid optimized scheduling method according to claim 6, characterized in that, The step of separately calculating the update losses of the integrated critic network and actor network to update their parameters includes: Obtain the mean squared error loss between the estimate of the current state-action pair and the target value of each Q network in the integrated critic network, and sum the mean squared error loss and the explicit penalty loss using preset weight coefficients to obtain the update loss of the critic network. The mean squared error loss drives the ensemble critic network to continuously improve the value estimation accuracy of each Q network for in-distribution samples in the offline dataset, and the explicit penalty loss drives the ensemble critic network to suppress the value estimation of out-of-distribution actions in order to update the parameters of the critic network through gradient backpropagation. The update loss of the actor network is an implicit estimation loss. The minimum value among the value estimates of the combination of the next state and the target action of all Q networks in the target commentator network is used as the evaluation index. The parameter update direction of the actor network is configured to maximize this minimum value so as to update the parameters of the actor network through gradient backpropagation.

8. A microgrid optimized dispatching device, characterized in that, include: The data module is used to acquire offline scheduling trajectory datasets of the microgrid system during its historical operation. Each frame sample in the dataset contains system state, action space, reward, and next state. The model building module is used to construct a scheduling policy network model based on the actor-critic reinforcement learning framework and the conservative Q-learning algorithm CQL. The model includes an actor network, a target actor network, an ensemble critic network, and a target critic network. The model training module is used to train the scheduling policy network model based on the offline scheduling trajectory dataset. In each round of training, the ensemble critic network samples out-of-distribution actions from the action space through a random action sampler, extracts the difference between the Q values ​​of in-distribution and out-of-distribution actions to suppress the value estimation of out-of-distribution actions and impose an explicit value penalty. At the same time, the target actor network generates the target action according to the next state. Multiple independent Q networks in the target critic network calculate the target Q value, and take the minimum value as the final target value for implicit estimation. The update loss of the ensemble critic network and the actor network are calculated separately to update their parameters in reverse until the loss function of the ensemble critic network converges to a preset threshold, and the trained scheduling policy network model is obtained. The scheduling module is used to input the current system state of the microgrid into the trained scheduling strategy network model, and obtain the optimal scheduling action through forward calculation to achieve optimized scheduling of the microgrid.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the steps of the microgrid optimization scheduling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the steps of a microgrid optimization scheduling method as described in any one of claims 1 to 7.

Images