Aggregation optimization operation method and system for distributed resources of virtual power plant

Abstract

The invention discloses an aggregation optimization operation method and system for distributed resources of a virtual power plant. The method comprises the following steps: constructing a dynamic operation model of distributed load resources in the virtual power plant; using an improved generative adversarial network to generate virtual power plant load operation time sequence data consistent with real data distribution; based on dynamic operation models of various distributed load resources and an electric power real-time market, comprehensively considering economical efficiency, equipment physical constraints and user comfort requirements, and constructing a virtual power plant dynamic optimization operation model based on a Markov decision process; and training the dynamic optimization operation model by adopting an MDL-SAC reinforcement learning algorithm to obtain a real-time aggregation optimization operation strategy. According to the invention, distributed load resource aggregation regulation and control considering market economy, user comfort, equipment safety and operation robustness is realized.

Description

Technical Field

[0001] This invention belongs to the field of virtual power plant optimization operation technology, and relates to a method and system for the aggregated optimization operation of distributed resources in a virtual power plant. Background Technology

[0002] With the continuous advancement of new power system construction, distributed flexibility resources are attracting increasing attention due to their strong regulation capabilities and good observability. These resources include energy storage systems, HVAC loads, and electric vehicles, which can flexibly adjust their power according to the power system's operational needs. They can participate in peak shaving and valley filling to smooth load curves, and can also generate revenue through price responses in market mechanisms. However, the spatial dispersion, diverse operating characteristics, and uncertainties of external environmental factors of distributed flexibility resources pose significant challenges to their optimized operation and coordinated dispatch.

[0003] In existing research and engineering practices both domestically and internationally, most methods for optimizing the operation of distributed flexible resources are based on optimization models or rule-based control strategies. Existing optimization model-based methods for regulating distributed flexible resources primarily rely on model-based control paradigms. Due to the high randomness and difficulty in understanding the operational characteristics of distributed flexible resources, constructing an accurate physical model that comprehensively reflects their spatiotemporal coupling relationships is extremely difficult. Therefore, existing methods often require the introduction of numerous approximate assumptions or reliance on predicted data, leading to deviations between the model and the actual operating state. Furthermore, as the scale of distributed flexible resources expands, the number of decision variables and constraints in the model increases dramatically, causing the optimization problem to grow exponentially, resulting in excessively long solution times. The decline in computational efficiency directly leads to lag in scheduling strategies, making it difficult to respond promptly to real-time market price fluctuations and load changes, ultimately limiting operational economy, robustness, and control effectiveness. Therefore, these methods are poorly adaptable to uncertainties such as changes in user behavior, weather fluctuations, and real-time electricity prices, and their operational strategies lack robustness. While rule-based control methods are computationally inexpensive and simple to implement, they struggle to cope with complex and ever-changing market and operating environments. Furthermore, strategy updates rely on manual adjustments based on experience, resulting in low levels of intelligence. For scenarios involving multi-source, heterogeneous, and dynamically coupled distributed flexible resources, existing methods generally suffer from high computational complexity, insufficient real-time performance, weak adaptability to uncertainty, and difficulty in balancing economic efficiency and security. These limitations hinder the efficient aggregation and flexible control of distributed flexible resources in market environments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for the aggregated and optimized operation of distributed resources in virtual power plants. This method enables the efficient aggregation of virtual power plant load resources and participation in the electricity market, forming a distributed load resource aggregation and control system that balances market economy, user comfort, equipment safety, and operational robustness.

[0005] The present invention adopts the following technical solution.

[0006] The first aspect of this invention proposes a method for the aggregated and optimized operation of distributed resources in a virtual power plant, comprising: Step 1: Establish a framework for the aggregation and optimization operation system of distributed load resources in the market scenario, and on this basis, construct a dynamic operation model for various distributed load resources in the virtual power plant; Step 2: Use an improved generative adversarial network to generate virtual power plant load operation time series data that is consistent with the distribution of real data; Step 3: Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, construct a virtual power plant dynamic optimization operation model based on Markov decision process; Step 4: Using the virtual power plant load operation time series data as training data, the dynamic optimization operation model is trained using the MDL-SAC reinforcement learning algorithm. The real-time aggregated optimization operation strategy for distributed load resources is obtained using the trained dynamic optimization operation model.

[0007] Preferably, the distributed load resources include electrical energy storage devices, HVAC systems, and electric vehicles.

[0008] Preferably, step 2 specifically includes: Step 2.1: Collect and preprocess historical operating data of the virtual power plant, and normalize the preprocessed time series data according to the following formula; Step 2.2: Based on the Gram angle field method, each sampling point in the normalized time series data is transformed from the Cartesian coordinate system to the polar coordinate system. According to the polar angle in the polar coordinate system, the value of each element in the Gram matrix is ​​calculated to form a two-dimensional Gram matrix GAF transformed from the one-dimensional time series data of the virtual power plant. The corresponding two-dimensional image data is obtained by scaling the values ​​of each element in the Gram matrix GAF. Step 2.3: Input the two-dimensional image data into the improved generative adversarial network and train the network using the WGAN-GP loss function; wherein, the improved generative adversarial network includes: a generator network, which introduces conditional information in the convolutional and transposed convolutional layers and uses batch normalization to stabilize the training process; and a discriminator network, which uses instance normalization in the convolutional layers to improve the ability to recognize sample detail features. Step 2.4: Use the trained generator network to generate virtual power plant load operation time series data that are consistent with the distribution of real data.

[0009] Preferably, in step 2.3, the WGAN-GP loss function is: (20) In the formula, The loss function of the discriminator network; The generator network loss function; The generator network G is represented by random noise as input. The generated samples output later; , These represent the discriminator network. For input real samples The scoring, discriminator network The output of the interpolated sample is relative to the input. The gradient; This indicates that for all generated distributions Mid-sample generation Calculate the corresponding discriminator network score. Expected value; This indicates that for all data from the true distribution... Real samples from the middle Calculate the corresponding discriminator network score. Expected value; This is a penalty applied to the gradient norm of the discriminator; Representing gradient -norm; This is the penalty coefficient; For real samples x With generated samples G ( z Samples are obtained by randomly interpolating along the connection lines. express The distribution it follows; K represents the Lipschitz constant constraint.

[0010] Preferably, step 3 specifically includes: Step 3.1: Construct the state space of the Markov decision process At time t, the state is for: (twenty one) In the formula, for tThe charging status of the energy storage device at all times; for t -1 is the market-clearing electricity price; for t The indoor temperature at any given time; for t The outdoor temperature at any given time; The target indoor temperature set by the user at time t; for t Real-time charging status of electric vehicles; The target amount of electricity needed for the next trip of the electric vehicle; T dep The time interval until the next trip of the electric vehicle; EN t for t The total power demand of the virtual power plant's rigid load at any given time; Step 3.2: Constructing the action space in the Markov decision process At time t, action include t The power of HVAC systems, electric vehicle charging power, and energy storage equipment at all times; Step 3.3: Based on the state space of Step 3.1 and the action space of Step 3.2, construct the reward function in the Markov decision process, including the reward functions for HVAC systems, electric energy storage systems and electric vehicles; Step 3.4: Construct the optimization objective function of the virtual power plant dynamic optimization operation model based on the reward function.

[0011] Preferably, in step 4, the neural network model of the reinforcement learning algorithm includes two Critic networks, one main Actor network, and two target Q-value networks.

[0012] Preferably, step 4 specifically includes: Step 4.1: Using the virtual power plant load operation time series data as training data, in the dynamic optimization operation model environment, based on the current strategy... The main Actor network selects the current action. Interact with the environment to earn rewards and the next state and will Store in the experience replay buffer; Step 4.2: During training, once the number of samples in the experience replay pool meets the minimum batch requirement, the parameter update phase begins. Each time parameters are updated, a batch of data is first sampled from the experience replay buffer to update the Critic network. Q θThe network parameters for the target Q-value are then softly updated. Secondly, based on batch data sampled from the empirical replay buffer, the Lagrange multipliers are updated using the MDL method, which adds momentum accumulation and damping terms to the Lagrange multiplier method. l : Step 4.3: Based on step 4.2, the SAC algorithm based on the MDL method is used to update the policy parameters of the main Actor network to obtain a new policy. Then, return to step 4.1 to continue interacting with the environment using the updated policy to obtain new samples until the number of training iterations reaches a preset threshold.

[0013] Preferably, in step 4.2, the Lagrange multipliers l The update formula is: (40) in, For momentum accumulation; subscript Indicates the first This is the second update; This is the dual momentum term in discrete form; It is the momentum decay factor; These are the driving constraint coefficients in discrete form; For strategy Expected cost of time constraints; The damping coefficient is in discrete form; This indicates a projection operation on a non-negative region.

[0014] Preferably, in step 4.3, the update formula for the main Actor network policy parameters is: (41) In this equation, the first term on the right side is the policy gradient update based on the SAC algorithm, and the second term is the constraint cost gradient update based on the MDL method. This indicates that under constraints, parameters are used. Defined strategy The optimization objective function; Indicates the policy parameters Gradient calculation; Indicating in strategy Under these conditions, take the expectation of the state-action trajectory distribution; Discount factor; This refers to the temperature coefficient in the SAC algorithm. Indicates the policy in the state Next generation action The logarithmic probability; Let be the Lagrange multiplier of the i-th constraint; For the first The cost function of each constraint represents the cost of the current policy π. The expected value of the lower constraint cost; The output of the Critic network is used to represent the state. Next action The resulting cumulative rewards; The policy output by the Actor network represents the state. The probability distribution of each action to be chosen; A second aspect of this invention proposes an aggregated optimization operation system for distributed resources in a virtual power plant, comprising: The model building module is used to establish a framework for the aggregated optimization operation system of distributed load resources in market scenarios and to build dynamic operation models of various distributed load resources in the virtual power plant based on this framework. Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, a dynamic optimization operation model of the virtual power plant based on Markov decision process is constructed. The data generation module is used to generate virtual power plant load operation time series data that is consistent with the real data distribution using an improved generative adversarial network; The model training module is used to train the dynamic optimization operation model using the virtual power plant load operation time series data as training data and the MDL-SAC reinforcement learning algorithm. The trained dynamic optimization operation model is then used to obtain the real-time aggregated optimization operation strategy for distributed load resources.

[0015] A third aspect of the present invention provides a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to perform operations according to the instructions to execute the steps of the method.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention proposes a virtual power plant operation data generation method based on an improved generative adversarial network (GAN). This method introduces a data dimensionality adjustment mechanism based on Gram's corner field, mapping one-dimensional time series data into two-dimensional image features for use as input to the generative network. This fully leverages the advantages of GANs in high-dimensional space modeling, improving the temporal consistency and spatial feature representation ability of the generated samples. Furthermore, this invention employs the WGAN-GP loss function in the training of the GAN, replacing weight clipping with gradient penalty, which significantly improves training stability and generation quality.

[0018] Based on the energy management environment characteristics of virtual power plants, this invention establishes a reward function that comprehensively considers multiple factors such as economy, equipment constraints, and indoor comfort. This function can guide intelligent agents to make virtual power plant scheduling and market participation decisions that maximize economic benefits while ensuring the safe operation of the system.

[0019] This invention employs the MDL method, adding momentum accumulation (inertia) and damping terms to the Lagrange multiplier method, thereby reducing oscillations in the cost function during optimization. This invention applies the MDL method to the policy update of the SAC algorithm, achieving adaptive optimization of constraints under continuous control conditions and improving the problem of frequent constraint violations by the agent during training.

[0020] Compared to distributed resource aggregation optimization methods based on optimization models, this invention employs an improved reinforcement learning algorithm (MDL-SAC reinforcement learning algorithm), which constructs a system that relies solely on historical electricity prices. With real-time observable operating parameters (states) t middle The virtual power plant can make intelligent and efficient operation decisions based on the current state (other than other quantities) when participating in the real-time electricity market, without having to repeat the optimization and solution process.

[0021] In the face of uncertainties such as user behavior, weather changes, and real-time electricity price fluctuations, this invention can continuously interact with the environment through historical data to adaptively learn the optimal strategy, rationally optimize the operation of internal flexible resources, and balance economy, robustness, and operational safety under uncertain conditions, thereby bringing higher economic benefits to virtual power plants. Attached Figure Description

[0022] Figure 1 This is a flowchart of the virtual power plant distributed flexibility resource aggregation and optimization operation method in this invention; Figure 2 A diagram showing the generator network structure and tensor size variations for generating adversarial networks; Figure 3 The discriminator network structure diagram for generating adversarial networks. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, other embodiments obtained by those skilled in the art without creative effort are all within the protection scope of this invention.

[0024] Embodiment 1 of this invention provides a method for the aggregated and optimized operation of distributed load resources in a virtual power plant. First, a framework for the aggregated and optimized operation of distributed load resources in a market scenario is designed. Typical distributed load resources such as energy storage systems, HVAC systems, and electric vehicles are selected to construct a refined dynamic operation model, and a reporting model for the aggregated distributed load resources of the virtual power plant participating in the real-time electricity market is built. Second, high-quality virtual power plant load operation data is generated based on an improved generative adversarial network and Gram's corner field method. Then, with the goal of maximizing economic efficiency and considering the physical constraints of load equipment and user comfort requirements, a dynamic optimization operation model for virtual power plant load aggregation is constructed based on a Markov decision process. Finally, the model is trained using the MDL-SAC reinforcement learning algorithm. The trained model supports decision-making even when the clearing price is unknown, thereby forming an aggregated and optimized operation strategy for distributed load resources in the virtual power plant. Figure 1 As shown, the method includes the following steps: Step 1: Establish a framework for the aggregation and optimization operation system of distributed load resources in the market scenario, and on this basis, construct a dynamic operation model for various distributed load resources in the virtual power plant; More preferably, a framework for the aggregated and optimized operation of distributed load resources in a market scenario is designed, and a dynamic operation model for various distributed load resources (including energy storage systems, HVAC systems, and electric vehicles) within a virtual power plant is constructed; specifically, the following steps are included: Step 1.1: Construct a power market operation framework. The market scenario is a real-time power market, which adopts a segmented non-rolling clearing mechanism with 15-minute intervals. Each cycle is an independent scheduling and settlement unit, that is, each 15-minute segmented non-rolling clearing cycle is an independent scheduling and settlement cycle. Virtual power plants participate in the market as aggregated resources on the load side. They are set as price takers, submitting only the amount of electricity they consume as the declared quantity, without submitting price information, and thus not affecting the marginal electricity price in the market. Before each scheduling cycle, the virtual power plant submits an adjustable load report. The transaction is settled based on the marginal market electricity price. The system has sufficient capacity, and the actual transaction volume is equal to the reported volume. Since the market clearing results have not yet been announced when the virtual power plant submits its load report, in order to adapt to the subsequent model, the electricity market time series is simplified to: the virtual power plant submits its load report at time t, and the market announces the clearing price at time t+1. During the optimized operation of a virtual power plant, the following energy balance equation must be satisfied: (1) In the formula: the left side of the equal sign represents the sum of the supplied power, and the right side represents the sum of the consumed power; PB t for t The power output declared to the market by the virtual power plant during that operating period, i.e., the power output of the day's virtual power plant. t The operating power during a 15-minute time period is positive, indicating that electricity is being purchased from the electricity market; EN t for t The total power demand of the virtual power plant's rigid load at any given time; P t EESC and P t EESD They are respectively t The charging and discharging power of the energy storage device is not simultaneously non-zero. P t HVAC for t The power of the HVAC equipment is displayed at any time; a positive value indicates cooling, and a negative value indicates heating. P t EV for t The charging power of electric vehicles at all times.

[0025] Step 1.2: Select typical distributed resources to construct a dynamic electrical model: Select three typical distributed flexible load resources, namely the energy storage system, heating and air conditioning system and the electric vehicle cluster system in the virtual power plant, and construct a refined model.

[0026] (1) Constructing a dynamic operation model for HVAC systems: (2) (3) (4) Equation (2) is the indoor temperature state transition model of the air conditioner; Equation (3) indicates that the power consumption should not exceed its maximum power; Equation (4) indicates that the indoor temperature should not exceed the user's comfort range.

[0027] In the formula: C andR These are the heat capacity and thermal resistance of the building where the air conditioner is located; i Temperature, superscript and They represent outdoor and indoor spaces, respectively. , for t +1 moment, t The indoor temperature at any given time; for t The outdoor temperature at any given time; or HVAC To improve the efficiency of HVAC systems; P t HVAC for t The power of the HVAC system at all times; This refers to the maximum power capacity of HVAC systems. , These are the upper and lower limits of indoor temperature; n The building's ventilation rate; t For time intervals.

[0028] (2) For energy storage systems, a piecewise linear model is introduced, and the dynamic model of electric energy storage is constructed as follows: (5) (6) (7) (8) (9) (10) (11) In the formula: SOC EES The state of charge of electrical energy storage, i.e., its percentage of charge; , for t +1 moment, t The charging status of the energy storage device at all times; This represents the maximum capacity of the electrical energy storage. and They are respectively t The charging and discharging power of the energy storage device at any given time is not simultaneously non-zero; and These are the charge and discharge efficiencies, respectively. t For time intervals; in addition, e Discount factor; R and F They are e - SOC The charging state and discount factor value at the nth segment point of the curve; , The upper and lower limits of the charging state of the energy storage device; N represents the energy storage device. e - SOC The Nth segment point of the curve; It is the first Energy storage period Available charging capacity; It is the first The safety reserve threshold for individual energy storage; This is the maximum power limit for electrical energy storage devices.

[0029] (3) Constructing a dynamic operation model for electric vehicles: (12) (13) (14) (15) In the formula: SOC EV The charging status of an electric vehicle, i.e., its percentage of charge; , for t +1 moment, t Real-time charging status of electric vehicles; P t EV for t Real-time electric vehicle charging power; or EV Improve electric vehicle charging efficiency; This represents the maximum capacity of an electric vehicle battery. t For time intervals; The amount of electricity consumed for electric vehicle travel; t arr , t dep These are the arrival time and departure time of the electric vehicle, respectively. The upper limit of charging power for electric vehicles; , The upper and lower limits of the charging status of electric vehicles.

[0030] binary variables A EV Indicates whether the electric vehicle is in a rechargeable state at the arrival time. t arr and departure time t dep Within a certain range, a value of 1 indicates that the electric vehicle can be charged, while a value of 0 indicates that the electric vehicle cannot be charged.

[0031] Step 2: Use an improved generative adversarial network to generate virtual power plant load operation time series data that is consistent with the distribution of real data; More preferably, an improved generative adversarial network-based method for generating virtual power plant load operation data is constructed, and the Gram corner field method is used to upscale the one-dimensional load time series into two-dimensional image data, thereby generating high-quality virtual power plant data samples. The specific steps include: Step 2.1: Collect and preprocess historical operating data of the virtual power plant, converting the time series data... Normalize to [the formula] The interval [1, 1] is shown below: (16) Step 2.2: Based on the Gramian Angle Field (GAF) method, each sampling point in the time series is transformed from the Cartesian coordinate system to the polar coordinate system according to Equation (17). The value of each element in the Gram matrix is ​​calculated according to Equation (18). Finally, a two-dimensional Gram matrix GAF is obtained by transforming the one-dimensional time series data of the virtual power plant. The values ​​of each element in the Gram matrix GAF are scaled to the range of [0, 255] to obtain the corresponding grayscale image.

[0032] (17) (18) (19) In the formula, i i for x i The transformed polar angle represents the polar angle of the sampling points in the normalized time series after the Cartesian coordinate system is transformed to the polar coordinate system. The specific value is the inverse cosine of the scaled data, with a range of [0, π], which preserves the numerical value information in the original data. r i for x i The converted radius representsx i The radius transformed from Cartesian to polar coordinates, specifically the time index of that sampling point. i Total length of virtual power plant data time series The ratio preserves the time index information in the original data and ensures that the position of each observation in polar coordinates is unique.

[0033] Step 2.3: Input the two-dimensional image data into the improved generative adversarial network; The generator network incorporates conditional information into its convolutional and transposed convolutional layers and employs batch normalization to stabilize the training process. The generator network structure and tensor size variations are as follows: Figure 2 As shown; The discriminator network employs instance normalization in the convolutional layers to improve its ability to recognize detailed features of samples. The discriminator network structure is as follows: Figure 3 As shown; The loss function of the generative adversarial network (GAN) is optimized using the Wasserstein distance framework with gradient penalty to improve the stability and diversity of generated samples.

[0034] Specifically, the loss function of generative adversarial networks in the WGAN framework is divided into the discriminator network loss function L. D And generator network loss function L G In WGAN, the loss functions for the discriminator network and the generator network are shown in the following equations:

[0035] This invention utilizes the Wasserstein distance generative adversarial network framework WGAN-GP with gradient penalty to optimize the loss function of the generative adversarial network in the WGAN framework, thereby... K - The Lipschitz continuity condition is directly added as a loss term to the loss function of the discriminator network. D This is called "gradient penalty" and is used to train the discriminator.

[0036] The discriminator network loss function of the WGAN-GP framework is as follows: (20) In the formula, The loss function of the discriminator network; Indicates: for all generated distributions Mid-sample generation Request a discriminator score Expected value; Indicates: for all data from the true distribution Real samples from the middle Request a discriminator score Expected value; The newly added term in the optimization is a penalty on the discriminator's gradient norm, ensuring that the discriminator satisfies the Lipschitz continuity constraint. That is, at any point between real and generated samples, the discriminator's gradient norm should be close to K; if it deviates, it is penalized.

[0037] G represents the generator network. Indicates input random noise The generated samples output later; D represents the discriminator network. , These represent the discriminators. For input samples The rating, discriminator The output of the interpolated sample is relative to the input. The gradient; Representing gradient -norm; This is the penalty coefficient, which controls the weight of the gradient penalty term; From real samples x With generated samples G ( z Random interpolation is obtained on the connection line, further... The distribution it follows is denoted as K represents the Lipschitz constant constraint, which is usually set to 1.

[0038] Step 2.4: After the network training is completed, the data generation stage only needs to use the generator and does not need to use the discriminator. Therefore, the trained generator network generates virtual power plant load operation time series data that is consistent with the distribution of real data. This data is used in step 4 as training data for the reinforcement learning model to train the virtual power plant dynamic optimization operation model.

[0039] Step 3: Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, construct a virtual power plant dynamic optimization operation model based on Markov decision process; More preferably, based on the real-time electricity market, and comprehensively considering economic efficiency, equipment physical constraints, and user comfort requirements, a dynamic optimization operation model for a virtual power plant based on a Markov decision process (MDP) is constructed. Specifically, with the goal of maximizing economic efficiency, considering distribution network topology constraints, a day-ahead market optimization control model is constructed by introducing conditional value-at-risk theory, including: Step 3.1: Construct the state space of the Markov decision process, wherein the state space include: The previous state space consists of the market-cleared electricity price, rigid load demand, regional and external temperatures, and the state of each controllable device. All physical quantities in each dimension of the state space are continuous quantities.

[0040] state space It is the set of observable variables of the agent. At time t, the system state can be represented as: (twenty one) In the formula, The electricity price is cleared in the market at the previous moment because the agent is making the decision to act. hour, Since the state is unknown, the state space is constructed using past states. i set The target indoor temperature set by the user; The target amount of electricity needed for the next trip of the electric vehicle; T dep The time interval until the next trip of the electric vehicle; for t The charging status of the energy storage device at all times; for t The indoor temperature at any given time; for t The outdoor temperature at any given time; for t Real-time charging status of electric vehicles; EN t for t The total power demand of the rigid load of the virtual power plant at any given time.

[0041] Subsequent status This will be fed as input into the deep reinforcement learning neural network in step 4, with the following specific uses: State Input Actor Network Output the probability distribution of actions; then sample actions based on this distribution. It is used to generate real-time control decisions for virtual power plants, where The definition is consistent with step 3.2; subsequently, based on the state-action pair Input Critic network It is used to evaluate the value of the current decision.

[0042] Step 3.2: Constructing the action space in the Markov decision process In the action space, all physical quantities in each dimension are continuous quantities.

[0043] Action space This represents the set of control variables that the agent can take at each time step. At time t, the specific action of the system can be represented as: (twenty two) In the formula, the charging and discharging power of electrical energy storage is... P t ESSC and P t ESSD Depend on P t ESS The value is uniform; a positive value indicates charging, and a negative value indicates discharging. P t HVAC for t The power of the HVAC system at all times; P t EV for t Real-time electric vehicle charging power; P t ESS This refers to the power of the energy storage device.

[0044] Step 3.3: Constructing the reward function in the Markov decision process. The reward function includes a comprehensive evaluation of the operating performance of various adjustable load devices in the virtual power plant, covering reward functions for HVAC systems, energy storage systems, and electric vehicles.

[0045] (1) The reward function of the HVAC system is: (twenty three) (twenty four) (25) in, Indicates a comfort reward; This indicates an economic benefit reward based on market-based electricity purchase expenditures; i set The target indoor temperature set by the user; for t The indoor temperature at any given time; Assuming the user's acceptable temperature difference is ± i diff Then in [ i set - i diff , i set + i diff Within the specified range, a smaller reward should be given to ensure that the temperature obtained by the intelligent agent in regulating the HVAC equipment meets the user's needs; For the present Market-cleared electricity prices at any time; and These are the weighting coefficients; P t HVAC This refers to the power of the HVAC system.

[0046] (2) The reward function of the energy storage system is: (26) (27) (28) (29) in, The reward represents the deviation of the energy storage state of charge from the target value, in order to avoid overcharging and over-discharging behavior of the agent; This indicates that the charging and discharging power is based on the economic benefit of market electricity purchase costs. This incentive, representing the cost of battery aging, aims to constrain the frequency and magnitude of energy storage device operations within a certain range. , , These are the weighting coefficients; e It is a step function; for t The charging status of the energy storage device at all times; , The upper and lower limits of the charging status of energy storage devices; P t ESS The power of the energy storage device; t For time intervals; for t Electricity prices will be cleared out by the market at any time.

[0047] (3) The reward function for electric vehicles is: (30) (31) (32) (33) (34) (35) in, Indicates a comfort level bonus during charging; Indicates the economic benefits of charging and discharging power; This represents a reward for the cost of battery aging. This refers to the power correction reward after the trip, which is used to reduce invalid dispatch instructions after the electric vehicle leaves the charging station; , , , These are the weighting coefficients; for t Real-time charging status of electric vehicles; t arr , t dep These are the arrival time and departure time of the electric vehicle, respectively. for t dep , t arr Time difference; P t EV for t Real-time electric vehicle charging power; The charging state that the user expects to achieve; The initial charging state of the electric vehicle; This refers to the energy consumed by the electric vehicle during the current time period. This represents the maximum capacity of an electric vehicle battery.

[0048] Step 3.4: Combining the reward functions of the various types of equipment mentioned above, the objective function of the virtual power plant dynamic optimization operation model based on Markov decision process is: (36) in, As a discount factor, , , Let be the total rewards at time t for the HVAC system, the energy storage system, and the electric vehicle, respectively. This model can be used as the optimization objective function for reinforcement learning.

[0049] Step 4: Using the virtual power plant load operation time series data as training data, the dynamic optimization operation model is trained using the MDL-SAC reinforcement learning algorithm. The real-time aggregated optimization operation strategy for distributed load resources is obtained using the trained dynamic optimization operation model.

[0050] More preferably, an advanced reinforcement learning algorithm is introduced and further improved. An MDL-SAC reinforcement learning algorithm is used to train the dynamic optimization operation model. The trained model supports decision-making even when the clearing price is unknown, enabling a real-time aggregated optimization operation strategy for distributed load resources. Specifically, during the real-time electricity market participation phase, corrections and feedback are performed, with a rolling optimization calculation for the entire day executed at regular time steps, updating the control plan using the latest data.

[0051] The algorithm is based on the Actor-Critic architecture, and its neural network model consists of two Critic networks, one main Actor network, and two target Q-value networks. Specifically, the following steps are included: Step 4.1: In a dynamic optimization operation model environment consisting of a HVAC system, an energy storage system, and an electric vehicle, based on the current strategy... The current action is selected by the main Actor network. Interact with the environment to earn rewards and the next state and will Store in the experience replay buffer; Step 4.2: During the training process, when the number of samples in the experience replay pool meets the minimum batch requirement, the algorithm enters the parameter update stage. Each time the parameters are updated, the batch data is sampled from the experience replay buffer, the Critic network is updated according to the loss function shown in Equation (37), and the target Q-value function network is softly updated according to Equation (38).

[0052] (37) (38) Equation (37) is the Critic network Q θ The loss function.

[0053] In the formula, is the loss function of the Critic network, which represents the difference between the Q-value predicted by the Critic network and the target Q-value calculated on average over the samples in the experience pool; This represents the expectation calculated on samples of state-action pairs in the experience replay buffer; For Q-value estimation of state-action pairs in the Critic network; The target Q value; The target Q-value network parameters; These are the parameters for the Critic network. D This represents the distribution of states and actions stored in the experience replay pool; Possession and Q Same definition, difference lies in By parameters The network is calculated and is called the target Q-value network. Equation (38) is its soft update formula. For soft update coefficients: Secondly, the Lagrange multipliers are updated based on the sampled data (batch data sampled from the experience replay buffer). l .

[0054] The control strategies obtained using reinforcement learning algorithms often fail to meet the constraints of each device during decision-making. In the traditional MDP model, the damage caused by a general action is defined as a negative reward.

[0055] Constrained Markov Decision Processes (CMDPs) define the mapping relationship between states and actions and damages as a cost function and explicitly write it as constraints.

[0056] CMDP can be solved using the Lagrange multiplier method, which involves directly incorporating constraints as multiplier terms into the objective function, thereby transforming a constrained optimization problem into an unconstrained optimization problem.

[0057] This invention proposes the MDL (Momentum-Damped Lagrange) method, which draws on the idea of ​​damped second-order dual dynamics and introduces momentum accumulation (inertia) based on the Lagrange multiplier method for solving constrained Markov decision processes. ) and damping term (i.e., damping correction term) This reduces oscillations in the cost function during the optimization process. By introducing momentum accumulation (inertia) and damping terms, the Lagrange multipliers... l The update formula is: (39) The superscript · indicates differentiation; It is the damping coefficient in continuous form, used to suppress multiplier oscillations and achieve "inertial braking"; These are the constraint driving coefficients in continuous form, used to adjust the constraint cost term. The driving force intensity of the multiplier dynamic system.

[0058] The constraint cost function is defined as follows: The expected value of the degree of constraint violation in the lower system, i.e., the strategy is... The expected constraint cost at that time, in actual calculations, needs to be obtained by calculating the average constraint cost of the sampled batch: That is: for the instantaneous constraint cost function The expectations, among which Composed of equations (42) and (43); in actual calculations, it is obtained by sampling batch data from the empirical playback buffer. The constraint cost is calculated and averaged to obtain the strategy. From parameters As defined, and the symbol is the same as in equation (41) They represent the same function.

[0059] It can be considered as a momentum term; It can be regarded as an inertial term. The discrete update form of equation (39) is: (40) Among them, subscript Indicates the first This is the second update; This is the dual momentum term in discrete form, used to measure the cumulative constraint error; This is the momentum decay factor, used to control the proportion of the momentum term retained in the current update from the previous moment. The typical value range is 0.8 to 0.95.

[0060] These are the driving constraint coefficients in discrete form, used for adjustment. The intensity of its influence on momentum renewal typically ranges from 0.3 to 0.8. Its physical meaning is related to... Both are consistent and satisfy the following during the discretization derivation process. When Δt→0, the two tend to be strictly equal.

[0061] The damping coefficient in discrete form has the same physical meaning as... Consistency is used to suppress multiplier oscillations and achieve "inertial braking," with a typical value range of [value range missing]. Both satisfy the following during the discretization derivation process: ,when → When the value is 0, the two tend to be strictly equal.

[0062] This represents a projection operation on a non-negative region, used to ensure... ; The first equation applies to the current constrained residual. Weighted accumulation resulted in the momentum term. This recursion is equivalent to a weighted integral of the constraint error: when the constraint is violated for a long time, momentum gradually accumulates, driving the multiplier to increase; when the constraint is satisfied for a long time, momentum gradually decays. This formula gives the multiplier update an inertial characteristic, avoiding oscillations caused by immediate multiplier updates when the constraint is violated.

[0063] The second equation introduces a damping correction term in the multiplier update. This is equivalent to applying friction to the rate of change of the multiplier, thereby suppressing overshoot and oscillation.

[0064] Traditional Lagrange methods are prone to sawtooth oscillations or overshoot divergence due to sampling noise or gradient fluctuations, while MDL enables multiplier updates to have inertial propulsion and self-stabilizing vibration suppression characteristics, which significantly improves the convergence smoothness and stability of the optimization process while satisfying constraints.

[0065] Understandably, each update involves first updating the Critic network parameters, and then updating... Finally, update the Actor network parameters. This includes updating... At the current moment And the previous moment There is an explicit correspondence; however, the updates of the Critic and Actor network parameters adopt an empirical replay mechanism. As the time step increases, new samples are added to the empirical pool, requiring resampling and recalculation. There is no temporal recursive relationship, therefore only updates are performed. When involved Other parameters were not involved in the update. .

[0066] Step 4.3: Based on step 4.2, the SAC algorithm based on MDL is used to update the main Actor network policy parameters according to the following formula to obtain a new policy. Then, return to step 4.1 and use the updated policy to continue interacting with the environment to obtain new samples until the number of training iterations reaches the preset threshold. Specifically, using the Critic network updated in step 4.2 With Acotor Network Calculate the policy gradient (i.e., the first term of Equation 41) and the Lagrange multipliers obtained in step 4.2. lBased on this, update the main Actor network policy parameters according to the following formula to obtain a new policy, then return to step 4.1, use the updated policy to continue interacting with the environment, and obtain new samples: (41) The first term is the policy gradient update based on SAC, and the second term is the constraint cost gradient update based on the MDL method. This indicates that under constraints, parameters are used. Defined strategy The optimization objective function; Indicates the policy parameters The gradient operation is used to update the Actor network (policy network); Indicating in strategy Under these conditions, take the expectation of the state-action trajectory distribution; Discount factor; This refers to the temperature coefficient in the SAC algorithm. Indicates the policy in the state Next generation action The logarithmic probability; The Lagrange multiplier for the i-th constraint is obtained by updating in step 4.2; For the first The cost function of each constraint represents the cost of the current policy π. The expected value of the lower constraint cost; it is understandable that here... The subscript is Equation (41) represents the parameters of the policy network. To update the parameters, step 4.2 involves updating the Lagrange multipliers. l In the formula, middle It can be understood as , indicating that the strategy at the k-th update is Equation (41) Emphasis on π Because the formula needs to be correct Calculate the gradient, while step 4.2 does not require specifying the parameters. .

[0067] The output of the Critic network is used to represent the state. Next action The resulting cumulative rewards; The policy output by the Actor network represents the state. The probability distribution of each action can be selected.

[0068] The constraint cost function includes the energy storage system's state-of-charge deviation cost. and the cost of electric vehicle availability To impose penalties on states that exceed safety constraints: (42) (43) In the formula, for t The charging status of the energy storage device at all times; , The upper and lower limits of the charging status of energy storage devices; P t EV for t Real-time electric vehicle charging power; t arr , t dep These are the arrival time and departure time of the electric vehicle, respectively.

[0069] By training the dynamic optimization operation model using the MDL-SAC reinforcement learning algorithm, a distributed flexibility resource aggregation optimization operation method for virtual power plants can be obtained. Using the trained model, intelligent and efficient virtual power plant operation decisions can be quickly and directly implemented based on the current state during real-time market participation.

[0070] This embodiment employs a distributed flexible resource aggregation optimization operation method for virtual power plants proposed in this invention. Based on sample data within a virtual power plant, the resources within the virtual power plant are regulated and participate in the real-time market.

[0071] Table 1 compares the MDL-SAC and SAC algorithms. The MDL-SAC algorithm ultimately achieves a higher convergence value, indicating that the BEM regulation strategy based on the MDL-SAC algorithm performs better under the current reward function that comprehensively considers multiple factors. Although the introduction of MDL increases the computational cost during training, the trained model no longer involves backpropagation and parameter updates in the neural network when making decisions; therefore, the MDL step does not affect the decision-making speed in application.

[0072] Table 1 Comparison of MDL-SAC and SAC algorithms

[0073] Embodiment 2 of the present invention provides an aggregated optimization operation system for distributed resources of a virtual power plant, comprising: The model building module is used to establish a framework for the aggregated optimization operation system of distributed load resources in market scenarios and to build dynamic operation models of various distributed load resources in the virtual power plant based on this framework. Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, a dynamic optimization operation model of the virtual power plant based on Markov decision process is constructed. The data generation module is used to generate virtual power plant load operation time series data that is consistent with the real data distribution using an improved generative adversarial network; The model training module is used to train the dynamic optimization operation model using the virtual power plant load operation time series data as training data and the MDL-SAC reinforcement learning algorithm. The trained dynamic optimization operation model is then used to obtain the real-time aggregated optimization operation strategy for distributed load resources.

[0074] Embodiment 3 of the present invention provides a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to perform operations according to the instructions to execute the steps of the method.

[0075] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0076] Compared with the prior art, the beneficial effects of the present invention include at least the following: Based on the energy management environment characteristics of virtual power plants, this invention establishes a reward function that comprehensively considers multiple factors such as economy, equipment constraints, and indoor comfort. This function can guide intelligent agents to make virtual power plant scheduling and market participation decisions that maximize economic benefits while ensuring the safe operation of the system.

[0077] This invention proposes a virtual power plant operation data generation method based on an improved generative adversarial network (GAN). This method introduces a data dimensionality adjustment mechanism based on Gram's corner field, mapping one-dimensional time series data into two-dimensional image features for use as input to the generative network. This fully leverages the advantages of GANs in high-dimensional space modeling, improving the temporal consistency and spatial feature representation ability of the generated samples. Furthermore, this invention employs the WGAN-GP loss function in the training of the GAN, replacing weight clipping with gradient penalty, which significantly improves training stability and generation quality.

[0078] This invention employs the MDL method, adding momentum accumulation (inertia) and damping terms to the Lagrange multiplier method, thereby reducing oscillations in the cost function during optimization. This invention applies the MDL method to the policy update of the SAC algorithm, achieving adaptive optimization of constraints under continuous control conditions and improving the problem of frequent constraint violations by the agent during training.

[0079] Compared to distributed resource aggregation optimization methods based on optimization models, this invention employs an improved reinforcement learning algorithm (MDL-SAC reinforcement learning algorithm), which constructs a system that relies solely on historical electricity prices. With real-time observable operating parameters (states) t middle The virtual power plant can make intelligent and efficient operation decisions based on the current state (other than other quantities) when participating in the real-time electricity market, without having to repeat the optimization and solution process.

[0080] In the face of uncertainties such as user behavior, weather changes, and real-time electricity price fluctuations, this invention can continuously interact with the environment through historical data to adaptively learn the optimal strategy, rationally optimize the operation of internal flexible resources, and balance economy, robustness, and operational safety under uncertain conditions, thereby bringing higher economic benefits to virtual power plants.

[0081] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0082] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0083] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0084] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for aggregated and optimized operation of distributed resources in a virtual power plant, characterized in that, include: Step 1: Establish a framework for the aggregation and optimization operation system of distributed load resources in the market scenario, and on this basis, construct a dynamic operation model for various distributed load resources in the virtual power plant; Step 2: Use an improved generative adversarial network to generate virtual power plant load operation time series data that is consistent with the distribution of real data; Step 3: Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, construct a virtual power plant dynamic optimization operation model based on Markov decision process; Step 4: Using the virtual power plant load operation time series data as training data, the dynamic optimization operation model is trained using the MDL-SAC reinforcement learning algorithm. The real-time aggregated optimization operation strategy of distributed load resources is obtained using the trained dynamic optimization operation model.

2. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 1, characterized in that: The distributed load resources include energy storage devices, HVAC systems, and electric vehicles.

3. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Collect and preprocess historical operating data of the virtual power plant, and normalize the preprocessed time series data according to the following formula; Step 2.2: Based on the Gram angle field method, each sampling point in the normalized time series data is transformed from the Cartesian coordinate system to the polar coordinate system. According to the polar angle in the polar coordinate system, the value of each element in the Gram matrix is ​​calculated to form a two-dimensional Gram matrix GAF transformed from the one-dimensional time series data of the virtual power plant. The corresponding two-dimensional image data is obtained by scaling the values ​​of each element in the Gram matrix GAF. Step 2.3: Input the two-dimensional image data into the improved generative adversarial network and train the network using the WGAN-GP loss function; wherein, the improved generative adversarial network includes: a generator network, which introduces conditional information in the convolutional and transposed convolutional layers and uses batch normalization to stabilize the training process; and a discriminator network, which uses instance normalization in the convolutional layers to improve the ability to recognize sample detail features. Step 2.4: Use the trained generator network to generate virtual power plant load operation time series data that are consistent with the distribution of real data.

4. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 3, characterized in that: In step 2.3, the WGAN-GP loss function is: （20） In the formula, The loss function of the discriminator network; The generator network loss function; The generator network G is represented by random noise as input. The generated samples output later; , These represent the discriminator network. For input real samples The scoring, discriminator network The output of the interpolated sample is relative to the input. The gradient; This indicates that for all generated distributions Mid-sample generation Calculate the corresponding discriminator network score. Expected value; This indicates that for all data from the true distribution... Real samples from the middle Calculate the corresponding discriminator network score. Expected value; This is a penalty applied to the gradient norm of the discriminator; Representing gradient -norm; This is the penalty coefficient; For real samples x With generated samples G ( z Samples are obtained by randomly interpolating along the connection lines. express The distribution it follows; K represents the Lipschitz constant constraint.

5. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 1, characterized in that: Step 3 specifically includes: Step 3.1: Construct the state space of the Markov decision process At time t, the state is for: （21） In the formula, for t The charging status of the energy storage device at all times; for t -1 is the market-clearing electricity price; for t The indoor temperature at any given time; for t The outdoor temperature at any given time; The target indoor temperature set by the user at time t; for t Real-time charging status of electric vehicles; The target amount of electricity needed for the next trip of the electric vehicle; T dep The time interval until the next trip of the electric vehicle; EL t for t The total power demand of the virtual power plant's rigid load at any given time; Step 3.2: Constructing the action space in the Markov decision process At time t, action include t The power of HVAC systems, electric vehicle charging power, and energy storage equipment at all times; Step 3.3: Based on the state space of Step 3.1 and the action space of Step 3.2, construct the reward function in the Markov decision process, including the reward functions for HVAC systems, electric energy storage systems and electric vehicles; Step 3.4: Construct the optimization objective function of the virtual power plant dynamic optimization operation model based on the reward function.

6. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 1, characterized in that: In step 4, the neural network model of the reinforcement learning algorithm includes two Critic networks, one main Actor network, and two target Q-value networks.

7. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 6, characterized in that: Step 4 specifically includes: Step 4.1: Using the virtual power plant load operation time series data as training data, in the dynamic optimization operation model environment, based on the current strategy... The main Actor network selects the current action. Interact with the environment to earn rewards and the next state and will Store in the experience replay buffer; Step 4.2: During training, once the number of samples in the experience replay pool meets the minimum batch requirement, the parameter update phase begins. Each time parameters are updated, a batch of data is first sampled from the experience replay buffer to update the Critic network. Q θ The network parameters for the target Q-value are then softly updated. Secondly, based on batch data sampled from the empirical replay buffer, the Lagrange multipliers are updated using the MDL method, which adds momentum accumulation and damping terms to the Lagrange multiplier method. λ : Step 4.3: Based on step 4.2, the SAC algorithm based on the MDL method is used to update the policy parameters of the main Actor network to obtain a new policy. Then, return to step 4.1 to continue interacting with the environment using the updated policy to obtain new samples until the number of training iterations reaches a preset threshold.

8. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 7, characterized in that: In step 4.2, the Lagrange multipliers λ The update formula is: （40） in, For momentum accumulation term; subscript Indicates the first This is the second update; This is the dual momentum term in discrete form; It is the momentum decay factor; These are the driving constraint coefficients in discrete form; For strategy Expected cost of time constraints; The damping coefficient is in discrete form; This indicates a projection operation on a non-negative region.

9. The method for aggregated and optimized operation of distributed resources in a virtual power plant according to claim 7, characterized in that: In step 4.3, the update formula for the main Actor network policy parameters is: （41） In this equation, the first term on the right side is the policy gradient update based on the SAC algorithm, and the second term is the constraint cost gradient update based on the MDL method. This indicates that under constraints, parameters are used. Defined strategy The optimization objective function; Indicates the policy parameters Gradient calculation; Indicating in strategy Under these conditions, take the expectation of the state-action trajectory distribution; Discount factor; This refers to the temperature coefficient in the SAC algorithm. Indicates the policy in the state Next generation action The logarithmic probability; Let be the Lagrange multiplier of the i-th constraint; For the first The cost function of each constraint represents the cost of the current policy π. The expected value of the lower constraint cost; The output of the Critic network is used to represent the state. Next action The resulting cumulative rewards; The policy output by the Actor network represents the state. The probability distribution of each action can be selected.

10. A virtual power plant distributed resource aggregation and optimization operation system, comprising the method described in any one of claims 1-9, characterized in that, The system includes: The model building module is used to establish a framework for the aggregated optimization operation system of distributed load resources in market scenarios and to build dynamic operation models of various distributed load resources in the virtual power plant based on this framework. Based on the dynamic operation models of various distributed load resources and the real-time electricity market, and taking into account economic efficiency, equipment physical constraints and user comfort requirements, a dynamic optimization operation model of the virtual power plant based on Markov decision process is constructed. The data generation module is used to generate virtual power plant load operation time series data that is consistent with the real data distribution using an improved generative adversarial network; The model training module is used to train the dynamic optimization operation model using the virtual power plant load operation time series data as training data and the MDL-SAC reinforcement learning algorithm. The trained dynamic optimization operation model is then used to obtain the real-time aggregated optimization operation strategy for distributed load resources.

11. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-9.

Images