Virtual power plant green energy consumption cooperation method and system fusing digital twinning and reinforcement learning

Abstract

The invention discloses a virtual power plant green energy consumption cooperation method and system fusing digital twinning and reinforcement learning, and the method comprises the steps: building a green energy power plant digital twinning model through a simulation tool, collecting data in real time, and carrying out the normalization and abnormal value cleaning; inputting the data into the digital twinborn model, mapping the operation state of a physical system, and rehearsing the influence of different energy scheduling strategies on the green energy consumption rate and the power grid frequency in a virtual environment; optimizing and updating the energy scheduling strategy based on a near-end strategy optimization PPO algorithm; the optimized and updated energy scheduling strategy is fed back to a physical system to be executed, the strategy execution effect is monitored in real time, the digital twin model parameters are updated, and closed-loop control is formed; the method can solve the problem that the intermittency of renewable energy sources is not matched with the dynamic demand of the load.

Description

Technical Field

[0001] The present invention relates to a collaborative method and system for green energy consumption in a virtual power plant, and specifically to a collaborative method and system for green energy consumption in a virtual power plant that integrates digital twin and reinforcement learning. Background Art

[0002] As the proportion of renewable energy in the power system continues to increase, its inherent intermittency and uncertainty pose severe challenges to the stable operation of the power grid and the efficient consumption of green energy. Especially with the widespread application of green energy such as wind energy and solar energy, these energies face a common problem, that is, they are unpredictable and unstable in time. For example, wind power generation depends on wind speed, while photovoltaic power generation depends on solar irradiance intensity and time. The uncertainty of these factors makes it difficult for renewable energies such as wind power and photovoltaic power to accurately match the load demand, resulting in a large amount of renewable energy being wasted or having to be abandoned.

[0003] Traditional virtual power plant (VPP) technology, by aggregating distributed energy, energy storage devices, and adjustable load resources, can alleviate the contradiction between supply and demand to a certain extent, but there are still some limitations. Traditional scheduling strategies mostly rely on static models and fixed rules, and it is difficult to adapt to the real-time fluctuations of green energy power and the dynamic changes of load demand, resulting in a low green energy consumption rate and frequent fluctuations in the power grid frequency; existing methods often use manual experience or single-objective optimization when balancing the green energy utilization rate, task execution timeliness (such as supercomputer computing tasks), and power grid stability, lacking an adaptive multi-objective dynamic weight mechanism; for high-energy-consuming adjustable load terminals represented by supercomputer clusters, their flexible power regulation capabilities have not been systematically integrated, resulting in low source-load interaction efficiency; traditional virtual power plants lack a "prediction - decision - feedback" closed-loop mechanism based on real-time data and are difficult to achieve dynamic iterative optimization of strategies. Summary of the Invention

[0004] Object of the Invention: The object of the present invention is to provide a collaborative method for green energy consumption in a virtual power plant that integrates digital twin and reinforcement learning to solve the problems of low consumption efficiency, difficult dynamic matching of supply and demand, and insufficient multi-objective collaborative optimization caused by the access of a high proportion of renewable energy to the power grid. On the other hand, a collaborative system for green energy consumption in a virtual power plant that integrates digital twin and reinforcement learning is provided.

[0005] Technical Solution: The collaborative method for green energy consumption in a virtual power plant described in the present invention includes the following steps:

[0006] S1. Construct a digital twin model of a green energy power plant through a simulation tool, collect real-time data on wind power, energy storage power, load demand, power grid frequency, and green energy fluctuation characteristics, and perform normalization and outlier cleaning;

[0007] S2. Input the data after normalization and outlier cleaning into the digital twin model to map the operating state of the physical system, and pre - simulate the impacts of different energy scheduling strategies on the green energy consumption rate and the grid frequency in the digital twin model;

[0008] S3. Optimize and update the energy scheduling strategy based on the Proximal Policy Optimization (PPO) algorithm. The optimization and update are achieved by constructing a reinforcement learning agent, obtaining training samples through the digital twin model, and dynamically optimizing the energy storage charge - discharge instructions, load regulation parameters, and grid interaction strategies;

[0009] S4. Feed back the optimized and updated energy scheduling strategy to the physical system for execution, monitor the execution effect of the strategy in real - time, and update the parameters of the digital twin model to form a closed - loop control.

[0010] Preferably, the digital twin environment formula in S3 is as follows:

[0011] Q(s,p)←Q(s,p)+p[r + qp′max Q(s′,p)-Q(s,p)];

[0012] Where p represents the learning rate, q represents the discount factor, p'maxQ(s',p) represents maximizing the future reward, Q(s,p) represents obtaining the reward, and r represents the initial value.

[0013] Preferably, the algorithm in S3 is as follows:

[0014] S31. Define the observed variable of energy scheduling for the system, establish a multi - dimensional composite observed variable system for the energy scheduling system, and define the system state parameters with time - varying characteristics;

[0015] S32. Design a multi - objective weighted reward function, balance the dynamic weight and constraint penalty, and clarify the action constraints;

[0016] S33. Design the Actor network architecture, collect data and perform pre - processing;

[0017] S34. Adopt an experience replay mechanism, automatically focus on important samples according to the current learning stage, and output the optimized instructions for energy storage charge - discharge, load regulation, and grid interaction.

[0018] Preferably, the formula for defining the system state parameters in S31 is as follows:

[0019] s=[Pwind(t),SOC(t),Pload(t),fgrid(t)];

[0020] Among them, s represents the input state, Pwind(t) represents the real-time wind power, SOC(t) represents the state of charge of the energy storage system, Pload(t) represents the load demand or energy demand, and fgrid(t) represents the grid frequency.

[0021] Preferably, the multi-objective weighted reward function in S32 is as follows:

[0022]

[0023] Among them, Rt represents the composite reward function, α, β, and γ represent dynamic adjustment coefficients, Egreen represents the usage amount of green energy, that is, the green energy consumption rate, Etotal represents the total energy consumption, Tcomplete represents the task completion time, that is, the task timeliness, Tdeadline represents the deadline, and ΔPgrid represents the fluctuation amount of the grid power, that is, the grid fluctuation suppression.

[0024] Preferably, the Actor network architecture in S33 is as follows:

[0025] a = μ(s) + σ(s)·ò;

[0026] Among them, μ(s) represents the action mean, σ(s) represents the action standard deviation, ò represents N(0,1) noise, and a is the network architecture.

[0027] Preferably, the optimization update formula in S3 is as follows:

[0028]

[0029] Rt = W1*F + W2*T - W3*|ΔP_grid|;

[0030] max rt + qV(s{t+1}) - plogπ(qt|st);

[0031] ||θ - θ{old}||2 ≤ δ;

[0032] Q1(s,p) ≈ Es′,p′[r + qp′min Q2(s′,p′)];

[0033] Q2(s,p) ≈ Es′,p′[r + qp′min Q1(s′,p′)];

[0034] Among them, L(θ) represents the policy gradient loss, represents the policy entropy coefficient, It represents the entropy regularization term, β represents the entropy regularization coefficient, H(st,πθ) represents the entropy regularization term, ò represents the truncation threshold, and min(rt+γV(s{t+1})) represents the truncation policy update step size; W represents the weight coefficient in the multi-objective reward function, W1 represents the weight of green energy utilization rate, W2 represents the weight of task timeliness, W3 represents the weight of power grid power fluctuation penalty, |ΔP_grid| represents suppressing the power grid power fluctuation, F represents the green energy utilization rate, and T represents the task timeliness; δ represents the new trust region radius, and Q1 and Q2 are used to measure the consistency of Q-value estimation.

[0035] The virtual power plant green energy consumption coordination system described in the present invention includes:

[0036] The digital twin module is used to build a digital twin model of the green energy power plant, map the operating state of the physical system, and preview the impact of different energy scheduling strategies on the green energy consumption rate and the power grid frequency.

[0037] The reinforcement learning module is used to build a reinforcement learning agent based on the proximal policy optimization (PPO) algorithm, adjust the working mode of the energy storage module and the power configuration of the supercomputer cluster module, and update the energy scheduling strategy.

[0038] The energy storage module includes aggregated distributed energy, energy storage devices, and adjustable load resources, and is used to ensure supplementary supply when the power grid needs more electric energy and store it when the power grid has surplus electric energy.

[0039] The supercomputer cluster module is used to provide data processing capabilities to meet the model calculation and data processing requirements.

[0040] Beneficial effects: Compared with the prior art, the present invention has the following remarkable advantages: 1. By building a high-precision virtual power plant digital twin model, it can map the operating state of the physical power plant and the fluctuation characteristics of green energy (such as wind power and photovoltaic) in real time, and dynamically optimize the energy scheduling strategy in combination with the reinforcement learning algorithm; 2. Taking the supercomputer cluster module center as the flexible load end, using its adjustable load characteristics, designing a "source-load" two-way interaction mechanism, and through virtual power plant aggregation of distributed energy, energy storage devices, and adjustable load resources based on green energy prediction and load demand response, realizing local consumption and global optimization of green energy; 3. The reinforcement learning algorithm continuously explores the optimal control strategy in the digital twin environment, adaptively adjusts the power configuration of the supercomputer cluster module, balances the green energy fluctuation and the energy efficiency of the computing task, and forms a closed-loop control of "prediction-decision-feedback". Description of the Drawings

[0041] Figure 1 It is a schematic structural framework diagram of the present invention;

[0042] Figure 2 It is a schematic flow diagram of the reinforcement learning algorithm of the present invention;

[0043] Figure 3 This is a schematic diagram of the working process of the present invention;

[0044] Figure 4 This is a schematic diagram of the overall process of the system of the present invention. Specific embodiments

[0045] Next, in conjunction with the accompanying drawings, the technical solution of the present invention will be described in detail.

[0046] As Figures 1-3 shown, the virtual power plant green energy consumption coordination method integrating digital twin and reinforcement learning has the following steps:

[0047] S1. Real-time collect data on wind power, energy storage power, load demand, grid frequency, and green energy fluctuation characteristics, and perform normalization and outlier cleaning. Use simulation tools (such as MATLAB / Simulink, OpenFMB) to build a high-precision digital twin model of the green energy power plant for the physical system, for data generation (training set), energy dispatch strategy simulation verification, real-time monitoring, and feedback;

[0048] Core formula of the digital twin model of the green energy power plant:

[0049] s = [PWind(t), SOC(t), Pload(t), fgrid(t)];

[0050] Among them, s represents the input state, Pwind(t) represents the real-time wind power, SOC(t) represents the state of charge of the energy storage system, Pload(t) represents the load demand or energy demand, and fgrid(t) represents the grid frequency.

[0051] The specific steps are as follows:

[0052] (1) Define the observation variables of the system: energy dispatch, establish a multi-dimensional composite observation variable system for the energy dispatch system. On this basis, a data-driven dynamic optimization framework needs to be constructed. By defining the system state parameters with time-varying characteristics, the formula is as follows:

[0053] s = [Pwind(t), SOC(t), Pload(t), fgrid(t)];

[0054] Among them, s represents the input state, Pwind(t) represents the real-time wind power, SOC(t) represents the state of charge of the energy storage system, Pload(t) represents the load demand or energy demand, and fgrid(t) represents the grid frequency.

[0055] (2) Design multi-objective weighted rewards, and it is necessary to balance dynamic weights and constraint penalties. The formula is as follows:

[0056]

[0057] Among them, \(R_t\) represents the composite reward function, \(\alpha\), \(\beta\), and \(\gamma\) represent dynamic adjustment coefficients (\(\alpha\), \(\beta\), \(\gamma\) generally take values of 0.7, 0.2, 0.1), \(E_{green}\) represents the usage of green energy, that is, the green energy consumption rate, \(E_{total}\) represents the total energy consumption, \(T_{complete}\) represents the task completion time, that is, the task timeliness, \(T_{deadline}\) represents the deadline, and \(\Delta P_{grid}\) represents the fluctuation amount of the grid power, that is, the grid fluctuation suppression.

[0058] (3) Network architecture design (Actor network), input state \(s\), output action \(a\), the formula is as follows:

[0059] \(a = \mu(s)+\sigma(s)\cdot\epsilon\);

[0060] Among them, \(\mu(s)\) represents the action mean, \(\sigma(s)\) represents the action standard deviation, and \(\epsilon\) represents \(N(0, 1)\) noise.

[0061] (4) Data collection and preprocessing, the initialization formula is as follows:

[0062]

[0063] Among them, \(s\), \(s'\) represent the number of times, \(\mu_s\) represents the mean of the state features, and \(\sigma_s\) represents the standard deviation of the state features.

[0064] (5) Experience replay helps to reduce the sequential dependence between training samples during random sampling, improve policy stability, and can automatically focus on important samples according to the current learning stage.

[0065] S2. Input the data after normalization and outlier cleaning into the digital twin model to map the operating state of the physical system, and preview the impact of different energy scheduling strategies on the green energy consumption rate and grid frequency in the digital twin model;

[0066] S3. Optimize and update the energy scheduling strategy based on the proximal policy optimization (PPO) algorithm, and perform regulation with the goal of achieving flexible balance by balancing energy demand and energy supply. The specific optimization results are as follows:

[0067] (1) Design multi-objective weighted rewards, balance dynamic weights and constraint penalties, clarify action constraints, and the core formula of multi-objective dynamic weights (PPO):

[0068]

[0069] Among them, \(L(\theta)\) represents the policy gradient loss, represents the policy entropy coefficient, represents the entropy regularization term, By adjusting the policy entropy Impact on the loss function; increase Encourage the policy to explore new actions (entropy increase) to prevent getting stuck in local optima; decrease Bias towards exploiting existing experience (entropy decrease) to improve policy stability;

[0070] β represents the entropy regularization coefficient, and H(st,πθ) represents the entropy regularization term. The intensity of entropy regularization is controlled by encouraging action diversity through the entropy regularization term H(st,πθ); increasing β strengthens action diversity and avoids premature convergence of the policy, while decreasing β reduces diversity and focuses on the efficiency of policy optimization;

[0071] ò represents the truncation threshold, and min(rt+γV(s{t+1})) represents the truncated policy update step size, which limits the policy update amplitude to prevent gradient explosion. The task timeliness weight ensures the timeliness of task completion (such as power dispatch timeliness), where W is the weight coefficient in the multi-objective reward function, and the formula is as follows:

[0072] Rt = W1*F + W2*T - W3*|ΔP_grid|;

[0073] Among them, W1 is the green energy utilization rate weight, which preferentially absorbs green energy such as wind power and photovoltaic power; W2 is the task timeliness weight, which ensures the timeliness of task completion (such as power dispatch timeliness); W3 is the grid power fluctuation penalty weight, which suppresses the grid power fluctuation (|ΔP_grid|), F is the green energy utilization rate, and T is the task timeliness.

[0074] (2) Core formula of safety constraint (TRPO):

[0075] max rt+qV(s{t+1}) - plogπ(qt|st);

[0076] ‖θ - θ{old}‖2 ≤ δ;

[0077] Among them, δ is the new trust region radius, and preventing policy mutation by restricting parameter changes (||θ - θ{old}||2 ≤ δ). Dynamic adjustment: reduce δ during high-risk periods (such as sudden drop in wind power) to enhance stability; increase δ to enhance exploration and avoid premature convergence of the policy; decrease Bias towards exploiting the current optimal action to enhance stability.

[0078] Energy storage SOC constraint: SOC > 20%, preventing over-discharge of the energy storage and ensuring system reliability;

[0079] Grid fluctuation penalty: q.max(0,|ΔP_grid| - 5%) suppresses the grid power fluctuation to ensure power supply safety;

[0080] Increasing λ results in a lower tolerance for power fluctuations (e.g., when the power grid is vulnerable), while decreasing q allows for moderate fluctuations to optimize other objectives (e.g., economy).

[0081] 5% threshold: The baseline for allowable power fluctuations, and exceeding this will trigger penalties.

[0082] (3) Karyotype formula in a high-noise environment:

[0083] Q1(s,p)≈Es′,p′[r + qp′min Q2(s′,p′)];

[0084] Q2(s,p)≈Es′,p′[r + qp′min Q1(s′,p′)];;

[0085] Among them, Q1 and Q2 are used to measure the consistency of Q-value estimation. A large difference will trigger policy correction. By taking the minimum value of the double Q-network, it suppresses the overestimation of Q-values and reduces the policy oscillation caused by overestimation of Q-values in a noisy environment.

[0086] S4. Feed the optimized and updated energy scheduling strategy back to the physical system for execution, monitor the execution effect of the strategy in real time, and update the parameters of the digital twin model to form a closed-loop control.

[0087] Such as Figure 4 As shown, the virtual power plant green energy consumption collaborative system integrating digital twin and reinforcement learning is as follows:

[0088] Digital twin module, used to construct the digital twin model of the green energy power plant, map the operating state of the physical system, and preview the impact of different energy scheduling strategies on the green energy consumption rate and grid frequency;

[0089] Reinforcement learning module, used to build a reinforcement learning agent based on the proximal policy optimization (PPO) algorithm, adjust the working mode of the energy storage module and the power configuration of the supercomputer cluster module, and update the energy scheduling strategy;

[0090] Energy storage module, including aggregated distributed energy, energy storage devices, and adjustable load resources, used to ensure supplementary supply when the power grid needs more electric energy and storage when the power grid has surplus electric energy;

[0091] Supercomputer cluster module, used to provide data processing capabilities, adapt to model calculation and data processing requirements, and based on green energy prediction and load demand response, aggregate distributed energy, energy storage devices, and adjustable load resources through the virtual power plant to achieve local green energy consumption and global optimization.

Claims

1. A collaborative method for virtual power plant green energy consumption integration with digital twin and reinforcement learning, characterized in that, It includes the following steps: S1. Build a digital twin model of a green energy power plant through a simulation tool, collect data on wind power, energy storage power, load demand, grid frequency, and green energy fluctuation characteristics in real time, and perform normalization and outlier cleaning. S2. Input the data after normalization and outlier cleaning into the digital twin model to map the operating state of the physical system, and preview the impact of different energy scheduling strategies on the green energy consumption rate and grid frequency in the digital twin model. S3. Optimize and update the energy scheduling strategy based on the Proximal Policy Optimization (PPO) algorithm. The optimization and update are achieved by constructing a reinforcement learning agent, obtaining training samples through the digital twin model, and dynamically optimizing energy storage charge and discharge instructions, load regulation parameters, and grid interaction strategies. S4. Feed back the optimized and updated energy scheduling strategy to the physical system for execution, monitor the execution effect of the strategy in real time, and update the parameters of the digital twin model to form a closed-loop control.

2. The virtual power plant green energy consumption coordination method according to claim 1, characterized in that The digital twin environment formula in S3 is as follows: Q(s,p)←Q(s,p)+p[r+qp′max Q(s′,p)-Q(s,p)]; Among them, p represents the learning rate, q represents the discount factor, p'max Q(s',p) represents maximizing future rewards, Q(s,p) represents obtaining rewards, and r represents the initial value.

3. The virtual power plant green energy consumption coordination method according to claim 1, characterized in that The algorithm in S3 is as follows: S31. Define the observed variable of energy scheduling for the system, establish a multi-dimensional composite observed variable system for the energy scheduling system, and define the system state parameters with time-varying characteristics. S32. Design a multi-objective weighted reward function, balance dynamic weights and constraint penalties, and clarify action constraints. S33. Design the Actor network architecture, collect data, and perform preprocessing. S34. Adopt an experience replay mechanism, automatically focus on important samples according to the current learning stage, and output optimized instructions for energy storage charge and discharge, load regulation, and grid interaction.

4. The virtual power plant green energy consumption coordination method according to claim 2, characterized in that The formula for defining the system state parameters in S31 is as follows: s = [Pwind(t), SOC(t), Pload(t), fgrid(t)]; Among them, s represents the input state, Pwind(t) represents the real-time wind power, SOC(t) represents the state of charge of the energy storage system, Pload(t) represents the load demand or energy demand, and fgrid(t) represents the grid frequency.

5. The virtual power plant green energy consumption coordination method according to claim 2, characterized in that, The multi-objective weighted reward function in S32 is as follows: Among them, Rt represents the composite reward function, α, β, and γ represent dynamically adjusted coefficients, Egreen represents the usage amount of green energy, that is, the green energy consumption rate, Etotal represents the total energy consumption, Tcomplete represents the task completion time, that is, the task timeliness, Tdeadline represents the deadline, and ΔPgrid represents the fluctuation amount of the grid power, that is, grid fluctuation suppression.

6. The virtual power plant green energy consumption coordination method according to claim 2, wherein, The Actor network architecture in S33 is as follows: a = μ(s)+σ(s)·ò; Among them, μ(s) represents the action mean, σ(s) represents the action standard deviation, and ò represents N(0,1) noise.

7. The collaborative method for virtual power plant green energy consumption according to claim 1, wherein The optimization and update formula in S3 is as follows: Rt = W1*F + W2*T - W3*|ΔP_grid|; max rt+qV(s{t+1})-plogπ(qt|st); ||θ - θ{old}||2 ≤ δ; Q1(s,p) ≈ Es,p′[r + qp′minQ2(s′,p′)];; Q2(s,p) ≈ Es′,p′[r + qp′minQ1(s′,p′)]; Among them, \(L(\theta)\) represents the policy gradient loss, represents the policy entropy coefficient, represents the entropy regularization term, \(\beta\) represents the entropy regularization coefficient, \(H(s_t,\pi_{\theta})\) represents the entropy regularization term, \(\delta\) represents the truncation threshold, \(\min(r_t + \gamma V(s_{t + 1}))\) represents the truncated policy update step size; \(W\) represents the weight coefficient in the multi-objective reward function, \(W_1\) represents the weight of green energy utilization rate, \(W_2\) represents the weight of task timeliness, \(W_3\) represents the weight of power grid power fluctuation penalty, \(|\Delta P_{grid}|\) represents suppressing the power grid power fluctuation, \(F\) represents the green energy utilization rate, \(T\) represents the task timeliness; \(\delta\) represents the new trust region radius, and \(Q_1\), \(Q_2\) are used to measure the consistency of Q-value estimation.

8. A virtual power plant green energy consumption collaborative system integrating digital twin and reinforcement learning, characterized in that, Comprising: A digital twin module for constructing a digital twin model of a green energy power plant, mapping the operating state of the physical system, and previewing the impact of different energy scheduling strategies on the green energy consumption rate and grid frequency; A reinforcement learning module for constructing a reinforcement learning agent based on the Proximal Policy Optimization (PPO) algorithm, adjusting the working mode of the energy storage module and the power configuration of the supercomputer cluster module, and updating the energy scheduling strategy; An energy storage module including aggregated distributed energy, energy storage devices, and adjustable load resources for ensuring supplementary supply when the grid needs more electrical energy and storage when the grid has surplus electrical energy; A supercomputer cluster module for providing data processing capabilities to meet the requirements of model calculation and data processing.

9. An electronic device, comprising: A processor; And a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform the method according to claims 1 to 7.

10. A computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the method according to claims 1 to 7.

Images