A Virtual Power Plant System Scheduling Method Based on Deep Reinforcement Learning

Abstract

This invention discloses a virtual power plant system scheduling method based on deep reinforcement learning, belonging to the field of power energy optimization management technology. The method includes: collecting multi-source real-time data from the virtual power plant; constructing a causal graph model and generating causal feature vectors and adjacency matrices; inputting the causal feature vectors and adjacency matrices into a reinforcement learning framework, performing enhancement processing through a causal attention mechanism to generate an optimized scheduling strategy; extracting economic and safety objective function values ​​from the optimized scheduling strategy, constructing a multi-objective optimization problem, and using gradient descent to track equilibrium points and obtain a flexible scheduling scheme; inputting the flexible scheduling scheme into a high-fidelity digital twin model, performing behavioral risk detection during simulation operation, and outputting a risk report and correction parameter vectors. This invention achieves a leap from statistical correlation analysis to causal reasoning, solving the problem of blind spots in scheduling decisions under high-dimensional uncertainty.

Description

Technical Field

[0001] This invention relates to the field of power energy optimization management technology, and in particular to a virtual power plant system scheduling method based on deep reinforcement learning. Background Technology

[0002] In recent years, with the large-scale grid connection of renewable energy and the deepening of power market reform, Virtual Power Plants (VPPs), as a key technology for aggregating distributed energy resources, have received widespread attention in the field of smart grids. VPPs integrate heterogeneous resources such as distributed photovoltaics, energy storage systems, and controllable loads into a coordinated operating whole through advanced information and communication technologies, participating in grid dispatch and market transactions. Regarding dispatch methods, traditional technologies mainly employ methods such as Optimal Power Flow (OPF) and Model Predictive Control (MPC) based on mathematical models. With the development of artificial intelligence technology, especially the successful application of Deep Reinforcement Learning (DRL) in the control of complex systems, researchers have begun to explore the introduction of DRL into VPP dispatch.

[0003] Despite some progress in existing technologies, two key shortcomings remain: First, in terms of adaptability to dynamic environments, existing DRL methods mostly employ static reward functions and state representations, making it difficult to effectively capture the spatiotemporal correlations and causal characteristics of multi-source data in virtual power plants, resulting in poor performance of the strategies when dealing with sudden operating conditions (such as sudden weather changes or equipment failures); Second, in terms of security assurance, existing methods typically use hard constraints or penalty terms to handle security limitations, lacking real-time risk prediction and correction mechanisms that are deeply integrated with physical mechanisms, making it impossible to ensure system operation safety while pursuing economic efficiency. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a virtual power plant system scheduling method based on deep reinforcement learning to solve the problems of missing spatiotemporal correlation modeling of multi-source data and insufficient real-time risk collaborative management capabilities.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a virtual power plant system scheduling method based on deep reinforcement learning. The method includes: collecting multi-source real-time data from the virtual power plant, constructing a causal graph model, and generating causal feature vectors and adjacency matrices; inputting the causal feature vectors and adjacency matrices into a reinforcement learning framework, performing enhancement processing through a causal attention mechanism to generate an optimized scheduling strategy; extracting economic and safety objective function values ​​from the optimized scheduling strategy, constructing a multi-objective optimization problem, and using gradient descent to track equilibrium points and obtain a flexible scheduling scheme; inputting the flexible scheduling scheme into a high-fidelity digital twin model, performing behavioral risk detection during simulation operation, and outputting risk reports and correction parameter vectors; performing online correction based on the risk reports and correction parameters, generating safety instructions and issuing them to the virtual power plant equipment, while simultaneously collecting feedback execution data and updating the reinforcement learning framework.

[0008] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the multi-source real-time data includes photovoltaic output, load demand, market electricity price, and meteorological information.

[0009] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the specific steps for collecting multi-source real-time data from the virtual power plant and constructing a causal graph model are as follows:

[0010] The buffered data stream is time-aligned and timestamp discrepancies are eliminated to generate an aligned multidimensional time-series dataset.

[0011] The PC algorithm is used to analyze the conditional independence between variables in a multidimensional time series dataset and obtain a preliminary list of causal edges.

[0012] Based on the preliminary list of causal edges, the causal strength is calculated and quantified through statistical tests. At the same time, weakly correlated edges are filtered out to generate a simplified causal relationship matrix.

[0013] Based on a simplified causal relationship matrix, a directed causal graph model is constructed and converted into a directed acyclic graph. At the same time, connectivity and consistency are verified to generate a causal graph model.

[0014] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the specific steps for generating causal feature vectors and adjacency matrices are as follows:

[0015] Multi-head attention feature extraction and softmax normalization are performed on the causal graph model to generate graph node embedding representations and dynamically calibrated topology matrices;

[0016] The graph node embedding representation and dynamically calibrated topology matrix are sorted by feature importance and dynamically weighted to generate an optimized feature set;

[0017] Based on the optimized feature set, combined with the topological information of the dynamically calibrated topological matrix, the features are refined through a dynamic weight adjustment mechanism to obtain causal feature vectors and adjacency matrices.

[0018] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the steps of inputting causal feature vectors and adjacency matrices into the reinforcement learning framework, performing enhancement processing through a causal attention mechanism, and generating an optimized scheduling strategy are as follows:

[0019] The causal feature vector and adjacency matrix are input into the reinforcement learning framework to initialize network parameters and construct the state representation space;

[0020] A spatiotemporal causal attention mechanism is applied to the state representation space, and an enhanced state matrix is ​​generated by performing feature enhancement processing through dynamic integral operation and matrix exponential function.

[0021] The enhanced state matrix is ​​collaboratively optimized using policy networks and value networks to obtain an optimized scheduling strategy.

[0022] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the specific steps for extracting economic objective function values ​​and safety objective function values ​​from the optimized scheduling strategy to construct a multi-objective optimization problem are as follows:

[0023] Economic and safety-related parameters are extracted from the optimized scheduling strategy to generate a parameter dataset;

[0024] Based on the parameter dataset, calculate the economic objective function value and the safety objective function value, and combine them to form objective function value pairs;

[0025] The spatiotemporal causal attention mechanism is used to assign attention weights and fuse features to the target function value pairs, and dynamic weight values ​​are obtained by combining historical state data to generate dynamic weight vectors.

[0026] A multi-objective optimization problem is constructed based on objective function value pairs and dynamic weight vectors.

[0027] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the step of obtaining an elastic scheduling scheme by tracking equilibrium points using gradient descent is as follows:

[0028] The multi-objective optimization problem is transformed into a scalar optimization problem using dynamic weight vectors, and the Pareto solution set is generated by tracking the equilibrium point through gradient descent.

[0029] The optimal solution is selected from the Pareto solution set using fuzzy decision rules to obtain an elastic scheduling scheme.

[0030] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the steps of inputting the elastic scheduling scheme into a high-fidelity digital twin model, performing behavioral risk detection during simulation operation, and outputting a risk report and correction parameter vector are as follows:

[0031] The elastic scheduling scheme is input into the high-fidelity digital twin model to initialize the model and load data, generating an initialized digital twin instance and the loaded scheduling scheme.

[0032] The initialized digital twin instance and the loaded scheduling scheme are simulated and run, and risky behaviors are identified through a dynamic risk detection mechanism to generate a set of risk events;

[0033] Based on a set of risk events, a comprehensive risk score is calculated using the analytic hierarchy process.

[0034] A multi-dimensional correlation mapping is performed on the risk event set and the comprehensive risk score to generate a risk report and a correction parameter vector.

[0035] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the specific steps of performing online correction based on risk reports and correction parameters, generating safety instructions, and issuing them to the virtual power plant equipment are as follows:

[0036] Based on the risk report and correction parameters, an online correction algorithm using double integral transform and sigmoid function is employed to generate a safety instruction vector.

[0037] The security command vector is sent to the virtual power plant equipment through the communication protocol, and the sending timestamp and equipment identifier are recorded to form a command tracking identifier.

[0038] As a preferred embodiment of the virtual power plant system scheduling method based on deep reinforcement learning described in this invention, the steps of simultaneously collecting feedback execution data and updating the reinforcement learning framework are as follows:

[0039] Based on instruction tracking identifiers, real-time collection of feedback execution data from virtual power plant equipment is performed.

[0040] Based on the feedback execution data, the policy network parameters of the reinforcement learning framework are updated using the policy gradient method.

[0041] The beneficial effects of this invention are as follows: by collecting real-time data from multiple sources and constructing a causal graph model, it achieves a leap from statistical correlation analysis to causal reasoning, solving the problem of blind spots in scheduling decisions under high-dimensional uncertainty; through multi-objective gradient equilibrium tracking and digital twin security pre-simulation, it achieves a dynamic trade-off between economy and safety and early warning of risks, achieving the effect of simultaneously optimizing operating costs and system stability in virtual power plant scheduling, breaking through the limitations of single-objective optimization. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a virtual power plant system scheduling method based on deep reinforcement learning.

[0044] Figure 2 A flowchart for generating causal feature vectors and adjacency matrices.

[0045] Figure 3 A flowchart for generating an optimized scheduling strategy.

[0046] Figure 4 A flowchart for generating risk reports and correction parameter vectors. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0050] Reference Figures 1-4 This is one embodiment of the present invention, which provides a virtual power plant system scheduling method based on deep reinforcement learning, including the following steps:

[0051] S1. Collect multi-source real-time data from the virtual power plant, construct a causal graph model, and generate causal feature vectors and adjacency matrices.

[0052] Collect multi-source real-time data from the virtual power plant, and receive and buffer the data through a distributed stream processing pipeline to generate a buffered data stream.

[0053] The specific process includes continuously acquiring photovoltaic output, load demand, market electricity price and meteorological information from relevant data sources inside and outside the virtual power plant. The multi-source real-time data is timestamped and generated in streaming form. The timestamped multi-source real-time data is received through a distributed stream processing pipeline, and the multi-source real-time data is temporarily stored and its order is maintained in the distributed stream processing pipeline to generate a buffered data stream.

[0054] A distributed stream processing pipeline is a computing architecture used to receive, transmit, and temporarily store continuously generated data streams. It consists of multiple processing nodes distributed in a distributed computing network and can receive, sort, and buffer real-time data from different sources in parallel to ensure the consistency of data in time order.

[0055] The buffered data stream is time-aligned and timestamp discrepancies are eliminated to generate an aligned multidimensional time-series dataset.

[0056] The specific process includes time-series alignment of the buffered data stream, which means rearranging and matching the time series from different data sources in the buffered data stream according to a unified time benchmark. By detecting and correcting the timestamp deviations between the time series caused by acquisition equipment or transmission delays, all variables (photovoltaic output, load demand, market electricity price and meteorological information) have corresponding data values ​​at the same time point, thereby generating an aligned multidimensional time series dataset.

[0057] The PC algorithm is used to analyze the conditional independence between variables in a multidimensional time series dataset and obtain a preliminary list of causal edges.

[0058] The specific process includes using the PC algorithm to analyze the conditional independence between variables in a multidimensional time series dataset. This means that for variables such as photovoltaic output, load demand, market electricity price, and meteorological information in the aligned multidimensional time series dataset, the PC algorithm is used to sequentially test the statistical independence of any two variables under different subsets of other variables. Based on the conditional independence test results, edges that do not have a direct causal relationship between variables are gradually eliminated, while the relationships that cannot be eliminated by conditional independence are retained, and a preliminary list of causal edges is obtained.

[0059] It should be noted that the PC algorithm is a constraint-based causal discovery method used to learn the causal structure between variables from observation data. The PC algorithm uses an undirected graph with complete connections between all variables as the initial graph. It checks the conditional independence of variable pairs under different condition sets layer by layer, gradually deletes edges that do not satisfy the dependency relationship, and determines some causal directions by combining orientation rules. Finally, it outputs a sparse graph structure representing the causal relationship between variables.

[0060] Based on the initial list of causal edges, the causal strength is calculated using the following expression:

[0061] ;

[0062] in, Indicates from variable to variable The causal strength, Indicates the causal variable in a causal relationship. Represents the outcome variable in a causal relationship. Represents a time variable. This represents the total length of the time series. Represents the normalization function. Indicates at a point in time Time variable The query vector, This represents the dimensions of the query vector and the key vector. Indicates at a point in time Time and query vector The corresponding key vector, Indicates the first Complete time series data for each variable, Indicates the first Complete time series data for each variable, Indicates the first Complete time series data of the first variable and the first A measure of the amount of information between complete time series data of each variable.

[0063] By quantifying the causal strength through statistical testing and filtering out weakly correlated edges, a simplified causal relationship matrix is ​​generated.

[0064] The specific process includes: for each causal edge in the preliminary causal edge list, the association scores at each time point are calculated using the query vector and key vector in the temporal attention mechanism for the two variables connected by each causal edge; the causal strength values ​​from the cause variable to the result variable are obtained by weighting through softmax normalization and information content measurement; the obtained causal strength values ​​are statistically tested through permutation tests; and the causal relationship is judged based on statistical significance (for example, the time series of the cause variable is randomly shuffled multiple times, the pseudo-causal strength is recalculated, a zero distribution is constructed, and the quantile of the original causal strength in the zero distribution is used as the p-value; if the p-value is less than the preset significance level, such as 0.05, the causal relationship is considered to be statistically significant). Edges that fail the statistical test or are strongly correlated are removed, and strong causal associations with statistical support are retained to generate a simplified causal relationship matrix.

[0065] It should be noted that statistical significance is obtained through hypothesis testing and is used to determine whether the observed causal strength is extremely unlikely to be caused by random factors.

[0066] The significance level is set in advance based on the reliability requirements of statistical inference, and is usually a standard value such as 0.05 or 0.01.

[0067] Temporal attention mechanism refers to the mechanism that calculates the dependencies between different time steps in a time series based on query vector, key vector, and value vector. It is composed of query vector, key vector, value vector, and softmax normalization function.

[0068] Based on a simplified causal relationship matrix, a directed causal graph model is constructed and converted into a directed acyclic graph. At the same time, connectivity and consistency are verified to generate a causal graph model.

[0069] The specific process includes: constructing a directed causal graph model based on a simplified causal relationship matrix. This means connecting the causal edges retained in the simplified causal relationship matrix to the corresponding variable nodes according to the causal direction to form an initial directed causal graph model. At the same time, loop detection is performed on the directed causal graph model to remove or adjust the edges that cause loops and convert it into a directed acyclic graph. Meanwhile, the path connectivity between nodes in the directed acyclic graph and the logical consistency of the causal directions are checked to ensure that there are no contradictory causal paths, and finally, a causal graph model is generated.

[0070] Multi-head attention feature extraction and softmax normalization are performed on the causal graph model to generate graph node embedding representations and dynamically calibrated topology matrices.

[0071] The specific process includes mapping the features of each node in the causal graph model to query vector, key vector, and value vector, obtaining attention weights between nodes through multiple parallel attention heads, measuring node relevance using the dot product of query vector and key vector for each attention head, obtaining normalized attention coefficients through softmax normalization, weighted summing of attention coefficients and value vectors, and fusing the outputs from multiple attention heads to generate graph node embedding representations, while updating the connection strength between nodes according to attention weights to form a dynamically calibrated topology matrix.

[0072] It should be noted that the normalized attention coefficient represents the degree of attention or dependence weight of a node on other nodes under a given state. Its value ranges from 0 to 1, and the sum of the coefficients of all corresponding nodes is 1. This reflects the relative importance of the contributions of each source node in the information aggregation process, and ensures the stability and interpretability of attention allocation.

[0073] The graph node embedding representation and the dynamically calibrated topology matrix are sorted by feature importance and dynamically weighted to generate an optimized feature set.

[0074] The specific process includes ranking the importance of features and optimizing the dynamic weights of the graph node embedding representation and the dynamic calibration topology matrix. This means combining the correlation between the features of each dimension in the graph node embedding representation and the connection strength between nodes in the dynamic calibration topology matrix, evaluating the role of each feature in characterizing the causal structure, ranking all features according to their role, and assigning different weights to the features based on the ranking results. This gives higher weights to features that are more discriminative in expressing causal relationships, weakens the influence of redundant or interfering features, and forms an optimized feature set that balances information integrity and expression efficiency.

[0075] Based on the optimized feature set, combined with the topological information of the dynamically calibrated topological matrix, the features are refined through a dynamic weight adjustment mechanism to obtain causal feature vectors and adjacency matrices.

[0076] The specific process includes fusing the feature representation of each node in the optimized feature set with the node connection structure described by the dynamically calibrated topology matrix, using a dynamic weight adjustment mechanism to reweight the features of each dimension according to the adjacency relationship and feature importance, strengthening the feature components consistent with the causal structure, and suppressing the components that conflict with or are redundant with the topological information, thereby extracting a causal feature vector that more accurately reflects the causal dependency relationship between variables, and generating an adjacency matrix based on the effective connection relationships retained in the dynamically calibrated topology matrix.

[0077] It should be noted that the dynamic weight adjustment mechanism refers to the process of adjusting the weights of each feature dimension in real time based on the feature importance ranking results and the node connection relationships in the dynamic calibration topology matrix. It is a weight update strategy composed of feature importance scores and topology information.

[0078] S2. Input the causal feature vector and adjacency matrix into the reinforcement learning framework, perform enhancement processing through the causal attention mechanism, and generate an optimized scheduling strategy.

[0079] The causal feature vector and adjacency matrix are input into the reinforcement learning framework to initialize network parameters and construct the state representation space.

[0080] The specific process includes feeding the causal feature vector as the attribute input of the node and the adjacency matrix as the input of the structural relationship between the nodes into the reinforcement learning framework. The parameters of the policy network and the value network are initially assigned using the neural network weight initialization method. Based on the graph structure defined by the causal feature vector and the adjacency matrix, a state representation space that can reflect the current operating state and interdependencies of each component of the virtual power plant is established within the reinforcement learning framework.

[0081] Furthermore, the causal feature vector and adjacency matrix are input into the reinforcement learning framework to initialize network parameters and construct a state representation space. A spatiotemporal causal attention mechanism is applied to the state representation space, and an augmented state matrix is ​​generated through dynamic integration and matrix exponential functions. The policy network outputs scheduling actions based on the augmented state matrix, the value network evaluates the value of the corresponding state, and the environment provides feedback rewards and the next state based on the scheduling actions, forming experience samples and storing them in an experience replay buffer. During training, batch experiences are sampled from the buffer, and the policy network parameters are updated using the policy gradient method. At the same time, the value network parameters are adjusted based on temporal difference errors. After multiple rounds of iterative optimization, the trained reinforcement learning framework is finally obtained.

[0082] A spatiotemporal causal attention mechanism is applied to the state representation space, and feature enhancement processing is performed through dynamic integral operation and matrix exponential function to generate an enhanced state matrix.

[0083] The specific process includes: utilizing the state characteristics of each node in the state representation space and their evolutionary relationships in the time dimension, combining the causal dependency structure between variables revealed by the causal graph model, constructing query, key, and value vectors, obtaining spatiotemporal causal attention weights between different time steps and different nodes, and weighting and fusing the spatiotemporal causal attention weights with the state characteristics, modeling the temporal continuity based on this, using dynamic integral operations to accumulate the state influence on the time path, and then using a matrix exponential function to perform nonlinear mapping and stability enhancement on the state transition process, generating an enhanced state matrix that can simultaneously characterize causal relationships, spatiotemporal dynamics, and state interactions.

[0084] It should be noted that the spatiotemporal causal attention mechanism refers to an attention computation method that integrates the time dimension, spatial structure and causal dependency relationship. It is composed of query vector, key vector, value vector in the state representation space, causal direction constraints provided by the causal graph model and the dynamic relationship between time steps.

[0085] The matrix exponential function is a mathematical tool that extends the scalar exponential function to matrices. It is defined as the matrix obtained by expanding a square matrix through a power series. It is often used to solve linear differential equations, describe the state evolution of continuous-time dynamic systems, and achieve nonlinear smooth mapping of matrices in stability analysis and feature enhancement.

[0086] The enhanced state matrix is ​​collaboratively optimized using policy networks and value networks to obtain an optimized scheduling strategy.

[0087] The specific process involves inputting the enhanced state matrix into the policy network to output the probability distribution of scheduling actions for each controllable unit of the virtual power plant. Simultaneously, the enhanced state matrix is ​​input into the value network to evaluate the expected value of the current state's cumulative future reward. During training, both networks share the representation of the enhanced state matrix and update their own parameters based on the policy gradient and temporal difference error. The policy network adjusts its action selection in the direction of improving long-term rewards, while the value network continuously approximates the true state value to provide accurate gradient signals. Through this collaborative optimization mechanism, an optimized scheduling policy that balances economy and safety is obtained.

[0088] It should be noted that the policy network is a neural network that outputs actions or action probability distributions based on the current state (such as a deep neural network with a Softmax output layer for discrete action spaces, or a neural network with Gaussian distribution parameterized outputs for continuous action spaces), while the value network is a neural network that estimates the expected value of future cumulative returns based on the current state (such as a deep neural network with a fully connected layer that outputs scalar values, often used in the Critic part of the Actor-Critic architecture).

[0089] S3. Extract the economic objective function value and the safety objective function value from the optimized scheduling strategy, construct a multi-objective optimization problem, and use the gradient descent method to track the equilibrium point to obtain an elastic scheduling scheme.

[0090] Economic and safety-related parameters are extracted from the optimized scheduling strategy to generate a parameter dataset.

[0091] The specific process includes obtaining economic parameters related to operating costs, market electricity price response, and power generation revenue, as well as safety parameters related to equipment operation constraints, power balance deviation, and reserve capacity margin, based on the scheduling actions determined by the optimized scheduling strategy. These two types of parameters are then organized and arranged according to time steps or scheduling cycles to form a structured parameter dataset.

[0092] Based on the parameter dataset, the economic objective function value and the safety objective function value are calculated and combined to form an objective function value pair, expressed as:

[0093] ;

[0094] ;

[0095] in, This represents the value of the economic objective function. This indicates the total duration of the scheduling cycle. Represents the exponential decay coefficient. This represents the sigmoid function. Indicates the cost scaling factor. Indicates time The total operating cost function at the location, This represents the return scaling factor. Indicates time The payoff function at that point This represents the value of the security objective function. Indicates the voltage scaling factor. Indicates time Voltage stability index function at the location, Represents the error function. Indicates the frequency scaling factor. Indicates time Frequency deviation risk function at a given location.

[0096] It should be noted that the exponential decay coefficient is a parameter used to describe how quickly a quantity decays over time in an exponential manner. It is usually set based on the characteristics of the physical process and historical data fitting.

[0097] The cost scaling factor is a proportional coefficient used to unify the dimensions or adjust the importance of different cost items. It is usually set based on the differences in the magnitude of cost components, the weighting of economic objectives, or historical scheduling experience.

[0098] The cost scaling factor is a proportional coefficient used to unify the dimensions or adjust the importance of different cost items. It is usually set based on the differences in the magnitude of cost components, the weighting of economic objectives, or historical scheduling experience.

[0099] The revenue scaling factor is a proportional coefficient used to align the dimensions of different revenue items or adjust their relative importance. It is usually set according to the magnitude range of the revenue components, the optimization requirements of economic objectives, or the revenue distribution characteristics in historical operating data.

[0100] The frequency scaling factor is a proportional coefficient used to normalize the dimensions or adjust the numerical range of frequency-related variables. It is usually set according to the rated frequency of the power grid, the allowable frequency deviation limit, and the fluctuation characteristics of historical frequency data.

[0101] The specific process includes: using the economic-related parameters contained in the parameter dataset, performing calculations according to the form of an economic objective function to obtain an economic objective function value that reflects the economic performance of the scheduling strategy; simultaneously using the safety-related parameters contained in the parameter dataset, performing calculations according to the form of a safety objective function to obtain a safety objective function value that reflects the safety performance of the scheduling strategy; and pairing the economic objective function value and the safety objective function value corresponding to the same scheduling time or the same scheduling period to form an objective function value pair.

[0102] It should be noted that the economic objective function is a mathematical expression used to quantify the economic performance of the dispatch strategy. It typically includes a weighted sum of terms such as generation costs, electricity purchase costs, market revenue, and start-up / shutdown costs, generally expressed as the negative of total operating costs and the maximization of net revenue. The safety objective function is a mathematical expression used to quantify the safety performance of the dispatch strategy. It typically includes a weighted sum of terms such as voltage deviation, frequency fluctuation, line overload risk, and insufficient reserve capacity, generally expressed as the minimization of safety risk indicators. Both are based on the virtual power plant operating mechanism and predefined optimization objectives.

[0103] The spatiotemporal causal attention mechanism is used to assign attention weights and fuse features to the target function value pairs, and dynamic weight values ​​are obtained by combining historical state data to generate dynamic weight vectors.

[0104] It should be noted that the adjustment coefficient of the attention term is a hyperparameter used to control the concentration of attention weight distribution, and is set through grid search based on the validation performance during model training.

[0105] The specific process includes: using the numerical relationship between the economic objective function value and the safety objective function value in the objective function value pair, normalizing the extreme value range in historical state data, and combining the attention score value of the historical state data set related to each objective with the exponential function and normalization term to calculate the dynamic weight value corresponding to each objective. Then, the dynamic weight values ​​of all objectives are normalized proportionally to form a dynamic weight vector that reflects the change in the importance of each objective under the current scheduling situation.

[0106] It should be noted that historical state data refers to the state records related to dispatch decisions accumulated during the operation of the virtual power plant before the current dispatch time. This includes historical time series of photovoltaic output, historical time series of load demand, historical time series of market electricity prices, historical time series of meteorological information, historical records of equipment operating status, historical data of dispatch action execution results, and corresponding historical economic objective function values ​​and safety objective function values.

[0107] A multi-objective optimization problem is constructed based on objective function value pairs and dynamic weight vectors.

[0108] The specific process includes taking the economic objective function value and the safety objective function value in the objective function value pair as two optimization objectives in the multi-objective optimization problem, and taking the corresponding dynamic weight value in the dynamic weight vector as the relative importance coefficient of each objective under the current scheduling situation. The multi-objective optimization problem is formalized into a mathematical optimization problem that minimizes or maximizes the weighted economic objective function value and the safety objective function value while satisfying the virtual power plant operation constraints.

[0109] It should be noted that the operating constraints of the virtual power plant refer to the operating restrictions such as equipment capacity limits, power balance equations, ramp rate limits, energy storage charge state boundaries, and grid connection specifications. These constraints are obtained based on the technical parameters of each physical device within the virtual power plant, grid dispatching procedures, and the law of conservation of energy.

[0110] The multi-objective optimization problem is transformed into a scalar optimization problem using dynamic weight vectors, and the Pareto solution set is generated by tracking the equilibrium point through gradient descent.

[0111] The specific process involves multiplying the economic objective function value and the safety objective function value by the corresponding dynamic weight values ​​in the dynamic weight vector and then summing the results to form a weighted scalar objective function. This transforms the original multi-objective optimization problem into a scalar optimization problem that optimizes the weighted scalar objective function. The gradient descent method is then used to iteratively update the decision variables along the negative gradient direction of the weighted scalar objective function, continuously tracking the optimal trade-off between economy and safety under the current weight configuration. By repeating this process under different weight combinations or iterative paths, mutually independent optimal solutions are obtained, which is the Pareto solution set.

[0112] It should be noted that gradient descent is an optimization algorithm that finds the minimum value of a function by gradually adjusting the parameters along the opposite direction of the gradient of the objective function.

[0113] The optimal solution is selected from the Pareto solution set using fuzzy decision rules to obtain an elastic scheduling scheme.

[0114] The specific process includes determining the degree of satisfaction in the economic and safety dimensions for the economic objective function value and the safety objective function value corresponding to each solution in the Pareto solution set through the membership function, and then combining the dynamic weight vector to weight and synthesize the degree of satisfaction in each dimension to obtain the comprehensive evaluation value of each solution. Finally, the solution with the highest comprehensive evaluation value is selected as the optimal solution, and the scheduling action sequence corresponding to the optimal solution is determined as the elastic scheduling scheme.

[0115] It should be noted that fuzzy decision rules are a multi-objective decision-making method based on fuzzy set theory. By defining a membership function, the numerical values ​​of each objective are mapped to fuzzy membership degrees that represent the degree of satisfaction. The membership degrees of different objectives are then combined according to weights, thereby selecting the solution with the highest overall satisfaction from multiple candidate solutions that cannot be directly compared. The membership function and weights in fuzzy decision rules are usually pre-set based on decision preferences, historical operating experience, or the importance of the optimization objective.

[0116] S4. Input the elastic scheduling scheme into the high-fidelity digital twin model, perform behavioral risk detection during the simulation operation, and output risk reports and correction parameter vectors.

[0117] The elastic scheduling scheme is input into the high-fidelity digital twin model to initialize the model and load data, generating an initialized digital twin instance and the loaded scheduling scheme.

[0118] The specific process includes: when inputting the flexible dispatch scheme into the high-fidelity digital twin model, establishing an initial state consistent with the real situation in the physical structure, equipment characteristics, current operating status, and external environmental conditions (including photovoltaic output forecast, load demand curve, market electricity price sequence, and meteorological information) of the virtual power plant; and injecting the dispatch instructions of each controllable unit contained in the flexible dispatch scheme into the high-fidelity digital twin model in a time sequence, so that the high-fidelity digital twin model is fully loaded with dispatch actions and operating boundaries before the simulation begins, thereby generating an initialized digital twin instance and a loaded dispatch scheme.

[0119] Furthermore, the high-fidelity digital twin model is constructed based on the physical structure, equipment parameters, operating mechanism, real-time operating status, and external environmental conditions of the virtual power plant. The physical structure includes the connection topology of various distributed energy sources, energy storage devices, and loads; the equipment parameters cover technical characteristics such as power generation efficiency, response time, and capacity limitations; the operating mechanism is reflected in the power flow equation, energy balance relationship, and control logic; the real-time operating status includes the current photovoltaic output, load demand, equipment switching status, and energy storage charging status; and the external environmental conditions include meteorological information, market electricity prices, and grid dispatch instructions. These elements together constitute a high-fidelity digital twin model that is highly consistent with the behavior and response of the real virtual power plant.

[0120] The initialized digital twin instance and the loaded scheduling scheme are simulated and run, and risky behaviors are identified through a dynamic risk detection mechanism to generate a set of risk events.

[0121] The specific process includes executing the scheduling actions specified by the loaded scheduling scheme step by step in the high-fidelity digital twin model according to the time sequence, simulating the dynamic operation of the virtual power plant under the flexible scheduling scheme, and continuously monitoring the changes of key indicators such as the status of each device, power balance, voltage and frequency during the simulation process through a dynamic risk detection mechanism. When an indicator is detected to exceed the safety threshold or an abnormal behavior pattern appears, it is judged as a risky behavior, and each identified risky behavior is recorded according to time, type and severity to generate a risk event set.

[0122] It should be noted that the dynamic risk detection mechanism refers to the method of monitoring key operating indicators and identifying abnormal behaviors in real time during the simulation operation of a high-fidelity digital twin model. It is composed of safety thresholds, equipment status monitoring logic, power balance, and voltage and frequency deviation criteria.

[0123] It should be noted that the safety thresholds are preset based on the technical specifications of the virtual power plant equipment, the power grid operation procedures, and historical safe operation data. Exemplary value ranges include voltage amplitude between 0.95 and 1.05, frequency deviation within ±0.2 Hz, energy storage charging and discharging power not exceeding 100% of rated power, and photovoltaic inverter reactive power output within the range of -0.3 to +0.3 power factor.

[0124] Based on a set of risk events, the comprehensive risk score is calculated using the analytic hierarchy process (AHP), expressed as follows:

[0125] ;

[0126] in, This represents the overall risk score. Represents the hyperbolic tangent function. This indicates the total number of risk events. Indicates a risk event index. Indicates the first The type weighting coefficient of each risk event Indicates the first The severity value of each risk event. Indicates the first The time interval from the occurrence of the risk event to the present. This represents the time decay coefficient.

[0127] It should be noted that the type weight coefficient of the risk event is determined by combining historical operating data with the analytic hierarchy process based on the degree of impact of the risk event on the operational safety of the virtual power plant.

[0128] The time decay coefficient is a parameter used to measure the degree to which the impact of a risk event weakens over time. It is set based on the empirical laws of the persistence effect of risk and the decay curves of the impact of historical events.

[0129] The specific process includes using the type weight coefficient, severity value, and time interval of each risk event recorded in the risk event set, combined with the time decay coefficient, to weight the impact of different risk events, summing the weighted impact of all risk events and inputting it into the hyperbolic tangent function for nonlinear mapping, and finally obtaining a comprehensive risk score that reflects the overall risk level of the current scheduling scheme.

[0130] It should be noted that the Analytic Hierarchy Process (AHP) decomposes complex decision problems into multiple levels such as objectives, criteria, and alternatives. By constructing a judgment matrix, it compares each factor pairwise and performs consistency checks, thereby achieving a multi-criteria decision analysis method that combines qualitative and quantitative approaches.

[0131] A multi-dimensional correlation mapping is performed on the risk event set and the comprehensive risk score to generate a risk report and a correction parameter vector.

[0132] The specific process includes establishing a correspondence between the type, occurrence time, severity, and comprehensive risk score of each risk event in the risk event set in the time, space, and causal dimensions; extracting key risk features and organizing them into structured text according to risk classification rules and weight allocation strategies to generate a risk report; and determining the adjustment direction and magnitude of each scheduling parameter based on the direction and intensity of the impact of risk events on scheduling variables to form a correction parameter vector.

[0133] S5. Perform online calibration based on risk reports and calibration parameters, generate safety instructions and issue them to virtual power plant equipment, and collect feedback execution data to update the reinforcement learning framework.

[0134] Based on the risk report and correction parameters, an online correction algorithm using double integral transform and sigmoid function is employed to generate a safety instruction vector.

[0135] The specific process includes taking the risk type identified in the risk report and the corresponding adjustment amount in the correction parameters as input, performing cumulative smoothing of the correction parameters in the time dimension through the first integral transformation, then combining the sigmoid function to perform nonlinear mapping of the risk severity to generate attention weights, and using the second integral transformation to perform weighted fusion of the attention weights and the smoothed correction parameters in the scheduling action space, and finally outputting the scheduling correction amount that meets the safety constraints to form a safety instruction vector.

[0136] It should be noted that safety constraints refer to the operating boundaries that a virtual power plant must meet during operation, such as equipment capacity limits, power balance requirements, and voltage and frequency stability ranges, which are set according to power grid operation regulations and equipment technical specifications.

[0137] The security command vector is sent to the virtual power plant equipment through the communication protocol, and the sending timestamp and equipment identifier are recorded to form a command tracking identifier.

[0138] The specific process includes sending safety command vectors to virtual power plant equipment via communication protocols. This means using communication protocols that conform to power industry standards to send the scheduling correction quantities contained in the safety command vectors to the corresponding virtual power plant equipment one by one. At the same time, during the sending process, the precise time point of each command is recorded as the sending timestamp, and the unique code of the specific virtual power plant equipment of the safety command vector is associated with it as the equipment identifier. The sending timestamp and the equipment identifier are combined and bound to form a command tracking identifier for full-process traceability.

[0139] Based on instruction tracking identifiers, feedback execution data from virtual power plant equipment is collected in real time.

[0140] The specific process includes locating the corresponding virtual power plant equipment using the device identifier contained in the instruction tracking identifier, determining the time window for the instruction to take effect based on the issued timestamp, continuously acquiring the operating status data of the virtual power plant equipment after the actual execution of the scheduling correction through the communication protocol within the time window, including power output, response delay, execution completion status and abnormal flags, and storing the operating status data in association with the corresponding instruction tracking identifier to form traceable feedback execution data.

[0141] Based on the feedback execution data, the policy network parameters of the reinforcement learning framework are updated using the policy gradient method.

[0142] The specific process includes comparing the actual response results of the virtual power plant equipment contained in the feedback execution data with the corresponding safety command vector to obtain the economic benefits and safety performance brought about by the actual execution, forming an actual reward signal. Then, the actual reward signal is used as the optimization objective of the policy gradient method. Combined with the corresponding state representation in the augmented state matrix, the product gradient of the log probability of the policy network output action and the reward signal is obtained, and the policy network parameters of the reinforcement learning framework are updated along the direction of the product gradient.

[0143] It should be noted that the policy gradient method is a reinforcement learning method that directly optimizes the parameters of the policy network. It updates the parameters along the product gradient of the log probability of the policy on the state-action pair and the corresponding reward, so as to maximize the expected cumulative reward.

[0144] In summary, this invention achieves a leap from statistical correlation analysis to causal reasoning by collecting multi-source real-time data and constructing a causal graph model, thus solving the problem of blind spots in scheduling decisions under high-dimensional uncertainty. Through multi-objective gradient equilibrium tracking and digital twin security pre-simulation, it achieves a dynamic trade-off between economy and safety and early warning of risks, thereby optimizing both operating costs and system stability in virtual power plant scheduling and overcoming the limitations of single-objective optimization.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A virtual power plant system scheduling method based on deep reinforcement learning, characterized in that: include, Collect multi-source real-time data from a virtual power plant, construct a causal graph model, and generate causal feature vectors and adjacency matrices; The causal feature vector and adjacency matrix are input into the reinforcement learning framework, and the causal attention mechanism is used for enhancement processing to generate an optimized scheduling strategy. The economic objective function value and the safety objective function value are extracted from the optimized scheduling strategy to construct a multi-objective optimization problem. The equilibrium point is tracked by the gradient descent method to obtain the elastic scheduling scheme. The elastic scheduling scheme is input into a high-fidelity digital twin model, and behavioral risk detection is performed during the simulation operation to output a risk report and correction parameter vector; Online calibration is performed based on risk reports and calibration parameters, safety instructions are generated and issued to virtual power plant equipment, and feedback execution data is collected to update the reinforcement learning framework.

2. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 1, characterized in that: The multi-source real-time data includes photovoltaic power output, load demand, market electricity prices, and meteorological information.

3. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 2, characterized in that: The specific steps for collecting multi-source real-time data from the virtual power plant and constructing a causal graph model are as follows. The buffered data stream is time-aligned and timestamp discrepancies are eliminated to generate an aligned multidimensional time-series dataset. The PC algorithm is used to analyze the conditional independence between variables in a multidimensional time series dataset and obtain a preliminary list of causal edges. Based on the preliminary list of causal edges, the causal strength is calculated and quantified through statistical tests. At the same time, weakly correlated edges are filtered out to generate a simplified causal relationship matrix. Based on a simplified causal relationship matrix, a directed causal graph model is constructed and converted into a directed acyclic graph. At the same time, connectivity and consistency are verified to generate a causal graph model.

4. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 3, characterized in that: The specific steps for generating the causal feature vector and adjacency matrix are as follows. Multi-head attention feature extraction and softmax normalization are performed on the causal graph model to generate graph node embedding representations and dynamically calibrated topology matrices; The graph node embedding representation and dynamically calibrated topology matrix are sorted by feature importance and dynamically weighted to generate an optimized feature set; Based on the optimized feature set, combined with the topological information of the dynamically calibrated topological matrix, the features are refined through a dynamic weight adjustment mechanism to obtain causal feature vectors and adjacency matrices.

5. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 4, characterized in that: The process involves inputting causal feature vectors and adjacency matrices into a reinforcement learning framework, performing enhancement processing through a causal attention mechanism, and generating an optimized scheduling strategy. The specific steps are as follows: The causal feature vector and adjacency matrix are input into the reinforcement learning framework to initialize network parameters and construct the state representation space; A spatiotemporal causal attention mechanism is applied to the state representation space, and an enhanced state matrix is ​​generated by performing feature enhancement processing through dynamic integral operation and matrix exponential function. The enhanced state matrix is ​​collaboratively optimized using policy networks and value networks to obtain an optimized scheduling strategy.

6. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 5, characterized in that: The steps for extracting economic and safety objective function values ​​from the optimized scheduling strategy and constructing a multi-objective optimization problem are as follows: Economic and safety-related parameters are extracted from the optimized scheduling strategy to generate a parameter dataset; Based on the parameter dataset, calculate the economic objective function value and the safety objective function value, and combine them to form objective function value pairs; The spatiotemporal causal attention mechanism is used to assign attention weights and fuse features to the target function value pairs, and dynamic weight values ​​are obtained by combining historical state data to generate dynamic weight vectors. A multi-objective optimization problem is constructed based on objective function value pairs and dynamic weight vectors.

7. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 6, characterized in that: The process of using gradient descent to track equilibrium points and obtain an elastic scheduling scheme involves the following steps. The multi-objective optimization problem is transformed into a scalar optimization problem using dynamic weight vectors, and the Pareto solution set is generated by tracking the equilibrium point through gradient descent. The optimal solution is selected from the Pareto solution set using fuzzy decision rules to obtain an elastic scheduling scheme.

8. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 7, characterized in that: The process involves inputting the elastic scheduling scheme into a high-fidelity digital twin model, detecting behavioral risks during simulation, and outputting a risk report and correction parameter vector. The specific steps are as follows: The elastic scheduling scheme is input into the high-fidelity digital twin model to initialize the model and load data, generating an initialized digital twin instance and the loaded scheduling scheme. The initialized digital twin instance and the loaded scheduling scheme are simulated and run, and risky behaviors are identified through a dynamic risk detection mechanism to generate a set of risk events; Based on a set of risk events, a comprehensive risk score is calculated using the analytic hierarchy process. A multi-dimensional correlation mapping is performed on the risk event set and the comprehensive risk score to generate a risk report and a correction parameter vector.

9. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 8, characterized in that: The steps for performing online calibration based on risk reports and calibration parameters, generating safety instructions, and issuing them to the virtual power plant equipment are as follows: Based on the risk report and correction parameters, an online correction algorithm using double integral transform and sigmoid function is employed to generate a safety instruction vector. The security command vector is sent to the virtual power plant equipment through the communication protocol, and the sending timestamp and equipment identifier are recorded to form a command tracking identifier.

10. The virtual power plant system scheduling method based on deep reinforcement learning as described in claim 9, characterized in that: The simultaneous collection and feedback of execution data to update the reinforcement learning framework involves the following steps: Based on instruction tracking identifiers, real-time collection of feedback execution data from virtual power plant equipment is performed. Based on the feedback execution data, the policy network parameters of the reinforcement learning framework are updated using the policy gradient method.

Images