Virtual power plant multi-temporal-spatial-scale regulation and control system based on deep reinforcement learning

Abstract

The invention discloses a virtual power plant multi-temporal-spatial-scale regulation and control system based on deep reinforcement learning, and relates to the field of intelligent power regulation and control, and the system comprises an acquisition construction module, a definition constraint module, a design optimization module, a strategy training module and a regulation and control execution module. According to the method, by introducing a multi-temporal-spatial feature coding structure combining the graph neural network and the time sequence convolutional network, the spatial coupling relationship and the time dynamic evolution features between distributed energy nodes can be captured at the same time, so that the global perception and prediction capability of a regulation and control strategy to a complex power system is improved. And a deep reinforcement learning algorithm is adopted to continuously train the strategy network, so that the regulation and control system can realize adaptive adjustment and dynamic optimal decision in uncertain environments such as electricity price fluctuation, load sudden change, meteorological disturbance and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent power control technology, and in particular to a virtual power plant multi-temporal scale control system based on deep reinforcement learning. Background Technology

[0002] With the increasing proportion of new energy sources, the operating characteristics of power systems exhibit high volatility and uncertainty. After a large number of distributed energy sources (such as photovoltaic, wind power, energy storage, and adjustable loads) are connected to the grid, traditional centralized dispatching methods struggle to effectively achieve multi-source coordination and flexible response. Virtual power plants, as a key technology for aggregating and optimizing distributed energy sources, can achieve intensive resource utilization and market participation through a unified control mechanism. However, existing virtual power plant dispatching methods are mostly based on static optimization or single-time-scale prediction, lacking characterization of multi-temporal and spatial correlation characteristics, making it difficult to cope with real-time dispatching needs in complex dynamic environments. On the other hand, existing control strategies often rely on linear programming or heuristic algorithms, lacking adaptive learning capabilities, making it difficult to achieve optimal decisions in scenarios with fluctuating electricity prices, changing loads, and dynamically changing equipment constraints. Furthermore, traditional methods suffer from high computational complexity and insufficient real-time performance when dealing with multi-dimensional constraints (such as energy storage safety, line power flow, and voltage stability), limiting the overall economic efficiency and safety of virtual power plants. In recent years, deep reinforcement learning has demonstrated strong self-learning and generalization capabilities in complex decision-making and control problems, continuously optimizing policy functions through interactive learning. However, directly applying reinforcement learning to the control of virtual power plants still faces challenges such as difficulty in modeling multi-temporal and spatial features, unstable strategies, and insufficient feasibility of actions.

[0003] Therefore, there is an urgent need for a virtual power plant intelligent control system that combines multi-temporal scale feature modeling with deep reinforcement learning to achieve collaborative optimization and dynamic decision-making of distributed energy, energy storage systems and adjustable loads at multiple nodes and time scales, so as to improve the system's economy, stability and real-time response capability. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the problem to be solved by this invention is how to achieve coordinated optimization and real-time control of distributed energy, energy storage units and adjustable loads within a virtual power plant under multiple time scales and multiple spatial nodes, and overcome the problems of slow response to dynamic environments, insufficient optimization accuracy and infeasibility of traditional scheduling methods.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a virtual power plant multi-temporal-scale control system based on deep reinforcement learning, comprising: a data acquisition and construction module, used to acquire distributed energy operation parameters and operating environment parameters in the virtual power plant, and to construct a state vector of the distributed energy operation parameters and operating environment parameters according to time steps and spatial nodes;

[0008] The constraint definition module is used to define the set of control actions and apply physical constraints and power grid operation constraints to the set of control actions;

[0009] The design optimization module is used to construct a multi-objective reward function and dynamically adjust the weights of each objective using a weighted strategy.

[0010] The policy training module is used to train the control policy network using deep reinforcement learning algorithms. It performs iterative training of policies based on historical data and simulated environments. A multi-temporal scale feature encoder is introduced during training. Spatial dependencies are extracted through graph neural networks, and temporal dependencies are extracted through temporal convolution.

[0011] The control execution module is used to input real-time status into the control strategy network, generate the optimal control action for each node, and adjust the output of each distributed energy source, energy storage and adjustable load according to the control action.

[0012] As a preferred embodiment of the virtual power plant multi-temporal-scale control system based on deep reinforcement learning described in this invention, wherein: the distributed energy operating parameters and operating environment parameters are used to construct a state vector according to time steps and spatial nodes, including:

[0013] The state vector is constructed by time step t and spatial node n. The state vector is obtained by concatenating the following sub-vectors: ;

[0014] in, For state vectors, This is the instantaneous running parameter vector of the node. This represents the historical time feature vector of the node. For runtime environment parameter vectors, A context information vector;

[0015] A graph neural network is used to spatially fuse the instantaneous operating parameter vectors of each node with the historical time feature vectors of some nodes based on the grid topology adjacency matrix. A time encoder is used to extract the time representation of the node's historical time feature vectors. Then, the spatial representation, time representation, operating environment parameter vector, and context information vector are concatenated in the fusion layer and passed through the attention fusion layer to obtain the final state vector used as input to the policy network.

[0016] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system of the present invention, the system includes: defining a set of control actions, and applying physical constraints and power grid operation constraints to the set of control actions, including:

[0017] The set of control actions includes the adjustment of active power output of each node, energy storage charging and discharging control, adjustable load response, and operation mode switching commands; it sets boundary limits on the power output range of various distributed energy sources, the state of charge of energy storage devices, charging and discharging rates, and load adjustment range; and it limits the total power balance, line power flow capacity, and node voltage stability.

[0018] The feasible domain of the control action is formed by combining physical constraints and power grid operation constraints. During the reinforcement learning training process, the feasibility of the action output by the agent is verified and the constraint is projected. When the output action exceeds the feasible domain, the correction strategy is automatically executed and the action is corrected through hard constraint projection.

[0019] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system of the present invention, wherein: when the output action exceeds the feasible region, a correction strategy is automatically executed to correct the action through hard constraint projection, including:

[0020] First, a feasibility test is conducted on the power adjustment amount, energy storage charging and discharging command, and adjustable load response of each node. If any action parameter exceeds its physical boundary or violates the operating constraints, the action is determined to be an unworkable action.

[0021] When an inoperable action is detected, the constraint projection module is invoked to project the original output action into the feasible domain boundary. The hard constraint projection aims to minimize the deviation between the original action and the corrected action, and adjusts the action parameters while ensuring that the constraints are not violated.

[0022] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system of the present invention, the system comprises: constructing a multi-objective reward function and dynamically adjusting the weights of each objective using a weighted strategy, including:

[0023] Define the immediate composite reward at time step t as the weighted sum of the normalized sub-objectives: ;

[0024] Where M is the number of sub-objectives; Let i be the normalized score of the i-th term. For the corresponding weights;

[0025] When a sub-objective performs below the expected value within the statistical window, its weight is automatically increased; conversely, its weight is decreased.

[0026] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system of the present invention, the system includes: training the control policy network using a deep reinforcement learning algorithm, and performing iterative policy training based on historical data and a simulated environment, including:

[0027] The policy network outputs actions, which are then projected through hard constraints to obtain executable actions. These actions are input into the simulation environment, and the next state and immediate reward are obtained based on the power grid physical model and historical data. The network parameters are updated using policy gradients to maximize the cumulative discount reward. The reward weights are adjusted based on the difference between the performance of each target and the reference value, thereby increasing the weight of low-performing targets.

[0028] Training ends when the cumulative reward change in consecutive training rounds is less than a threshold, the constraint violation rate is lower than a preset value, and the multi-objective performance meets the requirements.

[0029] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system of the present invention, the system includes: introducing a multi-temporal-scale feature encoder during training, extracting spatial dependencies through a graph neural network, and extracting temporal dependencies through temporal convolution, including:

[0030] The distributed energy output, energy storage status, load demand, environmental parameters, and grid operation constraint parameters are organized into an original state matrix by time steps and spatial nodes. The spatial dependencies between nodes are extracted using a graph neural network. The time dependencies are extracted by temporal convolution of the graph neural network output features to generate the final encoded feature vector, which is used as the input of the policy network.

[0031] Using graph neural networks to aggregate power grid nodes and their neighborhoods: ;

[0032] in, Let l be the feature matrix of the nodes in the l-th layer. Given an adjacency matrix with self-loops, For degree matrix, Here, σ represents the trainable weights, and σ is the activation function.

[0033] Temporal convolutional networks are used to extract time series features: ;

[0034] in, Given the input state feature matrix, It is a feature matrix encoded for multiple spatiotemporal scales.

[0035] As a preferred embodiment of the deep reinforcement learning-based virtual power plant multi-temporal-scale control system described in this invention, the system includes: inputting real-time status into the control strategy network to generate the optimal control action for each node; and adjusting the output of each distributed energy source, energy storage, and adjustable load according to the control action, including:

[0036] The encoded features of all nodes are batch-input into the saved policy network parameters for parallel forward inference to obtain the original action suggestion vector for each node; a delay check is performed on the inference results, and if the inference time exceeds the preset real-time budget, the degradation logic is triggered.

[0037] The normalized action output is mapped back to the actual physical quantity according to the rated upper and lower limits of each node and the equipment parameters; the discrete instructions are thresholded to generate control codes for equipment identification.

[0038] In a second aspect, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of the virtual power plant multi-temporal-scale control system based on deep reinforcement learning as described in the first aspect of the present invention.

[0039] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of the virtual power plant multi-temporal-scale control system based on deep reinforcement learning as described in the first aspect of the present invention.

[0040] The beneficial effects of this invention are as follows: By introducing a multi-temporal feature encoding structure combining graph neural networks and temporal convolutional networks, this invention can simultaneously capture the spatial coupling relationships and temporal dynamic evolution characteristics between distributed energy nodes, thereby improving the global perception and prediction capabilities of control strategies for complex power systems. The use of deep reinforcement learning algorithms to continuously train the policy network enables the control system to achieve adaptive adjustment and dynamic optimal decision-making under uncertain environments such as electricity price fluctuations, load abrupt changes, and meteorological disturbances. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the structure of a virtual power plant multi-temporal-scale control system based on deep reinforcement learning; Figure 2This is a diagram of computer equipment for a virtual power plant multi-temporal-scale control system based on deep reinforcement learning. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] Example 1

[0047] Reference Figures 1-2 This is the first embodiment of the present invention. This embodiment provides a virtual power plant multi-temporal scale control system based on deep reinforcement learning. The system consists of an acquisition and construction module, a constraint definition module, a design optimization module, a policy training module, and a control execution module.

[0048] Specifically, the data acquisition and construction module is used to collect the operating parameters and operating environment parameters of distributed energy in the virtual power plant, and to construct a state vector of the operating parameters and operating environment parameters of distributed energy according to time steps and spatial nodes.

[0049] Furthermore, the state vector is constructed using time step t and spatial node n, and is obtained by concatenating the following sub-vectors: ;

[0050] in, For state vectors, This is the instantaneous running parameter vector of the node. This represents the historical time feature vector of the node. For runtime environment parameter vectors, A context information vector;

[0051] A graph neural network is used to spatially fuse the instantaneous operating parameter vectors of each node with the historical time feature vectors of some nodes based on the grid topology adjacency matrix. A time encoder is used to extract the time representation of the node's historical time feature vectors. Then, the spatial representation, time representation, operating environment parameter vector, and context information vector are concatenated in the fusion layer and passed through the attention fusion layer to obtain the final state vector used as input to the policy network.

[0052] It should be noted that the adjacency matrix A∈R is constructed using the power grid topology. N×N If there is a physical connection between nodes i and j, then A ij =1, otherwise 0. The graph convolutional propagation layer performs the following operations:

[0053] ;

[0054] in, W is the normalized adjacency matrix. g For trainable weight matrix, The node feature input matrix is ​​σ, where σ is the activation function.

[0055] After multi-layer propagation, the spatial fusion feature Z of each node is obtained. tn .

[0056] Specifically, the constraint definition module is used to define the set of control actions and apply physical constraints and power grid operation constraints to the set of control actions.

[0057] Furthermore, the set of control actions includes adjustments to the active power output of each node, control of energy storage charging and discharging, adjustable load response, and commands for switching operating modes; it sets boundary limits on the power output range of various distributed energy sources, the state of charge of energy storage devices, charging and discharging rates, and load adjustment range; and it limits the total power balance, line power flow capacity, and node voltage stability.

[0058] The feasible domain of the control action is formed by combining physical constraints and power grid operation constraints. During the reinforcement learning training process, the feasibility of the action output by the agent is verified and the constraint is projected. When the output action exceeds the feasible domain, the correction strategy is automatically executed and the action is corrected through hard constraint projection.

[0059] Furthermore, the feasibility of the power adjustment, energy storage charging and discharging commands, and adjustable load response of each node is first tested. If any action parameter exceeds its physical boundary or violates the operating constraints, the action is determined to be an unworkable action.

[0060] When an inoperable action is detected, the constraint projection module is invoked to project the original output action into the feasible domain boundary. The hard constraint projection aims to minimize the deviation between the original action and the corrected action, and adjusts the action parameters while ensuring that the constraints are not violated.

[0061] It should be noted that the reinforcement learning agent outputs the original action... Then, the system performs a feasibility check:

[0062] like If so, the action can be executed directly; if If so, proceed to the projection correction step;

[0063] The Euclidean distance minimization strategy is modified to project non-compliant actions to the feasible region boundary;

[0064] After modifying the motion input system simulation environment, the instantaneous reward function is calculated:

[0065] ;

[0066] Among them, R t Let P be the reward value at time t. imbalance C represents the total power imbalance of the system. operation For system operation and control costs, P violation To address penalties for exceeding limits or violating restrictions, , , These are the weighting coefficients.

[0067] Furthermore, simulations were conducted on an IEEE 33-node system: the total capacity of distributed generation was 5.2MW, the rated power of the energy storage system was 1MW, the energy storage capacity was 2MWh, and the allowable node voltage range was 0.95-1.05pu.

[0068] Scheduling interval Δt = 15 min;

[0069] The following table shows the simulation results:

[0070]

[0071] Table 1 Comparison of system performance before and after constraint projection correction

[0072] Table 1 illustrates that: without the correction mechanism, the average power balance deviation of the system was 83.5kW, which decreased to 7.8kW after constraint projection correction, a reduction of approximately 90.7%, indicating that the present invention can effectively maintain the dynamic power balance of the system; without the correction scheme, the node voltage exceeded the limit 14 times, while the present method only caused 1 time, a reduction of 92.9%, verifying the effectiveness of constraint projection in voltage stability control; the line overload rate decreased from 11.3% to 0.8%, indicating that the correction mechanism can suppress excessive power adjustment of reinforcement learning output and avoid the risk of line overload operation; the average reward value increased from -0.73 to -0.18, indicating that the overall performance of the system operation has been significantly optimized.

[0073] Specifically, the design optimization module is used to construct a multi-objective reward function and dynamically adjust the weights of each objective using a weighted strategy.

[0074] Furthermore, the immediate composite reward at time step t is defined as the weighted sum of the normalized sub-objectives: ;

[0075] Where M is the number of sub-objectives; Let i be the normalized score of the i-th term. For the corresponding weights;

[0076] When a sub-objective performs below the expected value within the statistical window, its weight is automatically increased; conversely, its weight is decreased.

[0077] It should be noted that the following sub-objectives are selected in the scheduling optimization, and the number of sub-objectives is set to M=3: Sub-objective 1: Minimize power balance error; Sub-objective 2: Minimize node voltage over-limit rate; Sub-objective 3: Minimize economic operating cost;

[0078] At each time step, the composite reward serves as a feedback signal for the reinforcement learning scheduling algorithm. Through dynamic changes in weights, the algorithm can automatically balance the importance of different objectives, for example:

[0079] If the system experiences frequent voltage exceedances and its voltage stability score decreases, the system will automatically improve and prioritize voltage adjustment; if the power balance error remains large, the power consumption will be increased. 1,t+1 Once the system stabilizes, the weights gradually return to their initial equilibrium state.

[0080]

[0081] Table 2 Simulation results of instantaneous composite reward adaptive weighting

[0082] Table 2 shows that when the voltage stability score decreases in steps 11-20, the system automatically increases w2 to guide the scheduling strategy to prioritize voltage issues; in steps 21-30, both voltage and power balances return to normal, and the weight redistribution tends to be balanced; the final composite reward increases by about 9.5%, indicating that the adaptive weighting mechanism significantly improves the stability and overall performance of the scheduling algorithm.

[0083] Specifically, the policy training module is used to train the control policy network using deep reinforcement learning algorithms. It iterative training of policies is carried out based on historical data and simulated environments. A multi-temporal scale feature encoder is introduced during training. Spatial dependencies are extracted through graph neural networks, and temporal dependencies are extracted through temporal convolution.

[0084] Furthermore, the policy network outputs actions, which are then projected through hard constraints to obtain executable actions. These actions are input into the simulation environment, and the next state and immediate reward are obtained based on the power grid physical model and historical data. The network parameters are updated using policy gradients to maximize the cumulative discount reward, and the reward weights are adjusted based on the difference between the performance of each target and the reference value, thereby increasing the weight of low-performing targets.

[0085] Training ends when the cumulative reward change in consecutive training rounds is less than a threshold, the constraint violation rate is lower than a preset value, and the multi-objective performance meets the requirements.

[0086] Furthermore, the distributed energy output, energy storage status, load demand, environmental parameters, and grid operation constraint parameters are organized into an original state matrix by time steps and spatial nodes. The spatial dependencies between nodes are extracted using a graph neural network, and the time dependencies are extracted by temporal convolution of the graph neural network output features to generate the final encoded feature vector, which is used as the input to the policy network.

[0087] Using graph neural networks to aggregate power grid nodes and their neighborhoods: ;

[0088] in, Let l be the feature matrix of the nodes in the l-th layer. Given an adjacency matrix with self-loops, For degree matrix, Here, σ represents the trainable weights, and σ is the activation function.

[0089] Temporal convolutional networks are used to extract time series features: ;

[0090] in, Given the input state feature matrix, It is a feature matrix encoded for multiple spatiotemporal scales.

[0091] It should be noted that, based on the power grid topology, a graph neural network is used to perform spatial information aggregation processing on the node states. The specific method includes inputting the state vectors of each node and its neighboring nodes into the graph neural network; performing multi-layer aggregation on the node features so that each layer can capture the spatial dependencies from direct neighbors to remote nodes; using residual connections and normalization strategies in the aggregation process to ensure the stability and differences of multi-layer features during propagation; and obtaining the encoded feature vector of each node under the spatial structure, reflecting the mutual coupling relationship between nodes.

[0092] The obtained spatial features are input into a temporal convolutional network in time series for time dependency extraction. Specific methods include: using causal convolution to ensure that only historical time step information is used; and adopting a multi-scale convolutional kernel structure to enable the network to capture dynamic changes at different time scales, such as minute-level fluctuations and daily cycle trends.

[0093] The stability of convolutional layer features and training convergence are ensured by residual connections, normalization, and activation functions; the output is a multi-temporal feature representation of each node in the time dimension, describing the law of node running state evolution over time.

[0094] The spatial and temporal features are fused to form the final multi-spatiotemporal coding feature vector. Specific methods include: concatenating or weighting the spatial and temporal features; introducing an attention mechanism to dynamically adjust the weights of the spatial and temporal features according to the current operating scenario; and outputting a comprehensive feature vector, enabling the policy network to generate the optimal control action based on this feature vector. This feature vector not only contains the spatial coupling relationship between nodes but also reflects the dynamic change trend of nodes at different time scales, ensuring the accuracy and robustness of the agent's decision-making.

[0095] Specifically, the control execution module is used to input the real-time status into the control strategy network and generate the optimal control action for each node; and adjust the output of each distributed energy source, energy storage and adjustable load according to the control action.

[0096] Furthermore, the encoded features of all nodes are batch-input into the saved policy network parameters for parallel forward inference to obtain the original action suggestion vector for each node; a delay check is performed on the inference results, and if the inference time exceeds the preset real-time budget, degradation logic is triggered.

[0097] The normalized action output is mapped back to the actual physical quantity according to the rated upper and lower limits of each node and the equipment parameters; the discrete instructions are thresholded to generate control codes for equipment identification.

[0098] It should be noted that a delay check mechanism is introduced to ensure that action generation meets the system's real-time requirements. Specifically, this includes: recording the computation time of the policy network's forward inference; determining whether the inference time exceeds the preset real-time budget; if it exceeds the budget, triggering degradation logic, such as using the last valid action vector as the current action; simplifying the action, adjusting only the main load or critical nodes; and ensuring the system maintains safe operation even under network congestion or peak computational load conditions.

[0099] The normalized action vectors output by the policy network are mapped back to the actual physical control quantities of each node. Specifically, this involves: linearly denormalizing each action vector based on parameters such as the node's rated power, state of charge, and load regulation range; generating physical quantity commands that can directly drive the equipment, such as power output, energy storage charging and discharging rates, and load regulation amplitudes; and performing boundary checks on the upper and lower limits to ensure that the action values ​​do not exceed the safe operating range of the equipment. This step guarantees the executability and physical controllability of the action recommendations, avoiding equipment damage or grid safety risks.

[0100] For discrete actions included in the strategy network output, such as operating mode switching, on / off operations, and equipment start / stop, thresholding is performed to generate control codes. Specifically, this includes: determining the state corresponding to the action vector based on a threshold (e.g., "on" for actions greater than the threshold, "off" for actions less than the threshold); encoding the discrete actions into control codes recognizable by the equipment for direct execution by the downstream control system; and generating a unique control code for each node, matching it with the actual control equipment. This step ensures seamless integration between continuous output actions and discrete commands, enabling overall control and execution of the virtual power plant.

[0101] Example 2

[0102] This embodiment is the second embodiment of the present invention. This embodiment provides a virtual power plant multi-temporal scale control system based on deep reinforcement learning. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0103] Step 1: Collect operational data from all distributed energy sources, energy storage devices, and adjustable loads within the virtual power plant, including instantaneous power, state of charge, and load demand. Collect environmental data, such as irradiance, wind speed, and temperature, as well as grid constraint information (node ​​voltage, line capacity, and power flow limits).

[0104] The original state matrix is ​​constructed by time steps and spatial nodes, forming the state vector of each node, which includes the node's instantaneous operating parameters, historical time characteristics, operating environment parameters, and context information.

[0105] Step 2: Utilize a graph neural network to perform multi-layer feature aggregation on each node and its neighbors, capturing the coupling relationships between nodes. Use a temporal convolutional network to perform multi-scale convolution on the historical features of nodes to extract temporal dependencies. Attention-weighted fusion of spatial features, temporal features, runtime parameters, and contextual information generates the final multi-temporal-spatial encoded feature vector, which serves as the input to the policy network.

[0106] Step 3: Input the multi-temporal encoded features in batches into the saved policy network parameters for parallel forward inference to obtain the original action suggestion vector for each node. Perform a delay check on the inference time; if it exceeds the preset real-time budget, trigger degradation logic and adopt the previous round's action or simplify control of key nodes. Map the normalized actions back to physical quantities and generate executable instructions based on the node's rated power, energy storage charging and discharging rate, and load adjustment range. Threshold discrete actions (such as switching control and mode switching) to generate control codes that the device can recognize.

[0107] Step 4: Input the actions into the virtual simulation environment to simulate power grid operation and calculate the next state and immediate reward. Calculate node rewards based on the multi-objective reward function (power balance, line safety, voltage stability, energy utilization efficiency), and dynamically adjust the weights of each sub-objective based on reward performance. Update the policy gradient of the policy network to maximize the cumulative discounted reward.

[0108] Step 5: Training ends when the cumulative reward change over consecutive training rounds is less than a set threshold, the constraint violation rate is lower than a preset value, and the multi-objective performance metrics meet the requirements. Save the final policy network parameters for real-time adjustment.

[0109] The following results were obtained through verification on the IEEE 33-node distribution network and multiple distributed energy simulation platform:

[0110]

[0111] Table 3 Comparison of Simulation Results of Multi-Spatiotemporal Scale Regulation in Virtual Power Plant

[0112] Table 3 illustrates that this invention significantly improves power balance accuracy through multi-temporal feature encoding and a deep reinforcement learning strategy network, reducing the average deviation by approximately 91%. The number of voltage overruns and line overloads is greatly reduced, improving system safety and stability. The increased multi-objective reward value indicates that the system has significant advantages in multi-objective optimization such as power balance, load response, and energy storage regulation. Simulation verification shows that this invention can achieve efficient, robust, and real-time virtual power plant node control in complex power grid operation scenarios.

[0113] This embodiment also provides a computer device suitable for a deep reinforcement learning-based virtual power plant multi-temporal-scale control system, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the deep reinforcement learning-based virtual power plant multi-temporal-scale control system proposed in the above embodiment.

[0114] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0115] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the virtual power plant multi-temporal scale control system based on deep reinforcement learning as proposed in the above embodiments.

[0116] In summary, this invention, by introducing a multi-temporal feature encoding structure combining graph neural networks and temporal convolutional networks, can simultaneously capture the spatial coupling relationships and temporal dynamic evolution characteristics between distributed energy nodes, thereby enhancing the global perception and prediction capabilities of control strategies for complex power systems. By employing deep reinforcement learning algorithms to continuously train the strategy network, the control system can achieve adaptive adjustment and dynamic optimal decision-making under uncertain environments such as electricity price fluctuations, load abrupt changes, and meteorological disturbances.

[0117] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A virtual power plant multi-temporal-scale control system based on deep reinforcement learning, characterized in that, include: The data acquisition and construction module is used to collect the operating parameters and operating environment parameters of distributed energy in the virtual power plant, and construct a state vector of the operating parameters and operating environment parameters of distributed energy according to time steps and spatial nodes. The constraint definition module is used to define the set of control actions and apply physical constraints and power grid operation constraints to the set of control actions; The design optimization module is used to construct a multi-objective reward function and dynamically adjust the weights of each objective using a weighted strategy. The policy training module is used to train the control policy network using deep reinforcement learning algorithms. It performs iterative training of policies based on historical data and simulated environments. A multi-temporal scale feature encoder is introduced during training. Spatial dependencies are extracted through graph neural networks, and temporal dependencies are extracted through temporal convolution. The control execution module is used to input real-time status into the control strategy network, generate the optimal control action for each node, and adjust the output of each distributed energy source, energy storage and adjustable load according to the control action.

2. The virtual power plant multi-temporal-scale control system based on deep reinforcement learning as described in claim 1, characterized in that, The process of constructing a state vector by time steps and spatial nodes for distributed energy operating parameters and operating environment parameters includes: The state vector is constructed by time step t and spatial node n. The state vector is obtained by concatenating the following sub-vectors: ； in, For state vectors, This is the instantaneous running parameter vector of the node. This represents the historical time feature vector of a node. For runtime environment parameter vectors, A context information vector; A graph neural network is used to spatially fuse the instantaneous operating parameter vectors of each node with the historical time feature vectors of some nodes based on the grid topology adjacency matrix. A time encoder is used to extract the time representation of the node's historical time feature vectors. Then, the spatial representation, time representation, operating environment parameter vector, and context information vector are concatenated in the fusion layer and passed through the attention fusion layer to obtain the final state vector used as input to the policy network.

3. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 1, characterized in that: The defined set of control actions applies physical constraints and power grid operation constraints to the set of control actions, including: The set of control actions includes the adjustment of active power output of each node, energy storage charging and discharging control, adjustable load response, and operation mode switching commands; it sets boundary limits on the power output range of various distributed energy sources, the state of charge of energy storage devices, charging and discharging rates, and load adjustment range; and it limits the total power balance, line power flow capacity, and node voltage stability. The feasible domain of the control action is formed by combining physical constraints and power grid operation constraints. During the reinforcement learning training process, the feasibility of the action output by the agent is verified and the constraint is projected. When the output action exceeds the feasible domain, the correction strategy is automatically executed and the action is corrected through hard constraint projection.

4. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 3, characterized in that: When the output action exceeds the feasible region, an automatic correction strategy is executed to correct the action through hard constraint projection, including: First, a feasibility test is conducted on the power adjustment amount, energy storage charging and discharging command, and adjustable load response of each node. If any action parameter exceeds its physical boundary or violates the operating constraints, the action is determined to be an unworkable action. When an inoperable action is detected, the constraint projection module is invoked to project the original output action into the feasible domain boundary. The hard constraint projection aims to minimize the deviation between the original action and the corrected action, and adjusts the action parameters while ensuring that the constraints are not violated.

5. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 1, characterized in that, The construction of the multi-objective reward function, employing a weighted strategy to dynamically adjust the weights of each objective, includes: Define the immediate composite reward at time step t as the weighted sum of the normalized sub-objectives: ； Where M is the number of sub-objectives; Let i be the normalized score of the i-th term. For the corresponding weights; When a sub-objective performs below the expected value within the statistical window, its weight is automatically increased; conversely, its weight is decreased.

6. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 1, characterized in that: The method of training the control policy network using deep reinforcement learning algorithms, and iteratively training the policy based on historical data and simulated environments, includes: The policy network outputs actions, which are then projected through hard constraints to obtain executable actions. These actions are input into the simulation environment, and the next state and immediate reward are obtained based on the power grid physical model and historical data. The network parameters are updated using policy gradients to maximize the cumulative discount reward. The reward weights are adjusted based on the difference between the performance of each target and the reference value, thereby increasing the weight of low-performing targets. Training ends when the cumulative reward change in consecutive training rounds is less than a threshold, the constraint violation rate is lower than a preset value, and the multi-objective performance meets the requirements.

7. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 1, characterized in that: The method of introducing a multi-spatiotemporal scale feature encoder during training, extracting spatial dependencies through a graph neural network, and extracting temporal dependencies through temporal convolution includes: The distributed energy output, energy storage status, load demand, environmental parameters, and grid operation constraint parameters are organized into an original state matrix by time steps and spatial nodes. The spatial dependencies between nodes are extracted using a graph neural network. The time dependencies are extracted by temporal convolution of the graph neural network output features to generate the final encoded feature vector, which is used as the input of the policy network. Using graph neural networks to aggregate power grid nodes and their neighborhoods: ； in, Let l be the feature matrix of the nodes in the l-th layer. Given an adjacency matrix with self-loops, For degree matrix, Here, σ represents the trainable weights, and σ is the activation function. Temporal convolutional networks are used to extract time series features: ； in, Given the input state feature matrix, It is a feature matrix encoded for multiple spatiotemporal scales.

8. The virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in claim 1, characterized in that: The real-time status is input into the control strategy network to generate the optimal control action for each node; Adjusting the output of distributed energy sources, energy storage, and adjustable loads according to control actions includes: The encoded features of all nodes are batch-input into the saved policy network parameters for parallel forward inference to obtain the original action suggestion vector for each node; a delay check is performed on the inference results, and if the inference time exceeds the preset real-time budget, the degradation logic is triggered. The normalized action output is mapped back to the actual physical quantity according to the rated upper and lower limits of each node and the equipment parameters; the discrete instructions are thresholded to generate control codes for equipment identification.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the virtual power plant multi-temporal scale control system based on deep reinforcement learning as described in any one of claims 1 to 8.

Images