A long short-term combination reinforcement learning charging load prediction method and system

Abstract

The application provides a long-short-term combined reinforcement learning charging load prediction method and system, and belongs to the technical field of intelligent power distribution network optimal operation. The method comprises the following steps: acquiring multi-dimensional charging feature data of multiple charging stations in a region at continuous time steps to form a three-dimensional data matrix; extracting a spatial feature vector of each charging station; extracting a time feature vector of each charging station; generating a space-time fusion feature vector; constructing a multi-agent reinforcement learning prediction model, each charging station is defined as an agent, a global state vector is generated based on the local observation state of each agent and the global power grid state information, and the local value function of each agent is combined through a hybrid network to generate a global value function under the condition of meeting the monotonicity constraint to guide each agent to output a charging load prediction value at the next moment. The application deeply fuses the space-time features of the charging load, realizes multi-site collaborative prediction, and improves the prediction accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the field of smart distribution network optimization operation technology, and in particular relates to a long-term and short-term combined reinforcement learning method and system for predicting charging load. Background Technology

[0002] With the rapid development of the electric vehicle industry and its continuously rising penetration rate, charging load has become a key factor affecting the safe and stable operation of the power system. The mobility of electric vehicles, the randomness of charging behavior, and the complexity of bidirectional energy interaction between vehicles and the grid result in significant spatiotemporal heterogeneity and strong dynamic characteristics in charging load, making accurate modeling difficult with traditional load forecasting methods. Currently, the power system faces multiple challenges in electric vehicle charging load forecasting: on the one hand, charging load is influenced by diverse factors such as user travel habits, traffic flow distribution, and battery status, exhibiting both daily periodic patterns and sudden fluctuations in time, and complex spatial relationships between charging stations, making single-dimensional feature extraction methods unable to fully capture its essential characteristics; on the other hand, the dispersed access of massive numbers of electric vehicles leads to a surge in load forecasting scale, and traditional centralized forecasting architectures suffer from high data processing latency and poor coordination, making them unsuitable for multi-site collaborative forecasting needs.

[0003] Existing load forecasting technologies have several limitations: traditional statistical methods rely on manual feature engineering, which is insufficient for characterizing complex spatiotemporal relationships, and the prediction accuracy is greatly affected by data distribution; single neural network models, such as convolutional networks, can only process temporal or spatial features individually, failing to achieve organic integration of the two, resulting in insufficient feature mining; in multi-site forecasting scenarios, the lack of effective collaborative mechanisms makes it difficult to balance local prediction accuracy with global operational constraints, easily leading to mismatches between prediction results and actual grid demand. Furthermore, the high noise characteristics of electric vehicle charging loads further exacerbate the forecasting difficulty, and existing models perform poorly in terms of anti-interference capability and robustness. These problems directly lead to large errors in charging load forecasting, which in turn triggers a chain reaction of difficulties in grid dispatching, decreased power supply reliability, and insufficient resource utilization, severely restricting the large-scale development of vehicle-grid interaction and the optimized operation of smart distribution networks. Therefore, developing an electric vehicle load forecasting method that can deeply integrate spatiotemporal features, possess multi-site collaborative capabilities, and exhibit strong robustness has become a key technical problem urgently needing to be solved in the construction of new power systems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for predicting charging load using a combination of long-term and short-term reinforcement learning.

[0005] The present invention adopts the following technical solution.

[0006] A first aspect of the present invention provides a long-short-term combined reinforcement learning charging load prediction method, comprising: Acquire multi-dimensional charging feature data of multiple charging stations in the region at continuous time steps to form a three-dimensional data matrix; Using graph attention networks, spatial feature vectors of each charging station are extracted from a 3D data matrix. Using a long short-term memory neural network, the time feature vectors of each charging station are extracted from a three-dimensional data matrix; The spatial feature vector and the temporal feature vector belonging to the same charging station are fused to generate a spatiotemporal fusion feature vector; Construct a multi-agent reinforcement learning prediction model, in which each charging station is defined as an agent, and the local observation state of each agent contains at least the spatiotemporal fusion feature vector of the corresponding charging station and the real-time charging load power. A global state vector is generated based on the local observation state of each agent and the global power grid state information. The local value functions of each agent are combined through a hybrid network to generate a global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

[0007] Optionally, acquiring multi-dimensional charging feature data of multiple charging stations within a region at continuous time steps to form a three-dimensional data matrix includes: Based on a pre-built electric vehicle travel and charging model, a three-dimensional data matrix is ​​generated, which contains multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps. The first dimension of the three-dimensional data matrix corresponds to each independent charging station in the region, the second dimension corresponds to the time step sequence divided by fixed time slices, and the third dimension corresponds to the feature set characterizing the charging behavior and electrical state of each charging station. The feature set includes: the number of vehicles arriving at each charging station in each time step, the average parking time, the average charging probability, the average battery capacity, the average initial state of charge, and the charging load power of the previous time step.

[0008] Optionally, the generation of a three-dimensional data matrix based on a pre-built electric vehicle travel and charging model, containing multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps, includes: Using a non-homogeneous Poisson process, the number of vehicles arriving at each charging station in each time step is generated based on the baseline arrival rate of each charging station and the time correction coefficient reflecting intraday time series changes. For each arriving vehicle, the parking time is sampled using a first preset probability distribution, the battery capacity is sampled using a second preset probability distribution, and the initial state of charge is sampled using a third preset probability distribution. Based on the vehicle's parking time, initial state of charge, and reference charging time determined by battery capacity and current available charging power, calculate the probability of the vehicle engaging in charging behavior under the current conditions, and determine whether the vehicle actually engages in charging behavior based on the probability of the charging condition. For all vehicles identified as generating charging behavior at the same charging station within the same time step, their charging energy is aggregated to obtain the charging load power of the charging station at that time step. The number of vehicles arriving at the charging station, average parking time, average charging probability, average battery capacity, and average initial state of charge at that time step are calculated, and the calculation results are filled into a three-dimensional data matrix.

[0009] Optionally, the step of using a graph attention network to extract the spatial feature vectors of each charging station from the three-dimensional data matrix includes: For each fixed time step, feature data of all charging stations at that time step are extracted from the three-dimensional data matrix to form a spatial feature matrix with charging stations as rows and features as columns. The spatial feature matrix is ​​then input into a graph attention network. In the graph attention network, each charging station is defined as a node, and the connecting edges between nodes are used to represent the spatial relationship between the corresponding charging stations. For each node, an adaptive attention mechanism is used to calculate the unnormalized attention coefficients between the node and each of its neighboring nodes, and then the unnormalized attention coefficients are normalized to obtain the attention weights of each neighboring node for that node. Based on attention weights, the features of each neighboring node are weighted and aggregated. The aggregation result is then processed by a nonlinear activation function to generate the spatial feature vector of the charging station corresponding to that node at that time step.

[0010] Optionally, the step of extracting the time feature vector of each charging station from the three-dimensional data matrix using a long short-term memory neural network includes: For each charging station, feature data of the charging station at multiple consecutive time steps are extracted from the three-dimensional data matrix to form a time-series feature sequence, and the time-series feature sequence is input into a long short-term memory neural network. By leveraging the collaborative gating mechanism of the forget gate, input gate, and output gate of the Long Short-Term Memory Neural Network, temporal encoding and dynamic dependency capture are performed on the temporal feature sequence to extract the long-distance dependency of charging load in the time dimension. In the Long Short-Term Memory Neural Network, an input-output residual connection structure is introduced. By linearly mapping the original feature vector input at the current time step, the original feature vector is added element-wise to the hidden state output by the network at that time step to generate a residual-enhanced time feature vector, which is used as the time feature vector of the charging station at that time step.

[0011] Optionally, the generation of the global state vector based on the local observation states of each agent and the global power grid state information includes: For each intelligent agent corresponding to a charging station, the spatiotemporal fusion feature vector is concatenated with the real-time charging load power measurement value of the charging station at the current moment to form the local joint state vector of the intelligent agent. The local joint state vectors of all agents are aggregated along the charging station dimension, and the aggregated result is concatenated with the global grid state information that characterizes the overall operation of the distribution network to generate a global state vector. The global power grid status information includes: the voltage amplitude of multiple preset key power grid nodes, the total charging load of all charging stations in the region, and the load rate of the main transformer.

[0012] Optionally, the step of combining the local value functions of each agent through a hybrid network to generate the global value function corresponding to the global state vector under the condition of satisfying a preset monotonicity constraint includes: Construct a two-layer hybrid network structure that includes a local hybrid layer and a global hybrid layer; The local hybrid layer receives the local value functions of each agent as input, and uses the state-dependent weights and bias terms generated by the first supernetwork based on the global state vector to perform a preliminary nonlinear combination of the local value functions to obtain preliminary fusion features. The global hybrid layer receives the preliminary fusion features and uses the state dependency weights and bias terms generated by the second supernetwork based on the global power grid state information to perform deep fusion on the preliminary fusion features to generate a global value function. In this hybrid network, all parameters are set such that the partial derivative of the global value function with respect to any local value function is always non-negative, thereby satisfying the preset monotonicity constraint.

[0013] Optionally, the guidance for each agent to output the predicted charging load value for the next moment is implemented based on a preset Markov decision process framework, which includes: The state space is represented by the global state vector; Action space is represented by a joint action vector consisting of the predicted charging load values ​​for the next moment output by each agent. The reward function is negatively correlated with the deviation between the predicted charging load of each agent and the actual charging load of the corresponding charging station at the next moment.

[0014] Optionally, the multi-agent reinforcement learning prediction model is implemented using the QMIX multi-agent reinforcement learning framework; Each agent is configured with a local Q-network to output its local value function; The hybrid network is a centralized hybrid network used to nonlinearly combine the local Q values ​​of each agent based on the global state vector, and output a global value function.

[0015] A second aspect of the present invention provides a long-short-term combined reinforcement learning charging load prediction system for implementing the long-short-term combined reinforcement learning charging load prediction method described in the first aspect of the present invention, comprising: The system includes a data acquisition module, a spatial feature extraction module, a temporal feature extraction module, a feature fusion module, and a multi-agent reinforcement learning prediction module. The data acquisition module is used to acquire multi-dimensional charging feature data of multiple charging stations in the area under continuous time steps, forming a three-dimensional data matrix. The spatial feature extraction module is used to extract the spatial feature vectors of each charging station from the three-dimensional data matrix using a graph attention network. The time feature extraction module is used to extract the time feature vector of each charging station from the three-dimensional data matrix using a long short-term memory neural network. The feature fusion module is used to fuse the spatial feature vectors and the temporal feature vectors belonging to the same charging station to generate a spatiotemporal fusion feature vector. The multi-agent reinforcement learning prediction module is used to construct a multi-agent reinforcement learning prediction model, wherein each charging station is defined as an agent, and the local observation state of each agent includes at least the spatiotemporal fusion feature vector and real-time charging load power of the corresponding charging station; and, based on the local observation state of each agent and the global power grid state information, a global state vector is generated, and the local value functions of each agent are combined through a hybrid network to generate the global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

[0016] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. This invention combines graph attention networks with long short-term memory neural networks to extract spatial and temporal features of charging loads and fuse them into a spatiotemporal fusion feature vector. This solves the problem that existing single neural network models can only process spatial or temporal features separately and cannot achieve organic fusion of spatiotemporal features. It achieves a full characterization of the spatiotemporal heterogeneity of charging loads and improves the accuracy of charging load prediction.

[0017] 2. This invention constructs a multi-agent reinforcement learning prediction model, defining each charging station as an agent, and uses a hybrid network to combine the local value functions of each agent to generate a global value function under the condition of satisfying monotonicity constraints. This solves the problems of lack of effective coordination mechanism and difficulty in balancing local prediction and global operation constraints in existing multi-site charging load prediction scenarios, realizes collaborative prediction of multi-site charging load, and improves the matching degree between prediction results and actual grid demand.

[0018] 3. This invention introduces an input-output residual connection structure into the long short-term memory neural network, which adds the original input features of the current time step to the hidden state of the network output after linear mapping. This solves the problem of insufficient stability of model feature extraction under the high noise characteristics of electric vehicle charging load, and enhances the robustness and training stability of the model in complex data environments.

[0019] 4. This invention employs a time-series experience playback mechanism, storing and sampling experience samples of continuous time steps corresponding to the intraday cycle of charging load as indivisible time-series blocks for training. This solves the problem of traditional random sampling disrupting the temporal continuity of charging load, enabling the model to fully learn the dynamic change patterns of load across the entire intraday cycle, and further improving prediction accuracy. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the architecture of the long-term and short-term combined reinforcement learning charging load prediction method of the present invention.

[0021] Figure 2 This is the QMIX multi-agent reinforcement learning decision-making flowchart of the present invention.

[0022] Figure 3 This is the training loss curve of the present invention.

[0023] Figure 4 This is the training reward curve of the present invention.

[0024] Figure 5 This is a graph showing the prediction results of the long-term and short-term combined reinforcement learning charging load prediction method of the present invention.

[0025] Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0027] In Embodiment 1, this invention provides a combined long-term and short-term reinforcement learning method for predicting charging load, such as... Figure 6 As shown, it includes the following steps: Step 1: Obtain multi-dimensional charging feature data of multiple charging stations in the region at continuous time steps to form a three-dimensional data matrix.

[0028] Preferably, step 1 includes: Based on a pre-built electric vehicle travel and charging model, a three-dimensional data matrix is ​​generated, which contains multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps. The first dimension of the three-dimensional data matrix corresponds to each independent charging station in the region, the second dimension corresponds to the time step sequence divided by fixed time slices, and the third dimension corresponds to the feature set characterizing the charging behavior and electrical state of each charging station. The feature set includes: the number of vehicles arriving at each charging station in each time step, the average parking time, the average charging probability, the average battery capacity, the average initial state of charge, and the charging load power of the previous time step.

[0029] More preferably, the generation of a three-dimensional data matrix based on a pre-built electric vehicle travel and charging model, containing multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps, includes: Using a non-homogeneous Poisson process, the number of vehicles arriving at each charging station in each time step is generated based on the baseline arrival rate of each charging station and the time correction coefficient reflecting intraday time series changes. For each arriving vehicle, the parking time is sampled using a first preset probability distribution, the battery capacity is sampled using a second preset probability distribution, and the initial state of charge is sampled using a third preset probability distribution. Based on the vehicle's parking time, initial state of charge, and reference charging time determined by battery capacity and current available charging power, calculate the probability of the vehicle engaging in charging behavior under the current conditions, and determine whether the vehicle actually engages in charging behavior based on the probability of the charging condition. For all vehicles identified as generating charging behavior at the same charging station within the same time step, their charging energy is aggregated to obtain the charging load power of the charging station at that time step. The number of vehicles arriving at the charging station, average parking time, average charging probability, average battery capacity, and average initial state of charge at that time step are calculated, and the calculation results are filled into a three-dimensional data matrix.

[0030] More preferably, the reference charging time represents the theoretical time required to charge the current vehicle from the initial state of charge to a preset target state of charge; the calculation method of the charging condition probability is such that the lower the initial state of charge of the vehicle, or the longer the parking time relative to the reference charging time, the greater the calculated charging condition probability.

[0031] Specifically, the vehicle arrival model, parking duration model, battery capacity model, initial SOC model, and charging probability model are interdependent and have derivational relationships; they do not generate parallel statistical features independently. First, the number of arriving vehicles at each charging station within each time slice is generated based on a non-homogeneous Poisson process. Then, for each arriving vehicle, its parking duration, battery capacity, and initial SOC are sampled. Next, the charging condition probability is calculated based on the vehicle's parking duration and initial SOC, and the charging condition probability is used to determine whether the vehicle engages in charging behavior. Finally, the charging energy of vehicles that engage in charging behavior is aggregated to obtain the charging load power and station-level statistical characteristics of the corresponding charging station within the corresponding time slice.

[0032] Set up a site c In time t Vehicle arrival rate Then in the interval [ t , t +Δ t The number of arriving vehicles within a given time period follows a Poisson distribution. This method can combine high-resolution traffic flow data to achieve a refined fit of vehicle arrival density at different time periods.

[0033]

[0034]

[0035]

[0036] In the formula: For a time period [ t , t +Δ t Accumulated arrival rate within ] For the site c At any moment t Number of vehicles arriving For the site c At any moment Instantaneous arrival rate. , For the number of charging stations, for The number of vehicles arriving at station c within the specified time period. , T This represents the total time. For the site c The baseline arrival rate is obtained by dividing the historical daily average number of vehicles arriving at the station by 24 hours. This is a time correction factor used to characterize the intraday temporal variation of arrival rates.

[0037] It should be noted that the time correction coefficient is positively correlated with the real-time traffic flow of the regional road network: the higher the traffic flow, the higher the coefficient value; the lower the traffic flow, the lower the coefficient value, directly reflecting the impact of traffic flow fluctuations on vehicle arrivals at charging stations. Furthermore, because traffic flow exhibits temporal patterns, the time correction coefficient also follows the objective pattern of intraday bimodal traffic peaks in urban areas.

[0038] Secondly, parking duration The distribution of probability density functions typically exhibits a highly skewed characteristic; therefore, this embodiment uses a log-normal distribution as the first preset probability distribution for fitting to match the characteristics of actual parking behavior. It can accurately depict the vehicle's stopping behavior at stations.

[0039]

[0040]

[0041] In the formula: All are parameters of a log-normal distribution. d The specific value representing the parking duration.

[0042] The battery capacity of vehicle v is The initial SOC is The battery capacity is sampled using a second preset probability distribution:

[0043] The initial state of charge is sampled using a third preset probability distribution:

[0044] In the formula: To represent the specific values ​​that the random variable of battery capacity can take, for One specific value of , where, For the vehicle that arrives at station c at time t v Battery capacity, For the vehicle that arrives at station c at time t v The initial SOC. The probability density function is the standard normal distribution. The cumulative distribution function of the standard normal distribution. and These represent the lower and upper limits of the vehicle's battery capacity, respectively. and These are the mean and standard deviation of battery capacity, respectively. and These are the mean and standard deviation of the initial SOC, respectively.

[0045] To characterize the differences in the time required for different vehicles to charge from their current initial SOC to a target SOC, this invention introduces a reference charging time. This reference charging time indicates the vehicle's charging duration. v Charge from the current initial SOC to the target SOC. The required theoretical reference time varies depending on the vehicle's battery capacity, initial SOC, target SOC, charging efficiency, and actual available charging power; it is not a fixed constant. Therefore, the impact of parking duration on charging probability can be normalized, thus avoiding inconsistencies in time scales between vehicles with different battery capacities and charging powers.

[0046]

[0047]

[0048] In the formula: Rated charging power, For charging efficiency, Rated power of the site, The maximum charging power allowed for the vehicle. It is not a fixed constant, but a vehicle-specific variable that varies with the vehicle's battery capacity, initial SOC, target SOC, charging efficiency, and charging power.

[0049]

[0050] In the formula: Minimum charging time, Allow time for vehicle access, scanning, plugging in the charging port, and starting the engine. The minimum amount of electricity required to determine a valid charge. This is based on the vehicle's parking time meeting certain conditions. Only when it is in this condition is it considered to have the conditions to generate effective charging behavior.

[0051] Therefore, vehicles v charging probability Defined as:

[0052] In the formula: These are bias parameters; This represents the initial SOC influence coefficient. This is the parking duration impact coefficient. Because... When the initial SOC When it decreases, As the size increases, the probability of charging also increases; due to When parking time As the demand increases, the probability of charging also increases. Therefore, this conditional probability function can simultaneously reflect the impact of vehicle charging demand and parking time constraints on charging behavior.

[0053] The site charging load is aggregated, and the site c In time slice t charging load power for:

[0054] After completing the vehicle-by-vehicle charging behavior judgment and charging energy calculation, vehicle samples within the same charging station and the same time slice are aggregated to obtain a station-level feature vector:

[0055] Finally, step 1 outputs a three-dimensional data matrix containing the following: The first dimension (charging station dimension) represents all independent charging stations in the area, denoted as M charging stations, with each charging station being an independent index unit.

[0056] The second dimension (time dimension) represents the complete time series divided into fixed time slices, denoted as T time steps (e.g., 15 minutes or 30 minutes as a time slice), covering the entire prediction period.

[0057] The third dimension (feature dimension) represents the set of charging behavior and electrical features of each charging station at each time step, providing raw data for subsequent spatiotemporal feature extraction.

[0058] The feature dimensions include site-level statistical features aggregated from vehicle-level samples, specifically including: the number of vehicles arriving at each charging station in each time step, average parking time, average charging probability, average battery capacity, average initial state of charge, and charging load power of the previous time step. Therefore, the three-dimensional data matrix can be represented as:

[0059] in, For the number of charging stations, 6 represents the number of time steps, indicating the number of features corresponding to each station at each time step. It should be noted that the parking duration distribution parameters, battery capacity distribution parameters, and initial SOC distribution parameters are used for vehicle-level sample generation; the instantaneous arrival rate, baseline arrival rate, and time correction coefficient are used to determine the number of vehicle arrivals. These are all parameters of the probability generation model in step 1 or intermediate computational costs, and are not used as direct input features for the subsequent graph attention network and long short-term memory neural network.

[0060] Step 2: Use graph attention network to extract spatial feature vectors of each charging station from the 3D data matrix.

[0061] Preferably, step 2 includes: For each fixed time step, feature data of all charging stations at that time step are extracted from the three-dimensional data matrix to form a spatial feature matrix with charging stations as rows and features as columns, and the spatial feature matrix is ​​input into the graph attention network. In the graph attention network, each charging station is defined as a node, and the connecting edges between nodes are used to represent the spatial association between the corresponding charging stations. For each node, the unnormalized attention coefficient between the node and each of its neighboring nodes is calculated through an adaptive attention mechanism, and the unnormalized attention coefficient is normalized to obtain the attention weight of each neighboring node for the node. Based on the attention weights, the features of each neighboring node are weighted and aggregated. After the aggregation result is processed by a nonlinear activation function, a spatial feature vector of the charging station corresponding to that node at that time step is generated. More preferably, before inputting the spatial feature matrix into the graph attention network, the method further includes: filling missing values ​​in the feature data with linear interpolation using similar feature data of the same charging station at adjacent time steps, and performing a preset normalization process on all features.

[0062] Specifically, electric vehicle charging stations are not isolated entities; their load status is affected by road network connectivity and traffic flow distribution, exhibiting significant spatial correlation. To accurately characterize this complex topological relationship, this embodiment employs a Graph Attention Network (GAT) to construct a spatial feature extraction module.

[0063] From the M×T×6 3D data matrix output in step 1, spatial slices are extracted at fixed time steps t to obtain an M×6 spatial feature matrix, which serves as the input to the graph attention network. Each electric vehicle charging station is treated as an independent node, and edges represent the spatial relationships between two charging stations. The input feature vector of each node includes [number of vehicles arriving at station t, average parking time at station t, charging probability at station t, average battery capacity at station t, average initial SOC at station t, and charging load power at station t-1]. All feature data are derived from the 3D basic data matrix generated in step 1.

[0064] In the preprocessing stage, missing value imputation is performed first: if data is missing at a certain time, it is filled by linear interpolation of data from the same station at adjacent time points (t-1 or t+1). Then, all features are normalized using Min-Max.

[0065] Compared to traditional graph convolutional networks, graph attention networks introduce an adaptive attention mechanism that dynamically identifies the influence weights of different neighboring nodes on the central node. First, the node... i with neighboring nodes j Unnormalized attention coefficient:

[0066] In the formula: Attention relevance score; and For nodes i and j Input features; For linear transformation weights, This is the attention weight vector. This represents vector concatenation. This is the leakage rectified linear activation function.

[0067] Subsequently, the attention weights within the neighborhood are normalized using the Softmax function to ensure the probabilistic meaning of the weights:

[0068] In the formula: For nodes j For nodes i Attention weights For nodes i The set of neighboring nodes.

[0069] Finally, the features of neighboring nodes are weighted and aggregated based on the calculated attention weights to extract high-order spatial features containing rich topological information:

[0070] In the formula: For nodes i The output space characteristics, It is a non-linear activation function.

[0071] Step 3: Use a long short-term memory neural network to extract the time feature vectors of each charging station from the three-dimensional data matrix.

[0072] Preferably, step 3 includes: For each charging station, feature data of the charging station at multiple consecutive time steps are extracted from the three-dimensional data matrix to form a time-series feature sequence, and the time-series feature sequence is input into a long short-term memory neural network. The temporal feature sequence is temporally encoded and dynamically dependently captured by the collaborative gating mechanism of the forget gate, input gate and output gate of the long short-term memory neural network, so as to extract the long-distance dependency of charging load in the time dimension. An input-output residual connection structure is introduced into the long short-term memory neural network. By linearly mapping the original feature vector input at the current time step, the original feature vector is added element-wise to the hidden state output by the network at that time step to generate a residual-enhanced time feature vector, which is used as the time feature vector of the charging station at that time step, thereby improving the robustness of feature extraction in high-noise charging load data environments.

[0073] More preferably, before inputting the temporal feature sequence into the long short-term memory neural network, the method further includes: filling missing values ​​in the feature data with linear interpolation using similar feature data from the same charging station at adjacent time steps, and performing a preset normalization process on all features.

[0074] Specifically, the charging load sequence exhibits significant temporal dynamics, including daily periodic regularity and abrupt changes caused by random user behavior. To effectively capture the dynamic dependencies in long sequences, this embodiment employs a Long Short-Term Memory (LSTM) neural network as the core of temporal modeling and makes improvements to address the noise characteristics of the load data.

[0075] From the M×T×6 three-dimensional data matrix output in step 1, time slices are extracted according to a fixed charging station c to obtain a T×6 time-series feature sequence, which is used as the input data for LSTM. This time-series sequence uses the same missing value imputation and Min-Max normalization preprocessing method as the GAT input, but the input is a single-site full-time time-series structure, used to capture the time dependency of the charging load of a single station.

[0076] The input to LSTM is consistent with that of GAT, using the 6 site-level features output in step 1 instead of the original 12 features. Specifically, the LSTM input features include: the number of vehicles arriving at each charging station in each time step, the average parking time, the average charging probability, the average battery capacity, the average initial state of charge, and the charging load power of the previous time step.

[0077] Unlike GAT, which extracts the M×6 spatial feature matrix of all charging stations at a fixed time step t, LSTM extracts the temporal feature sequence of a fixed charging station c across multiple consecutive time steps to learn temporal dependencies. Let the input sequence be... Hidden state and memory unit are respectively Traditional LSTM solves the gradient vanishing problem in long sequence training through the synergistic effect of the forget gate, input gate, and output gate. Its calculation process is as follows:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] In the formula: Here, sigmoid is the activation function, and tanh is the hyperbolic tangent function; For element-wise multiplication, , , and , , , For the gated weight matrix, , , , This is a bias term.

[0084] To enhance the robustness of the model under high-noise EV load environments, this invention introduces a residual mechanism, creating a direct mapping path between layers. This allows gradients to propagate directly across layers, effectively mitigating the degradation problem in deep networks and improving the stability of feature extraction. Specifically, this invention employs input-output residual mapping. The input feature vector at the current time step is linearly mapped and then added to the hidden state output by the LSTM at that time step, forming the residual-enhanced temporal feature representation.

[0085] In the formula: The hidden state after adding residuals. These are the residual mapping weights. The purpose of this design is to preserve key information in the original input features, reduce information decay that may occur during LSTM gating updates, and provide a direct gradient propagation path from input to output, thereby enhancing the training stability and robustness of the model under high-noise charging load data.

[0086] Step 4: Fuse the spatial feature vectors and temporal feature vectors belonging to the same charging station to generate a spatiotemporal fusion feature vector.

[0087] Preferably, step 4 includes: The spatial feature vector and the temporal feature vector corresponding to the same charging station are concatenated sequentially to obtain the original fused vector; The original fusion vector is subjected to a preset normalization process to eliminate the dimensional differences between different feature dimensions, thereby generating the spatiotemporal fusion feature vector.

[0088] Specifically, after spatial and temporal feature extraction, the two types of features need to be fused into a unified spatiotemporal integrated feature vector for use by the subsequent multi-agent reinforcement learning prediction module. The specific steps are as follows: Each charging station corresponds to a feature vector of dimension D (D = spatial feature dimension + temporal feature dimension, default spatial feature dimension = 16, temporal feature dimension = 16, D = 32). The 16-dimensional spatial feature vector output by the graph attention network and the 16-dimensional temporal feature vector output by the long short-term memory neural network are concatenated sequentially to obtain a 32-dimensional original fused vector. The concatenated 32-dimensional vector is then subjected to L2 normalization to eliminate the dimensional differences between different feature dimensions, resulting in the final spatiotemporal fused feature vector.

[0089] Step 4 outputs a two-dimensional "site-fusion feature" matrix of dimension M×32, where M is the number of charging stations. This matrix will serve as the input data for the multi-agent reinforcement learning module in step 5.

[0090] Step 5: Construct a multi-agent reinforcement learning prediction model, where each charging station is defined as an agent; generate a global state vector based on the local observation state of each agent and the global power grid state information.

[0091] Preferably, step 5 includes: A multi-agent reinforcement learning prediction model is constructed, in which each charging station is defined as an agent, and the local observation state of each agent includes at least the spatiotemporal fusion feature vector and real-time charging load power of the corresponding charging station.

[0092] More preferably, the generation of the global state vector based on the local observation states of each agent and the global power grid state information includes: For each intelligent agent corresponding to a charging station, the spatiotemporal fusion feature vector is concatenated with the real-time charging load power measurement value of the charging station at the current moment to form the local joint state vector of the intelligent agent. The local joint state vectors of all agents are aggregated along the charging station dimension, and the aggregated result is concatenated with the global grid state information that characterizes the overall operation of the distribution network to generate the global state vector. The global power grid status information includes: the voltage amplitude of multiple preset key power grid nodes, the total charging load of all charging stations in the region, and the load rate of the main transformer.

[0093] Specifically, to achieve collaborative prediction among multiple charging stations under the same power grid constraints, this embodiment employs the QMIX multi-agent reinforcement learning framework. In the QMIX architecture, each charging station is treated as an independent agent, whose local... Q Value function It relies solely on its own local observation history and actions. To ensure consistency between the local strategy and the global optimal objective, the global... Q Values ​​are obtained through a hybrid network from local Q The values ​​are composed of nonlinear combinations and must satisfy local conditions. Q Monotonicity constraints on the value. Among them... For intelligent agents l Local state information, For intelligent agents l The action.

[0094] The M×32 "Site-Fusion Feature" two-dimensional matrix output in step 4 is split row-wise and assigned row by row to the corresponding charging station agents, giving each agent a 32-dimensional spatiotemporal fusion feature. Simultaneously, the actual charging load power measurement value (not predicted value) of each charging station at the current moment is obtained from the charging station's smart metering terminal or the power grid SCADA system; this measurement value is 1-dimensional data. The 32-dimensional spatiotemporal fusion feature of each agent is concatenated with the 1-dimensional real-time load power to form a 33-dimensional local joint vector.

[0095] Subsequently, the 33-dimensional local joint vectors of all agents are concatenated and aggregated according to the charging station index order to obtain an M×33-dimensional aggregated joint feature. Simultaneously, key node voltages, total regional charging load, and main transformer load rate are collected to construct an 8-dimensional global grid state. Specifically, six key nodes in the distribution network related to the charging station access location, voltage safety constraints, or feeder end operation status are selected, and their voltage amplitudes are collected. The total charging load of all charging stations in the region and the main transformer load rate are also collected. This yields an 8-dimensional global grid state vector:

[0096] in, This is the global power grid state vector. , , , , , These represent the six key nodes at time points. t voltage amplitude, Indicates the total charging load of the area. This indicates the main transformer load rate.

[0097] The aggregated joint features are flattened and then concatenated with the global power grid state to generate a global state vector.

[0098] Finally, the local joint vector and the global state vector are normalized and encapsulated into input tensors for the local Q-network and the hybrid network, respectively. The input tensor of the local Q-network has a dimension of 33, and the input tensor of the hybrid network has a dimension of (33×M+8), awaiting the QMIX algorithm to participate in the computation.

[0099] Step 6: Combine the local value functions of each agent through a hybrid network to generate a global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

[0100] Preferably, the step of combining the local value functions of each agent through a hybrid network to generate a global value function under the condition of satisfying a preset monotonicity constraint includes: Construct a two-layer hybrid network structure that includes a local hybrid layer and a global hybrid layer; The local hybrid layer receives the local value functions of each agent as input, and uses the state-dependent weights and bias terms generated by the first supernetwork based on the global state vector to perform a preliminary nonlinear combination of the local value functions to obtain preliminary fusion features. The global hybrid layer receives the preliminary fusion features and uses the state dependency weights and bias terms generated by the second supernetwork based on the global power grid state information to perform deep fusion on the preliminary fusion features to generate the global value function. In this hybrid network, all parameters are set such that the partial derivative of the global value function with respect to any local value function is always non-negative, thereby satisfying the preset monotonicity constraint.

[0101] More preferably, the guidance for each agent to output the predicted charging load value for the next moment is implemented based on a preset Markov decision process framework, which includes: The state space is represented by the global state vector; Action space is represented by a joint action vector consisting of the predicted charging load values ​​for the next moment output by each agent. The reward function is negatively correlated with the deviation between the predicted charging load of each agent and the actual charging load of the corresponding charging station at the next moment. More preferably, the multi-agent reinforcement learning prediction model is implemented using the QMIX multi-agent reinforcement learning framework; wherein each agent is configured with a local Q-network to output its local value function; the hybrid network is a centralized hybrid network used to nonlinearly combine the local Q-values ​​of each agent based on the global state vector, and output the global value function.

[0102] Specifically, such as Figure 2 As shown, to achieve collaborative prediction among multiple charging stations under the same power grid constraint, this invention employs the QMIX multi-agent reinforcement learning framework. Utilizing monotonicity constraints, the global prediction objective is decomposed into the decision-making processes of each local agent. In the QMIX architecture, each charging station is considered an agent, and each agent's local Q-value function depends only on its own local observation history and actions. To ensure consistency between local policies and the global optimal objective, the global Q-value is nonlinearly combined from local Q-values ​​through a hybrid network, and must satisfy the monotonicity constraints on the local Q-values.

[0103] The global representation is achieved through the QMIX additivity hybrid network. Q value :

[0104] And it satisfies the monotonicity constraint:

[0105] In the formula: s For the global state of the environment, a For the actions of all intelligent agents, It is an additive mixing function for hypernetwork parameterization. , It refers to the number of agents. This monotonicity ensures that when an agent optimizes its own policy (improves local efficiency), it will eventually succeed. Q When the value is high, the overall performance of the global system can also be improved.

[0106] QMIX error function as follows:

[0107] In the formula: As a discount factor, a For multi-agent joint actions, The target Q-value is used to supervise the training of the current Q-network.

[0108] To enhance the collaborative capability of multi-region load forecasting, this invention introduces a two-layer hybrid network in QMIX: First layer (local hybrid layer): Utilizes state-related weights generated by the hypernetwork for individual... Q The values ​​are initially integrated to capture local inter-station collaborative relationships.

[0109]

[0110] In the formula: State-related weights generated by the state generation network; It is a bias term generated by the hypernetwork. As input for the second layer of mixing.

[0111] The second layer (global hybrid layer) deeply integrates the preliminary fusion features with the global state information of the power grid to generate the final joint evaluation. This layer gives the model the ability to correct local prediction behavior from a global perspective.

[0112]

[0113] In the formula: For the state dependency weights of the second layer, This is the second layer bias.

[0114] Specifically, in order to transform the charging load prediction problem into a reinforcement learning task for solution, this embodiment defines the following key elements of the Markov decision process.

[0115] 1) State Space: In this embodiment, the MDP state Consistent with the global state vector constructed in step 5, it is used to characterize the multi-charging station load prediction task at time [time value missing]. The complete environmental state. This state includes the local joint state vector of all charging station agents and the global grid state vector. The local joint state vector of each agent consists of the 32-dimensional spatiotemporal fusion features of the charging station and the 1-dimensional real-time charging load power measurement value; the global grid state vector consists of the voltage amplitude of six key nodes, the total regional charging load, and the main transformer load rate. Therefore, the MDP state can be represented as:

[0116]

[0117] In the formula: The spatiotemporal fusion characteristics of charging station c at time t. This is a real-time charging load power measurement value. For the first c Local joint state vector of each charging station agent.

[0118] Within the QMIX framework, each agent's local Q-network uses its corresponding local joint state vector. As local observation input; the hybrid network uses the global state during the intensive training phase. The local Q-values ​​of each agent are combined. Therefore, local observations... It is the global state. It is a component of MDP states, rather than a separate set of state definitions independent of MDP states.

[0119] 2) Action Space: The agent's decision variables directly correspond to the objective of the prediction task. Actions Defined as the predicted value of the electric vehicle charging load at the next moment. Through interaction with the environment, the agent continuously refines its prediction strategy.

[0120]

[0121] In the formula: Forecast values ​​for electric vehicle charging load.

[0122] 3) Reward function: The feedback value obtained by the agent after taking an action.

[0123]

[0124] In the formula: The actual value of the charging load for electric vehicles.

[0125] 4) Action-value function: Given a state The expected sum of future rewards is given by the action value function. To evaluate:

[0126] In the formula: To map system states to scheduling policies, This represents the discount rate indicating the importance of future rewards relative to current rewards. The goal of this scheduling problem is to find the optimal strategy. To obtain the optimal action-value function :

[0127] Preferably, when training the multi-agent reinforcement learning prediction model, a temporal experience replay mechanism is employed, the mechanism comprising: The empirical samples of a continuous preset number of time steps corresponding to the daily cycle of charging load are stored as an indivisible time series block in the experience playback pool. When sampling training samples from the experience replay pool to update network parameters, complete time series blocks are preferentially extracted so that the model can learn the dynamic changes in load across the entire intraday cycle.

[0128] Specifically, this embodiment introduces a time-series experience playback mechanism, which stores a 24-hour continuous experience sample (corresponding to the daily cycle of charging load) as a time-series block in the playback pool. When sampling, complete time-series blocks are extracted first for training to ensure that the model can learn the load change pattern within the daily cycle.

[0129] In Embodiment 2, this invention provides a long-short-term combined reinforcement learning charging load prediction system to implement the long-short-term combined reinforcement learning charging load prediction method described in Embodiment 1, comprising: The system includes a data acquisition module, a spatial feature extraction module, a temporal feature extraction module, a feature fusion module, and a multi-agent reinforcement learning prediction module. The data acquisition module is used to acquire multi-dimensional charging feature data of multiple charging stations in the area under continuous time steps, forming a three-dimensional data matrix. The spatial feature extraction module is used to extract the spatial feature vectors of each charging station from the three-dimensional data matrix using a graph attention network. The time feature extraction module is used to extract the time feature vector of each charging station from the three-dimensional data matrix using a long short-term memory neural network. The feature fusion module is used to fuse the spatial feature vectors and the temporal feature vectors belonging to the same charging station to generate a spatiotemporal fusion feature vector. The multi-agent reinforcement learning prediction module is used to construct a multi-agent reinforcement learning prediction model, wherein each charging station is defined as an agent, and the local observation state of each agent includes at least the spatiotemporal fusion feature vector and real-time charging load power of the corresponding charging station; and, based on the local observation state of each agent and the global power grid state information, a global state vector is generated, and the local value functions of each agent are combined through a hybrid network to generate the global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

[0130] Figure 1 This is a diagram illustrating the architecture of the present invention. Considering the travel and charging characteristics of electric vehicles, a method for predicting electric vehicle charging load is constructed. This architecture consists of a spatiotemporal feature extraction module and a multi-agent reinforcement learning prediction module. In this embodiment, the multi-agent reinforcement learning prediction module is specifically implemented using the QMIX framework. The feature information of the electric vehicle charging load is input into GAT and LSTM. GAT is responsible for extracting spatial features, and LSTM extracts temporal features. The extracted feature information is input into the QMIX prediction algorithm, which performs load prediction through joint actions of multiple agents and improves the prediction accuracy through reward learning. Finally, the predicted EV charging load is output.

[0131] Figure 3 The curves showing the change in loss value of the proposed method when extracting the time-series characteristics of electric vehicle load are presented. Figure 4 This displays the reward values ​​and average reward obtained during the QMIX training process. Figure 3 As can be seen from the loss curves, the model exhibits good convergence characteristics. In the early stages of training, due to random parameter initialization, both training and test losses are high and fluctuate wildly. As training progresses, the two-layer hybrid network gradually captures the spatiotemporal characteristics of the load, and the curves show a rapid downward trend. Furthermore, the test set loss closely follows the training set loss, indicating no significant overfitting. Looking at the reward curves, the agent's reward value is low and oscillates significantly in the early stages of exploration. As the QMIX algorithm continuously optimizes the joint strategy through monotonicity constraints, the reward value shows an evolutionary pattern of rapid increase, slow fluctuation, and eventual stabilization.

[0132] Figure 5The above are the prediction results of the method proposed in this invention. Electric vehicle charging load exhibits a typical "double peak" pattern, closely matching residents' travel patterns due to time-based clustering caused by commuting behavior. The overall curve shows a low point in the early morning, a slow increase during the day as vehicle activity increases, reaching its peak in the evening, and then rapidly declining as vehicles gradually charge. The predicted curve closely matches the actual curve, indicating that the model accurately captures the dynamic temporal dependence of the load. The load peak is concentrated around 7 PM, with a maximum peak power of 3600 kW. During the low point period from 3 AM to 6 AM, the load remains at a relatively low level.

[0133] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A combined long-term and short-term reinforcement learning method for predicting charging load, characterized in that, include: Acquire multi-dimensional charging feature data of multiple charging stations in the region at continuous time steps to form a three-dimensional data matrix; Using graph attention networks, spatial feature vectors of each charging station are extracted from a 3D data matrix. Using a long short-term memory neural network, the time feature vectors of each charging station are extracted from a three-dimensional data matrix; The spatial feature vector and the temporal feature vector belonging to the same charging station are fused to generate a spatiotemporal fusion feature vector; Construct a multi-agent reinforcement learning prediction model, in which each charging station is defined as an agent, and the local observation state of each agent contains at least the spatiotemporal fusion feature vector of the corresponding charging station and the real-time charging load power. A global state vector is generated based on the local observation state of each agent and the global power grid state information. The local value functions of each agent are combined through a hybrid network to generate a global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

2. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 1, characterized in that: The acquisition of multi-dimensional charging feature data of multiple charging stations within the region at continuous time steps, forming a three-dimensional data matrix, includes: Based on a pre-built electric vehicle travel and charging model, a three-dimensional data matrix is ​​generated, which contains multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps. The first dimension of the three-dimensional data matrix corresponds to each independent charging station in the region, the second dimension corresponds to the time step sequence divided by fixed time slices, and the third dimension corresponds to the feature set characterizing the charging behavior and electrical state of each charging station. The feature set includes: the number of vehicles arriving at each charging station in each time step, the average parking time, the average charging probability, the average battery capacity, the average initial state of charge, and the charging load power of the previous time step.

3. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 2, characterized in that: The three-dimensional data matrix generated based on the pre-built electric vehicle travel and charging model, containing multi-dimensional charging feature data of multiple charging stations at multiple consecutive time steps, includes: Using a non-homogeneous Poisson process, the number of vehicles arriving at each charging station in each time step is generated based on the baseline arrival rate of each charging station and the time correction coefficient reflecting intraday time series changes. For each arriving vehicle, the parking time is sampled using a first preset probability distribution, the battery capacity is sampled using a second preset probability distribution, and the initial state of charge is sampled using a third preset probability distribution. Based on the vehicle's parking time, initial state of charge, and reference charging time determined by battery capacity and current available charging power, calculate the probability of the vehicle engaging in charging behavior under the current conditions, and determine whether the vehicle actually engages in charging behavior based on the probability of the charging condition. For all vehicles identified as generating charging behavior at the same charging station within the same time step, their charging energy is aggregated to obtain the charging load power of the charging station at that time step. The number of vehicles arriving at the charging station, average parking time, average charging probability, average battery capacity, and average initial state of charge at that time step are calculated, and the calculation results are filled into a three-dimensional data matrix.

4. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 1, characterized in that: The method of extracting spatial feature vectors of each charging station from a 3D data matrix using a graph attention network includes: For each fixed time step, feature data of all charging stations at that time step are extracted from the three-dimensional data matrix to form a spatial feature matrix with charging stations as rows and features as columns. The spatial feature matrix is ​​then input into a graph attention network. In the graph attention network, each charging station is defined as a node, and the connecting edges between nodes are used to represent the spatial relationship between the corresponding charging stations. For each node, an adaptive attention mechanism is used to calculate the unnormalized attention coefficients between the node and each of its neighboring nodes, and then the unnormalized attention coefficients are normalized to obtain the attention weights of each neighboring node for that node. Based on attention weights, the features of each neighboring node are weighted and aggregated. The aggregation result is then processed by a nonlinear activation function to generate the spatial feature vector of the charging station corresponding to that node at that time step.

5. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 1, characterized in that: The extraction of time feature vectors for each charging station from the three-dimensional data matrix using a long short-term memory neural network includes: For each charging station, feature data of the charging station at multiple consecutive time steps are extracted from the three-dimensional data matrix to form a time-series feature sequence, and the time-series feature sequence is input into a long short-term memory neural network. By leveraging the collaborative gating mechanism of the forget gate, input gate, and output gate of the Long Short-Term Memory Neural Network, temporal encoding and dynamic dependency capture are performed on the temporal feature sequence to extract the long-distance dependency of charging load in the time dimension. In the Long Short-Term Memory Neural Network, an input-output residual connection structure is introduced. By linearly mapping the original feature vector input at the current time step, the original feature vector is added element-wise to the hidden state output by the network at that time step to generate a residual-enhanced time feature vector, which is used as the time feature vector of the charging station at that time step.

6. The long-term and short-term combined reinforcement learning charging load prediction method according to claim 1, characterized in that: The generation of the global state vector based on the local observation states of each agent and the global power grid state information includes: For each intelligent agent corresponding to a charging station, the spatiotemporal fusion feature vector is concatenated with the real-time charging load power measurement value of the charging station at the current moment to form the local joint state vector of the intelligent agent. The local joint state vectors of all agents are aggregated along the charging station dimension, and the aggregated result is concatenated with the global grid state information that characterizes the overall operation of the distribution network to generate a global state vector. The global power grid status information includes: the voltage amplitude of multiple preset key power grid nodes, the total charging load of all charging stations in the region, and the load rate of the main transformer.

7. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 1, characterized in that: The step of combining the local value functions of each agent through a hybrid network to generate the global value function corresponding to the global state vector under the condition of satisfying a preset monotonicity constraint includes: Construct a two-layer hybrid network structure that includes a local hybrid layer and a global hybrid layer; The local hybrid layer receives the local value functions of each agent as input, and uses the state-dependent weights and bias terms generated by the first supernetwork based on the global state vector to perform a preliminary nonlinear combination of the local value functions to obtain preliminary fusion features. The global hybrid layer receives the preliminary fusion features and uses the state dependency weights and bias terms generated by the second supernetwork based on the global power grid state information to perform deep fusion on the preliminary fusion features to generate a global value function. In this hybrid network, all parameters are set such that the partial derivative of the global value function with respect to any local value function is always non-negative, thereby satisfying the preset monotonicity constraint.

8. The method for predicting charging load using a combination of long-term and short-term reinforcement learning according to claim 1, characterized in that: The method of guiding each agent to output the predicted charging load value for the next moment is based on a preset Markov decision process framework, which includes: The state space is represented by the global state vector; Action space is represented by a joint action vector consisting of the predicted charging load values ​​for the next moment output by each agent. The reward function is negatively correlated with the deviation between the predicted charging load of each agent and the actual charging load of the corresponding charging station at the next moment.

9. The long-term and short-term combined reinforcement learning charging load prediction method according to claim 1, characterized in that: The multi-agent reinforcement learning prediction model is implemented using the QMIX multi-agent reinforcement learning framework. Each agent is configured with a local Q-network to output its local value function; The hybrid network is a centralized hybrid network used to nonlinearly combine the local Q values ​​of each agent based on the global state vector, and output a global value function.

10. A long-short-term combined reinforcement learning charging load prediction system, used to implement the long-short-term combined reinforcement learning charging load prediction method according to any one of claims 1-9, characterized in that, include: The system includes a data acquisition module, a spatial feature extraction module, a temporal feature extraction module, a feature fusion module, and a multi-agent reinforcement learning prediction module. The data acquisition module is used to acquire multi-dimensional charging feature data of multiple charging stations in the area under continuous time steps, forming a three-dimensional data matrix. The spatial feature extraction module is used to extract the spatial feature vectors of each charging station from the three-dimensional data matrix using a graph attention network. The time feature extraction module is used to extract the time feature vector of each charging station from the three-dimensional data matrix using a long short-term memory neural network. The feature fusion module is used to fuse the spatial feature vectors and the temporal feature vectors belonging to the same charging station to generate a spatiotemporal fusion feature vector. The multi-agent reinforcement learning prediction module is used to construct a multi-agent reinforcement learning prediction model, wherein each charging station is defined as an agent, and the local observation state of each agent includes at least the spatiotemporal fusion feature vector and real-time charging load power of the corresponding charging station. Furthermore, a global state vector is generated based on the local observation state of each agent and the global power grid state information. The local value functions of each agent are combined through a hybrid network to generate the global value function corresponding to the global state vector under the condition of satisfying the preset monotonicity constraint, so as to guide each agent to output the charging load prediction value at the next moment.

Images