Park electric vehicle charging and discharging control method and system based on reinforcement learning and storage medium

Abstract

The invention discloses a park electric vehicle charging and discharging control method and system based on reinforcement learning and a storage medium, and the method comprises the following steps: S1, obtaining related historical data, and constructing a historical environment state; s2, predicting the total load of a plurality of time steps of the park in the future according to the historical data of the power grid load and the external environment factors by using a joint model of a long-short-term memory neural network and a deep kernel extreme learning machine; s3, inputting the historical environment state and the predicted total load of a plurality of time steps of the future park into DDPG, wherein the DDPG takes minimization of the peak-valley difference of the power grid load, maximization of the economic benefit of the load aggregator and maximization of the demand satisfaction degree of the electric vehicle user as a reward function; and S4, the operation state of the park is monitored in real time, real-time environment state input of the DDPG is constructed, and the DDPG outputs an electric vehicle charging and discharging power instruction based on the dynamic state. The charging and discharging behaviors of the electric vehicle can be regulated and controlled in real time in a complex dynamic environment so as to stabilize a power grid.

Description

Technical Field

[0001] This invention relates to electric vehicles, and more particularly to a method, system, and storage medium for controlling the charging and discharging of electric vehicles in a campus environment based on reinforcement learning. Background Technology

[0002] With the rapid popularization of electric vehicles, vehicle-to-grid (V2G) technology is increasingly becoming an important solution for balancing the contradiction between grid load fluctuations and electric vehicle charging demand. V2G treats electric vehicles as flexible mobile energy storage units, forming a synergistic effect with the grid through two-way energy and information interaction. By dynamically adjusting charging and discharging behavior, V2G technology can charge electric vehicles during grid off-peak hours and feed energy back to the grid during peak hours, thereby achieving the goal of peak shaving and valley filling to stabilize the grid. Furthermore, with the continuous development of new power systems, electric vehicle charging and discharging equipment is gradually evolving towards informatization and intelligence, laying a solid foundation for the practical application of V2G technology.

[0003] However, most existing optimization strategies for electric vehicle charging and discharging under vehicle-grid interaction rely on traditional optimization algorithms, such as linear programming, dynamic programming, and heuristic algorithms. These methods typically require the construction of accurate mathematical models, and when faced with environmental changes, they need to be re-solved based on static optimization objectives, a time-consuming and labor-intensive process. In complex and ever-changing power system environments, these traditional algorithms often exhibit insufficient adaptability when facing external disturbances or dynamic changes in grid and electric vehicle loads, making it difficult to meet the needs of real-time optimization. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a reinforcement learning-based method, system, and storage medium for controlling the charging and discharging behavior of electric vehicles in real time under complex dynamic environments to stabilize the power grid.

[0005] Technical solution: The reinforcement learning-based electric vehicle charging and discharging control method for campuses described in this invention includes the following steps:

[0006] S1. Obtain historical data on grid load, electric vehicle user behavior, and external environmental factors, as well as the corresponding historical SOC status and time-of-use electricity price data of electric vehicles, and construct historical environmental status.

[0007] S2. A joint model using a long short-term memory neural network and a deep kernel extreme learning machine is used to predict the total load of the park at multiple time steps in the future based on historical data of power grid load and external environmental factors.

[0008] S3. Input the total load of the historical environmental status and the predicted future park at multiple time steps into DDPG. DDPG uses the minimization of the peak-valley difference of the power grid load, the maximization of the economic benefits of the load aggregator and the maximization of the satisfaction of electric vehicle users' needs as the reward function.

[0009] S4. Monitor the park's operating status in real time, construct the real-time environmental status input of DDPG, and DDPG dynamically outputs electric vehicle charging and discharging power commands based on this.

[0010] Based on the above technical solution, the acquisition of the initial DDPG strategy relies on the accurate prediction of future park load trends. To improve the rationality and convergence efficiency of the initial strategy, step S2 introduces an LSTM-DKELM prediction model jointly composed of a Long Short-Term Memory Neural Network (LSTM) and a Deep Kernel Extreme Learning Machine (DKELM). This model first extracts the time-dependent features from historical load sequences and external influencing factors (such as time, weather, user behavior, etc.) through LSTM, and then inputs its output time-series features into DKELM for high-dimensional nonlinear mapping and regression prediction, thereby obtaining the predicted total park load values ​​for multiple future time steps. The prediction results not only serve as the state input background for the reinforcement learning environment to construct future load change trends, but also help limit the output range of the initialization strategy network (Actor), enabling it to generate a relatively reasonable charging and discharging power strategy in the early stages of training. Because DDPG training requires adjusting the strategy based on actual external feedback, without the guidance of LSTM-DKELM prediction results, the initial strategy generated by DDPG would be too random and could disrupt grid stability. Therefore, by using the predicted load as the basis for the initial exploration strategy, the strategy bias caused by random initialization can be effectively avoided, thereby improving the stability and overall convergence speed of the deep reinforcement learning model in the early training stage.

[0011] Step S3 of DDPG uses parameters such as real-time grid load status, electric vehicle battery state of charge (SoC), user departure time, and minimum power demand as the state space. It constructs a reward function based on load aggregator revenue, grid peak-to-valley difference, and user satisfaction, outputting continuous actions, namely the charging and discharging power of electric vehicles at different times. During training, actions are generated through the policy network and evaluated by the value network. The policy network is iteratively optimized using an experience replay mechanism and a soft update mechanism to achieve adaptive scheduling under different load scenarios. During the execution of the charging and discharging strategy, the DDPG algorithm can update its model parameters in real time, gradually optimizing the performance of the policy network and value network based on changes in grid load and feedback from electric vehicle charging behavior. Through this adaptive adjustment mechanism, the model can continuously adapt to environmental changes, ensuring that the scheduling of grid load and electric vehicle charging and discharging is always in an optimal state. These features enable this method to not only regulate electric vehicle charging and discharging in real time in complex dynamic environments to respond to grid conditions and achieve grid stability, but also to guarantee user travel needs, ultimately achieving a comprehensive scheduling effect that balances load curve smoothing, revenue maximization, and user satisfaction.

[0012] Preferably, in step S3, during the DDPG training process, samples to be added to the experience replay pool are randomly discarded based on their similarity to the state space in the experience replay pool; specifically, this includes the following process:

[0013] Calculate state space similarity

[0014]

[0015] in, Let x be the state space of the i-th sample to be added to the experience recycling pool. ki Let's define the state space of the k-th sample in the pre-emptive pool before the i-th sample is added to the empirical recycling pool. For state space similarity; for With x ki The Euclidean distance between them, M i The total number of samples in the pool before the i-th sample is placed into the empirical recycling pool;

[0016] Calculate sample discard parameters

[0017]

[0018] Where λ is a hyperparameter. The average similarity in the state space of the i-th sample to be included.

[0019] The calculation formula is

[0020]

[0021] Generate a random number r in the range [0,1]. If r < p, discard the current sample; otherwise, retain it and add it to the experience pool.

[0022] The above strategy of randomly discarding samples is more flexible and adaptable than traditional direct deduplication or fixed threshold filtering methods. It not only avoids repeated learning of highly similar samples, but also preserves the diversity contribution of potential boundary samples, thereby effectively improving the generalization ability and overall convergence speed of the policy network.

[0023] Preferably, the following constraints are included in the DDPG training process.

[0024]

[0025] Among them, SOC i (t dep ) represents the time t that the i-th electric vehicle departs at, as scheduled by the user. dep The state of charge, E i,min This represents the minimum amount of electricity required for a trip set by the i-th electric vehicle user.

[0026] Introducing the above constraints ensures that the control strategy must meet the user's travel needs and avoids the situation where the charging volume of electric vehicles does not meet the standard in order to stabilize the power grid, thus affecting the normal use of the vehicle.

[0027] The reinforcement learning-based electric vehicle charging and discharging control system for industrial parks described in this invention includes:

[0028] Data acquisition module: used to acquire historical data on grid load, electric vehicle user behavior and external environmental factors, as well as the corresponding historical SOC status and time-of-use electricity price data of electric vehicles, and to construct historical environmental status.

[0029] Park load forecasting module: It is used to predict the total load of the park at multiple time steps in the future based on historical data of power grid load and external environmental factors through a joint model of long short-term memory neural network and deep kernel extreme learning machine;

[0030] Model training module: used to input the total load of the historical environmental state and the predicted future park at multiple time steps into DDPG. DDPG uses the minimization of the peak-valley difference of the power grid load, the maximization of the economic benefits of the load aggregator, and the maximization of the satisfaction of the needs of electric vehicle users as the reward function.

[0031] Charge and discharge control module: used to monitor the park's operating status in real time, construct the real-time environmental status input of DDPG, and DDPG dynamically outputs electric vehicle charging and discharging power commands based on this.

[0032] The computer-readable storage medium for storing one or more programs according to the present invention includes one or more programs comprising instructions that, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0033] Beneficial effects: Using the prediction results of the joint model of long short-term memory neural network and deep kernel extreme learning machine as the basis for the initial exploration strategy of DDPG can effectively avoid the strategy bias caused by random initialization and ensure the stability and reliability of the control strategy. Using DDPG to generate electric vehicle charging and discharging strategies and its set reward function can not only regulate the charging and discharging of electric vehicles in real time in complex dynamic environments to respond to the grid status and achieve grid stability, but also ensure users' travel needs, ultimately achieving a comprehensive scheduling effect that balances load curve smoothing, revenue maximization and user satisfaction. Attached Figure Description

[0034] Figure 1 This is a flowchart of the electric vehicle charging and discharging scheduling process in the park based on DDPG according to the present invention;

[0035] Figure 2 This is a diagram of the campus electric vehicle charging and discharging scheduling algorithm based on DDPG according to the present invention.

[0036] Figure 3 This is a campus load prediction method based on a combination of long short-term memory neural networks and deep kernel extreme learning machines. Detailed Implementation

[0037] As shown in the figure, the reinforcement learning-based electric vehicle charging and discharging control method for parks described in this invention uses the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning as its core to optimize the charging and discharging behavior of electric vehicles in the park, thereby reducing the load peak-valley difference in the park and maximizing the revenue of load aggregators, while also taking into account the needs of electric vehicle users.

[0038] The specific embodiments of the present invention can be applied to the dispatching of power grids in industrial parks. The method can be executed by a power grid dispatching device in industrial parks. The dispatching device can be implemented in hardware and software and embedded in the power grid control system.

[0039] The reinforcement learning-based electric vehicle charging and discharging control method for industrial parks described in this invention includes the following steps:

[0040] S1. Collect historical data on the park's power load, electric vehicle charging and discharging, and external influencing factors, as well as historical SOC status and time-of-use pricing data for electric vehicles at corresponding historical moments. This step uses data acquisition devices to monitor factors such as power load, traffic flow, and weather changes within the park in real time, ensuring that the acquired data includes historical information from multiple time periods to provide a more comprehensive understanding of the patterns in electric vehicle charging and discharging behavior. For example, weather data can predict trends in electric vehicle charging demand, while traffic flow data reflects the frequency of electric vehicle trips and dwell time. Based on all the above data, an environmental model is constructed to simulate the operating status and changing patterns of the park's power system, providing data support for subsequent scheduling decisions.

[0041] S2. A joint model using a long short-term memory neural network and a deep kernel extreme learning machine is used to predict the total load of the park at multiple time steps in the future based on historical data of power grid load and external environmental factors.

[0042] Figure 3 This paper presents a campus load forecasting method based on a combination of Long Short Memory Neural Networks and Deep Kernel Extreme Learning Machines. The forecasting module aims to anticipate grid load changes over multiple future time steps, providing environmental context information for the Deep Reinforcement Learning Model (DDPG) and assisting in the construction of initial strategies, thereby improving the learning efficiency and convergence stability of the DDPG algorithm in the early stages of training.

[0043] Specifically, LSTM is used to capture the time-dependent characteristics of historical data sequences in the park and extract long-term fluctuation patterns. DKELM then performs nonlinear mapping and regression, outputting the predicted total load for multiple future time steps. Although load forecasting itself does not directly output charging / discharging strategies, by inputting the predicted load curve as part of the state space into the DDPG policy network (Actor), and combining it with information such as current electric vehicle battery charge, user departure time, and electricity purchase price, the policy network can generate preliminary charging / discharging power commands. To avoid large-scale exploration and convergence instability caused by random initialization strategies in the early stages of training, the system can limit the output range of the policy network based on load forecast results and user demand. For example, it can encourage charging during off-peak periods and suppress charging or even moderate discharging during peak periods, thus forming an initial "guiding strategy" based on electricity supply and demand trends. This initial strategy is not the final decision result, but serves as an initial action reference for the reinforcement learning model, helping to reduce the policy search space, accelerate model training efficiency, and improve the practicality and robustness of the final strategy.

[0044] Long Short-Term Memory (LSTM) networks are a type of recurrent neural network (RNN) that uses "memory gates" and "forget gates" to handle long-term and short-term dependencies in time series data. Given input x... tThe hidden state h from the previous moment t-1 Memory unit state C t-1 The formula for calculating LSTM is:

[0045] The forgetting gate determines the memory of the previous moment. t-1 How much of it should be retained:

[0046] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0047] Among them, [h t-1 ,x t The symbol ] indicates that the two are concatenated and used as input. W f and b f Here, σ(·) represents the weight matrix and bias of the forget gate, respectively, and σ(·) is the sigmoid function, whose output value is between 0 and 1, determining the information retention ratio.

[0048] The input gate determines the current input x. t What new information needs to be written into the memory unit:

[0049] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0050]

[0051] Among them, i t The input gate weights (sigmoid output); To prepare candidate memories (tanh output) to be written to the current cell, W i and W C These are the corresponding weight matrices, b i and b C This corresponds to the bias.

[0052] Updating a memory unit involves updating the current memory state, retaining old memories proportionally, and adding new information.

[0053]

[0054] The output gate determines the output at the current time step:

[0055] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0056] h t =o t ·tanh(C t )

[0057] σ is the sigmoid activation function, tanh is the hyperbolic tangent function, and h t This is the final hidden state (output). In this paper, this structure is used as a temporal feature extractor in actual park load prediction, and combined with the subsequent Deep Kernel Extreme Learning Machine (DKELM) to build a complete prediction model.

[0058] Kernel methods possess powerful nonlinear mapping capabilities, transforming complex problems that exhibit nonlinearity in low-dimensional space into linear problems in high-dimensional space, thus enabling linearly separable data from those that were originally linearly inseparable to become linearly separable in high-dimensional space. By constructing appropriate kernel functions, not only can the distribution of data be altered, but the regression accuracy of the model can also be significantly improved. Building upon Multilayer Extreme Learning Machine (ML-ELM) and Kernel Extreme Learning Machine (KELM), this paper proposes a Deep Kernel Extreme Learning Machine (DK-ELM) model with a stacked multi-hidden-layer structure. ML-ELM is a multilayer feature extraction network composed of multiple stacked Extreme Learning Machine Autoencoder (ELM-AE) structures. ELM-AEs randomly initialize input weights and biases, train output weights using analytical solutions, and achieve nonlinear transformation and reconstruction of the original data at each layer. The multilayered ML-ELM extracts more abstract, deeper features layer by layer, ultimately culminating in the output feature x from the k-th layer. k This serves as the output of the entire feature extraction module.

[0059] ML-ELM uses ELM-AE to train the parameters in each layer. The output of the last hidden layer in ML-ELM is used as the input to KELM. Then, a kernel function K(x,y) is constructed to replace the last hidden layer. The algorithm combining multi-layer extreme learning machine and kernel extreme learning machine is called Deep Kernel Extreme Learning Machine (DK-ELM). Its network structure is as follows: Figure 3 As shown, the original input data is transformed through k hidden layers of a multi-layer extreme learning machine to obtain the output features x of the ML-ELM. k It also serves as input to KELM, where the kernel function applies the kernel to the feature x. k Mapping is performed. In KELM, the output matrix of the hidden layer is replaced by a kernel function. The kernel function K(x,y) can be used to map the HH layer. T It is expressed as follows:

[0060] HH T (i,j)=K(x i ,x j )

[0061]

[0062] The kernel function used is the commonly used Gaussian kernel function K(x,x) i ) = exp(||xx i || 2 / γ), where γ is the kernel parameter.

[0063] h(x)H T This represents the kernel mapping that the model performs on the input samples during the prediction phase, used to calculate the predicted output. It can be represented as follows:

[0064]

[0065] The network output of the Deep Kernel Extreme Learning Machine (DK-ELM) is as follows:

[0066]

[0067] The specific steps for campus load prediction combining Long Short-Term Memory Networks and Deep Kernel Extreme Learning Machines, and for generating initialization strategies, are as follows:

[0068] 1) Data collection and preprocessing: Collect historical load data (hourly or minute-level) and standardize or normalize the data.

[0069]

[0070] Sliding window generation of training samples: Input: Load and related features X from the past n hours t =[P scalcd (tn),…,P scalcd (t)]. The output target is the normalized load value Y for the next T hours. t =[P scaled (t+1),…,P scaled [(t+T)], where Y t This serves as the supervised learning objective for the load forecasting model, used to train the network to output future short-term load trends.

[0071] 2) Training and test set division: Divide the data into training and test sets according to the time series, with a common ratio of 4:1.

[0072] 3) Generating training samples X using a sliding window t The input is fed into an LSTM-based time series feature extraction model to extract long-term dependency features of the time series, and the output is H. t As input to a deep kernel extreme learning machine.

[0073] 4) Nonlinear prediction based on DKELM, using features H extracted by LSTM. t With the target load sequence Y t=(P scaled (t+1),…,P scaled [(t+T)] is input into the training DKELM, and the prediction results are denormalized. The purpose of denormalization is to make the prediction results directly usable for scheduling decisions and subsequent reinforcement learning modeling, ensuring that the model output has engineering interpretability and practicality.

[0074] 5) Based on the predicted load curves for the next T time steps and the predicted total load level (i.e., statistics such as the integral or mean of the future load), these are input into the deep reinforcement learning model as the initial input and reference boundary of the environmental state, used to generate the initial action policy. This policy will then be dynamically adjusted according to the model feedback, with its action space upper and lower limits fluctuating around the initial predicted value, thereby achieving policy initialization and early guidance.

[0075] S3. Construct a deep reinforcement learning model and input the total load of historical environmental states and predicted future campus conditions at multiple time steps into it:

[0076] Based on the collected data from the industrial park, a deep reinforcement learning model (DDPG) is established. This model contains two main optimization objectives: an outer layer and an inner layer. The outer layer objective is to minimize the peak-to-valley load difference in the power grid to achieve stable power system operation. The inner layer objective focuses on maximizing the economic benefits of load aggregators. In addition to these two objectives, the needs of electric vehicle users must also be considered.

[0077] Define the objective function for the peak-valley difference in power grid load:

[0078]

[0079] In the formula, P load (t) represents the base load power of the park during time period t, including electricity consumption by users in offices and residential areas. P EV,i (t) represents the charging and discharging power of the i-th electric vehicle during time period t (positive value indicates charging, negative value indicates discharging), in kW, and N is the total number of electric vehicles participating in vehicle-to-grid interaction in the park.

[0080] The revenue generated by the park through load aggregator management includes revenue from residential base load and electric vehicle charging behavior.

[0081] Define the objective function for the economic returns of the load aggregator:

[0082]

[0083] Among them, f s f represents the total revenue generated from the overall electricity load of the park. c For electricity purchase costs, f EVD For the cost of electric vehicle battery depletion, fs f c and f EVD The calculation formula is as follows:

[0084]

[0085]

[0086]

[0087] Among them, S grid (t) represents the base electricity price for time period t, S EV (t) represents the time-of-use electricity price for electric vehicle charging stations during time period t, which is the electricity price paid by electric vehicle users to the park's load aggregator. This is the park's "electricity price" to users, expressed in yuan / kWh; C grid (t) represents the electricity purchase cost for time period t, which is the real-time electricity price paid by the park to the upper-level power grid. This is the park's "electricity purchase price," expressed in yuan / kWh; C battery,i Let be the battery price of the i-th electric vehicle, cyc be the total energy throughput of the battery under maximum cycle life, Δt be the time interval between adjacent time periods, T be the total time interval for optimization, which is usually segmented according to time step (e.g., 15 minutes or 1 hour), and t be the current time step.

[0088] Overall community benefits f s This reflects the electricity sales revenue that aggregators obtain through basic electricity consumption and electric vehicle charging activities; while the grid purchase cost f c This reflects the expenditures incurred by the industrial park to meet overall electricity demand. This is because load aggregators purchase electricity at low prices (C...). grid (t)) High-priced electricity sales (S grid (t) and S EV Since (t)) achieves profitability, the two are calculated separately in the model and do not overlap, forming a complete revenue-cost analysis system, which helps to truly reflect the economic benefits under the scheduling strategy.

[0089] It should be noted that the battery degradation cost f EVD Essentially, this cost should be borne by electric vehicle users. However, in this model, the park load aggregator, as the coordinator of overall scheduling and incentives, needs to consider the perceived cost and acceptance level of users when designing scheduling strategies and incentive mechanisms. Therefore, f EVD Including it in the revenue objective function is to reasonably measure the real cost of user response in scheduling decisions, thereby improving the feasibility and implementability of the model.

[0090] Electric vehicle user satisfaction is modeled by the relationship between the user's actual battery level at the start of the trip and their minimum battery requirement for the trip.

[0091] Define the objective function for satisfying the needs of electric vehicle users as follows:

[0092]

[0093] in, The formula for calculating the degree to which the charging needs of the i-th electric vehicle user are met is as follows:

[0094]

[0095] Among them, SOC i (t dep ) represents the time t is the user's scheduled departure time for the i-th electric vehicle. dep The state of charge (E) i,min This represents the minimum electricity consumption required for a user's trip, and α is a satisfaction adjustment coefficient used to measure the additional satisfaction derived from electricity consumption exceeding demand. This user satisfaction reward item is weighted in the total reward and combined with other indicators to ensure that while optimizing grid load and aggregator revenue, normal user travel is not sacrificed, thereby improving user acceptance and practical application value of the dispatch strategy.

[0096] Based on the reward function, theoretically, the DDPG iteration process should strive to meet users' charging needs as much as possible to align with the electric vehicle user demand satisfaction objective function. However, to be on the safe side, the following constraints can be added during training:

[0097]

[0098] The above-mentioned conditions must be met to ensure that the electric vehicle's charge level is not less than the minimum charge required for the trip when the user leaves, so that the user can travel normally.

[0099] Based on the function defined above, the formula for calculating the reward function is as follows:

[0100] r t =λ1F pv +λ2F ec +λ3F sat

[0101] Where, r t For the reward function, F pv Let F be the objective function for the peak-valley difference in power grid load. ec Let F be the objective function for the economic returns of the load aggregator. re Let λ1, λ2, and λ3 be the objective function for satisfying the needs of electric vehicle users, and let λ1, λ2, and λ3 be the weight parameters.

[0102] The DDPG model consists of a policy network (Actor) and a value network (Critic). The policy network is responsible for generating the charging and discharging strategies for electric vehicles, while the value network evaluates the effectiveness of the strategies. Through iterative optimization, the model can effectively adapt to different load scenarios and make optimal decisions.

[0103] Among them, such as Figure 2 The environmental model shown is constructed based on the "state-action-reward-transition" mechanism in reinforcement learning, specifically consisting of the following parts: State information includes the current grid load state, the remaining power (SOC) of each electric vehicle, the user's preset departure time, and the minimum power requirement for travel; External factors include weather conditions, time information (such as whether it is peak period), historical load data, and traffic flow information; In addition, future load change trends predicted by the LSTM-DKELM model are introduced to enhance the environment's ability to perceive future dynamics. This information collectively constitutes the state space input (i.e., historical environmental states), which is input to the policy network (ActorNetwork) to generate the optimal charging and discharging strategy for the current moment. The state of the environment is determined based on the current state s. t Output discharge point parameter power a t In essence, this is a control variable in a continuous action space, representing the specific charging / discharging power value (in kW) that each electric vehicle should perform at the current time step. Positive values ​​represent charging, and negative values ​​represent discharging. This action affects the electric vehicles and the power distribution system, causing changes in the environmental state. The system then observes the new state information and reward signals, which are evaluated by the Critic network and fed back to the Actor network to update the policy. Through continuous iterative learning, this environmental model achieves accurate modeling and real-time decision support for the multi-variable, multi-objective scheduling process in the park, thereby optimizing and controlling the charging and discharging behavior of electric vehicles.

[0104] The environmental change becomes the state s in the next moment. t+1 :

[0105] s t+1 =E(s) t ,a t ,r t )

[0106] Where E represents the environment state transition function, r t The reward signal (i.e., the reward function) is used. The policy network generates a charging and discharging policy, which is then output to the electric vehicle charging and discharging system. By inputting state information, the policy network learns and generates continuous actions (such as charging and discharging power). The target value is calculated using the following formula:

[0107] y i =R i +γQ'(si+1 ,μ'(s i+1 ))

[0108] Among them, s i+1 The state transition process, which considers factors such as grid load, electric vehicle battery status, electricity price changes, and user demand changes, is designed to represent the state changes in the next moment of environmental change. This ensures that the model has strong responsiveness and generalization ability in multi-factor dynamic environments. R i Let μ′ be the current reward, Q′ be the target policy network, and Q′ be the target value network. The purpose of this network is to provide an optimal charging and discharging strategy to maximize grid balance and economic benefits. Simultaneously, the value network (CriticNetwork) evaluates the effectiveness of the current strategy, and its output is the value or reward of the current strategy. It combines the current state with the output of the policy network to calculate the action value function. It helps the Actor network adjust the strategy by evaluating the strategy's value (such as load stability, revenue, etc.) to optimize the charging strategy. The network parameters gradually approach the current network parameters through a soft update mechanism.

[0109] θ μ′ ←τθ μ +(1-τ)θ μ′ (17)

[0110] θ Q′ ←τθ Q +(1-τ)θ Q′ (18)

[0111] Where τ∈(0,1) is the update rate.

[0112] In the DDPG algorithm, maintaining the diversity of samples in the experience replay pool is beneficial for network convergence. Therefore, during network training, an algorithm based on sample similarity is proposed to randomly discard samples, thus improving the original algorithm. The details are as follows:

[0113] set up Let x be the state space of the i-th sample to be added to the experience recycling pool. ki Let the state space of the i-th sample be the state space of the k-th sample in the pool before it is added to the empirical pool. Then the similarity between the state space of the newly added sample and the samples in the pool is... Calculate as follows:

[0114]

[0115] in, for With x k The Euclidean distance between i, M iLet be the total number of samples in the pre-recovery pool before the i-th sample is placed into the experience pool, and max(·) be the maximum value of the Euclidean distance between the i-th sample and all samples in the pre-recovery pool before the i-th sample is placed into the experience pool.

[0116] Calculate sample discard parameters

[0117]

[0118] Where λ is a hyperparameter. The average similarity in the state space of the i-th sample to be included.

[0119] The calculation formula is

[0120]

[0121] Generate a random number r in the range [0,1]. If r < p, discard the current sample; otherwise, retain it and add it to the experience pool.

[0122] As can be seen from the above process, the higher the similarity between the sample to be added and the samples in the experience replay pool, the higher the probability of discarding the sample. Sample deduplication is used to reduce sample correlation and improve training stability. Then, the experience replay pool is used to improve sample utilization and enhance generalization ability. This improves the target measurement network's long-term evaluation ability of the target.

[0123] S4. Based on real-time environmental conditions (including real-time grid load, electric vehicle user behavior and external environmental factors data, as well as the corresponding real-time SOC status of electric vehicles and time-of-use electricity price data), predict charging and discharging demand and generate specific electric vehicle charging and discharging strategies: After the model completes its initial training, it can be deployed and put into use within the park.

[0124] Dynamic adjustment of charging and discharging during peak grid load: When the grid load reaches its peak, the DDPG model automatically generates a dynamic charging adjustment strategy to adjust the charging and discharging behavior of electric vehicles to reduce peak load.

[0125] For example, during peak hours, the model reduces the charging power of electric vehicles (EVs) or instructs EVs to feed their stored battery energy back to the grid, thereby alleviating the pressure from high loads. This step not only smooths the grid load curve and reduces the peak-to-valley difference but also brings additional revenue to load aggregators. Through this dynamic adjustment mechanism, the model achieves efficient scheduling of EV groups under peak load conditions.

[0126] Charging and discharging optimization strategy during off-peak hours: Under off-peak load conditions, the DDPG model generates a discharging optimization strategy, instructing electric vehicles to increase charging power during off-peak periods to maximize the park's power utilization rate. The model can rationally utilize off-peak load periods, helping electric vehicles complete charging when electricity prices are lower, thereby reducing charging costs. During this process, the model continuously adjusts the charging power based on load conditions to ensure that it does not affect users' normal electric vehicle usage needs, achieving a dual optimization of power efficiency and economic benefits.

[0127] Among its features, the DDPG algorithm possesses the ability to generate emergency control strategies in the event of sudden changes in grid load forecasts or emergency situations. This capability does not rely solely on the addition of specific abnormal samples during the training phase, but rather on the learning of sensitivity to state changes during reinforcement learning and the continuous action output characteristics of the policy network. Based on the magnitude and speed of load changes, the model can rapidly adjust the charging and discharging behavior of electric vehicles to cope with sudden load demands. For example, if the load suddenly increases, the model will quickly reduce the charging demand of electric vehicles or release the stored energy in the vehicles to ensure a smooth transition of the grid and prevent grid instability caused by load fluctuations. Simultaneously, the model also ensures that the travel needs of electric vehicle users are not affected, balancing grid stability and user experience.

[0128] During the execution of the charging and discharging strategy, the DDPG algorithm can update its model parameters in real time, gradually optimizing the performance of the policy network and value network based on changes in grid load and feedback from electric vehicle charging behavior. Through this adaptive adjustment mechanism, the model can continuously adapt to environmental changes, ensuring that the scheduling of grid load and electric vehicle charging and discharging is always in an optimal state.

[0129] Through the above technical solutions, this invention achieves significant improvements in electric vehicle charging and discharging scheduling. It not only optimizes scheduling accuracy and response efficiency through deep reinforcement learning, but also reduces the peak-valley load difference in the power grid while meeting basic user needs, increasing the revenue of load aggregators and improving grid stability. Furthermore, the model possesses continuous adaptive learning capabilities, allowing for flexible expansion and optimization based on the needs of different park scenarios, thus demonstrating broad practical application prospects.

Claims

1. A method for controlling the charging and discharging of electric vehicles in a campus based on reinforcement learning, characterized in that, Includes the following steps: S1. Obtain historical data on grid load, electric vehicle user behavior, and external environmental factors, as well as the corresponding historical SOC status and time-of-use electricity price data of electric vehicles, and construct historical environmental status. S2. A joint model using a long short-term memory neural network and a deep kernel extreme learning machine is used to predict the total load of the park at multiple time steps in the future based on historical data of power grid load and external environmental factors. S3. Input the total load of the historical environmental status and the predicted future park at multiple time steps into DDPG. DDPG uses the minimization of the peak-valley difference of the power grid load, the maximization of the economic benefits of the load aggregator and the maximization of the satisfaction of electric vehicle users' needs as the reward function. S4. Monitor the park's operating status in real time, construct the real-time environmental status input of DDPG, and DDPG dynamically outputs electric vehicle charging and discharging power commands based on this.

2. The method according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21. Use a long short-term memory neural network to extract long-term dependency features of historical power grid load data time series; S22. Input the extracted long-term dependency features into a deep kernel extreme learning machine for nonlinear mapping to obtain the prediction results; S23. The prediction results of the deep kernel extreme learning machine are denormalized to obtain the total load of the future campus at multiple time steps.

3. The method according to claim 1, characterized in that: In step S3, during the DDPG training process, samples to be added to the experience replay pool are randomly discarded based on their similarity to the state space in the experience replay pool; specifically, this includes the following process: Calculate state space similarity in, Let x be the state space of the i-th sample to be added to the experience recycling pool. ki Let's define the state space of the k-th sample in the pre-emptive pool before the i-th sample is added to the empirical recycling pool. For state space similarity; for With x ki The Euclidean distance between them, M i The total number of samples in the pool before the i-th sample is placed into the empirical recycling pool; Calculate sample discard parameters Where λ is a hyperparameter. The average similarity in the state space of the i-th sample to be included. The calculation formula is Generate a random number r in the range [0,1]. If r < p, discard the current sample; otherwise, retain it and add it to the experience pool.

4. The method according to claim 1, characterized in that: The formula for calculating the reward function in step S3 is as follows: r t =λ1F pv +λ2F ec +λ3F sat Where, r t For the reward function, F pv Let F be the objective function for the peak-valley difference in power grid load. ec Let F be the objective function for the economic returns of the load aggregator. re Let λ1, λ2, and λ3 be the objective function for satisfying the needs of electric vehicle users, and let λ1, λ2, and λ3 be the weight parameters. F pv The calculation formula is: Among them, P load (t) represents the base load power of the park during time period t, P EV,i (t) represents the charging and discharging power of the i-th electric vehicle during time period t, and N represents the total number of electric vehicles participating in vehicle-to-grid interaction in the park; F ec The calculation formula is: Among them, f s f represents the total revenue generated from the overall electricity load of the park. c For electricity purchase costs, f EVD Let T be the cost of battery degradation in an electric vehicle, and T be the total time for optimization. F sat The calculation formula is: in, The formula for calculating the degree to which the charging needs of the i-th electric vehicle user are met is as follows: Among them, SOC i (t dep ) represents the time t that the i-th electric vehicle departs at, as scheduled by the user. dep The state of charge, E i,min Let α represent the minimum electricity required for the i-th electric vehicle user's trip, and let α be the satisfaction adjustment coefficient.

5. The method according to claim 4, characterized in that: The f s The calculation formula is: Among them, S grid (t) represents the base electricity price for time period t, S EV (t) represents the time-of-use electricity price for electric vehicle charging stations during time period t, P EV (t) represents the total charging and discharging power of all electric vehicles at time t; f c The calculation formula is: Among them, C grid (t) represents the electricity purchase cost during time period t; f EVD The calculation formula is: Among them, C battery,i Let be the battery price of the i-th electric vehicle, cyc be the total energy throughput of the battery at its maximum cycle life, and Δt be the time interval between adjacent time periods.

6. The method according to claim 1, characterized in that: The following constraints are added during the DDPG training process. Among them, SOC i (t dep ) represents the time t that the i-th electric vehicle departs at, as scheduled by the user. dep The state of charge, E i,min This represents the minimum amount of electricity required for a trip set by the i-th electric vehicle user.

7. The method according to claim 1, characterized in that: The DDPG adopts a soft update strategy.

8. The method according to claim 1, characterized in that: The external environmental factors in step S1 include weather, traffic, and holidays.

9. A reinforcement learning-based electric vehicle charging and discharging control system for a campus, characterized in that, include: Data acquisition module: used to acquire historical data on grid load, electric vehicle user behavior and external environmental factors, as well as the corresponding historical SOC status and time-of-use electricity price data of electric vehicles, and to construct historical environmental status. Park load forecasting module: It is used to predict the total load of the park at multiple time steps in the future based on historical data of power grid load and external environmental factors through a joint model of long short-term memory neural network and deep kernel extreme learning machine; Model training module: used to input the total load of the historical environmental state and the predicted future park at multiple time steps into DDPG. DDPG uses the minimization of the peak-valley difference of the power grid load, the maximization of the economic benefits of the load aggregator, and the maximization of the satisfaction of the needs of electric vehicle users as the reward function. Charge and discharge control module: used to monitor the park's operating status in real time, construct the real-time environmental status input of DDPG, and DDPG dynamically outputs electric vehicle charging and discharging power commands based on this.

10. A computer-readable storage medium for storing one or more programs, characterized in that: The program includes one or more instructions that, when executed by a computing device, cause the computing device to perform any of the methods according to claims 1 to 8.

Images