Electric vehicle charging station local energy storage cooperative scheduling method based on deep reinforcement learning

Abstract

The invention discloses an electric vehicle charging station local energy storage cooperative scheduling method based on deep reinforcement learning, and relates to the technical field of charging station operation and energy management. According to the method, a charging station power model is constructed, a future electric vehicle arrival rate is obtained as state input, and a scheduling strategy is guided to carry out prospective power and price planning. Through constructing a scheduling controller based on deep reinforcement learning, combined optimization control of charging power, energy storage power and an electricity price strategy is realized, and a reward function containing multiple indexes is designed to give consideration to economic benefits and operation stability. And a priority experience sampling mechanism combining time correlation and profit contribution is further provided and is used for improving strategy training efficiency and scheduling performance. According to the invention, the resource scheduling capability of the charging station under the conditions of load fluctuation and dynamic change of electricity price is effectively improved, the balance between income maximization and safe operation of the system is realized, and the method has good expandability and application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of electric vehicle charging infrastructure management and intelligent scheduling, and in particular to a method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning. Background Art

[0002] With the rapid development of new energy vehicles and the widespread deployment of charging infrastructure, electric vehicle charging stations are increasingly constraining the power system. This high-frequency, centralized charging not only significantly impacts the power grid but also places higher demands on charging station operators for resource coordination and revenue management. Given the complexities of dynamic electricity pricing, uncertain electric vehicle behavior, and limited resource allocation, effectively dispatching local energy storage devices and rationally setting charging prices to manage peak loads, reduce operating costs, and improve service efficiency have become key challenges that electric vehicle charging dispatch systems urgently need to address.

[0003] Traditional charging station scheduling methods mostly rely on static rules, linear optimization, or prediction-based heuristic algorithms, which are difficult to handle the time-varying nature and multi-source uncertainty of charging loads. Especially when faced with non-stationary scenarios such as instantaneous large-scale arrival of vehicles, drastic fluctuations in electricity prices, or tight energy storage resources, the scheduling effect is poor, which can easily lead to problems such as power limit violations, energy storage abuse, and reduced profits.

[0004] In recent years, the development of deep reinforcement learning (RL) technology, with its high-dimensional perception and adaptive decision-making capabilities in complex dynamic systems, has gradually demonstrated its potential for application in charging scheduling. However, existing approaches still face numerous challenges in applying RL to energy storage and pricing control. For example, experience replay mechanisms fail to fully reflect the temporal and economic characteristics of tasks, reward design fails to effectively balance operational safety and profit optimization, and control strategies fail to respond promptly to future changes in traffic flow.

[0005] Therefore, it is urgent to propose an intelligent scheduling method that is oriented to future load changes, has strong autonomous learning capabilities, and can achieve real-time response and joint optimization, so as to give full play to the regulation capacity of energy storage resources and enhance the operational resilience and economy of charging stations. Summary of the Invention

[0006] To address the aforementioned issues in the existing technology, the present invention proposes a method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning. This method is applicable to charging stations equipped with both local energy storage systems and dynamic electricity pricing mechanisms. By training a scheduling controller based on deep reinforcement learning, this method achieves a joint optimization of charging power, energy storage power, and charging prices. This method maximizes overall operating profits while meeting grid power constraints, alleviates peak power supply pressures, and improves user service quality.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning includes the following steps:

[0009] Establish a charging station power model, where the charging station includes multiple charging piles and a local energy storage system with bidirectional power regulation for charging and discharging. The energy storage system power is subject to the maximum access limit of the grid, and the energy storage charge state is updated during each scheduling cycle and maintained within the preset upper and lower limits.

[0010] Obtain a predicted arrival rate for electric vehicles over a period of time. The arrival rate represents the number of electric vehicles expected to arrive at a charging station per unit time and serves as one of the key input parameters for the charging station status, guiding the dispatch controller to pre-allocate power resources and pricing strategies. Build a dispatch controller based on deep reinforcement learning, taking the energy storage state of charge, the predicted arrival rate, the current electricity price, the charging pile occupancy rate, and the current pricing strategy as state inputs. The output includes a combined action on the power allocation of each charging pile, the energy storage system power adjustment value, and the charging price.

[0011] Design a reward function for multi-objective optimization. This reward function includes six indicators: charging profit, grid power over-limit penalty, energy storage price mismatch penalty, energy storage reasonable arbitrage reward, energy storage switching frequency penalty, and energy storage under-utilization penalty. This is used to balance economic objectives with operational stability objectives.

[0012] A priority experience sampling mechanism based on the dual indicators of time correlation and profit contribution is constructed. During the control strategy training process, sampling weights are dynamically assigned based on the economic benefits and time distance of each historical experience, so that the scheduling controller prioritizes learning high-value, highly relevant scheduling samples.

[0013] The trained control strategy is deployed at the actual charging station. The charging station status is sensed in real time during each scheduling cycle. The scheduling controller calls the control strategy network to generate control actions, and drives the charging pile power output, energy storage power regulation and charging price setting to achieve an integrated closed-loop scheduling operation of charging station power distribution, energy storage system power regulation and electricity price control.

[0014] The power regulation value of the energy storage system is an adjustable continuous value, and maximum limit values ​​are set in the charging direction and the discharging direction respectively to ensure the safe operation of the battery system.

[0015] The reward function is as follows:

[0016] R t =Ω t -ω1P exc-ω2P diff +ω3R diff -ω4P swi -ω5P util

[0017] Among them, R t is the reward value at the current moment t, Ω t is the profit of the charging station at the current time t, P exc is the penalty for exceeding the power limit of the power grid, P diff is the energy storage price difference penalty, R diff To provide reasonable arbitrage rewards for energy storage, P swi Penalty for frequent energy storage switching, P utili is the penalty for insufficient energy storage utilization, ω1 to ω5 are the weight coefficients of various penalties and rewards;

[0018] Charging profit Ω is adjusted according to the energy storage power value P t S The symbols are calculated piecewise:

[0019]

[0020] Among them, P t i represents the power of the i-th charging pile at time t, The user-side charging electricity price set for the site, p t is the basic electricity price on the grid side, p s is the electricity purchase cost of the energy storage system, Δt is the time interval, P t S is the power regulation value of the energy storage system, which is positive for discharge and negative for charge;

[0021] Grid power over-limit penalty P exc The expression is as follows:

[0022]

[0023] in, is the total charging power, P g is the maximum power limit for grid connection, is the relative limit-crossing ratio, λ dyn =λ base (1+γ·n exceed ) is the dynamic penalty coefficient, λ base is the basic penalty weight, γ is the growth coefficient, n exceed The number of consecutive exceeding limits;

[0024] The function f(δ) is defined as:

[0025]

[0026] Among them, δ1 and δ2 are two thresholds of relative over-limit ratio, which are used to adjust the penalty size;

[0027] In order to unify the modeling of energy storage price difference penalties and energy storage reasonable arbitrage incentives, an auxiliary variable is introduced:

[0028] V=|p s -p t |·|P t S |·Δt

[0029] The energy storage price mismatch penalty and reward items are:

[0030]

[0031] The symbolic function is defined as:

[0032]

[0033] Energy storage frequent switching penalty P swi The definition is as follows:

[0034]

[0035] in, is the energy storage power at the previous moment, is the energy storage rated power, n swi For continuous switching count, To activate the indicator function, when the condition C is met swi hour, otherwise The conditions are defined as:

[0036]

[0037] Among them, T P is the energy storage active power threshold, T change is the power change threshold;

[0038] Penalty for insufficient energy storage utilization P utili is defined as:

[0039]

[0040] The specific calculation method of the priority experience sampling mechanism based on the dual indicators of time correlation and profit contribution is as follows:

[0041] For each sample (s i ,a i ,r i ,s′ i ), its sampling priority is calculated as follows:

[0042]

[0043] Among them, Pr i represents the comprehensive sampling priority of the i-th transition, which is composed of the profit-based priority Pr i Priority based on temporal continuity By weight ω profit and ω temp Linear weighting is obtained, where ω profit +ω temp =1, κ is the time attenuation coefficient, d i is the time distance between the current time step and transition i; profit priority Pr i And by the transition profit Ω i Calculated, specifically:

[0044] If Ω i >0, then

[0045] Pr i =(Ω i +∈) α

[0046] otherwise

[0047]

[0048] Among them, ∈ is a very small positive number that prevents the sampling probability from being zero, and α∈[0,1] controls the degree of preference of sampling for high-profit transitions.

[0049] The dispatch controller supports a dynamic pricing mechanism. In each dispatch cycle, the charging price is determined based on the arrival rate of electric vehicles, the current electricity price and the load level of the charging station, achieving price elasticity and load guidance, thereby improving user regulation capabilities and operating benefits.

[0050] The deep reinforcement learning-based dispatch controller introduces future electric vehicle arrival rate prediction information as state input to dynamically adjust energy storage charging and discharging plans and pricing strategies to achieve early response to peak loads and forward-looking scheduling of power resources.

[0051] The present invention proposes a method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning, which has the following beneficial effects:

[0052] By constructing a unified scheduling framework that includes charging piles, energy storage systems, and a dynamic pricing mechanism, the method of the present invention can fully coordinate multiple types of resources while meeting grid access restrictions, thereby improving overall system operating efficiency and scheduling flexibility.

[0053] The method of the present invention introduces the prediction results of electric vehicle arrival rate as the state input of the dispatching controller of reinforcement learning, effectively improving the dispatching controller's response ability to non-stationary scenarios such as traffic peaks and sudden changes in user behavior, and realizing early prediction, reasonable arrangement of energy storage charging and discharging, and dynamic price adjustment.

[0054] By designing a reward function that includes five factors: charging profit, grid power over-limit penalty, energy storage price mismatch penalty, energy storage switching frequency penalty, and energy storage under-utilization penalty, the method of the present invention takes into account both profit maximization and system stability during the strategy training process, ensuring that the scheduling behavior is economical and engineering feasible.

[0055] During the strategy learning phase, the present invention adopts a sampling mechanism that combines historical experience gains and time distance. This is different from the traditional uniform or error-driven experience replay method, which enables the training process to focus more on the utilization of recent high-value samples, effectively improving the strategy convergence speed and generalization ability. It is particularly suitable for complex control tasks with strong timing dependencies and economic objectives.

[0056] The method of the present invention introduces charging prices into the scheduling action, allowing the scheduling controller based on deep reinforcement learning to dynamically set charging prices under different traffic densities and electricity price fluctuations, and combines the user behavior model to form price incentive feedback, effectively achieving the dual goals of load regulation and operating profit improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Structure diagram of the coordinated dispatch method for local energy storage at electric vehicle charging stations DETAILED DESCRIPTION

[0058] The present invention will be described in further detail below with reference to the accompanying drawings.

[0059] Step S1: Construct charging station power model

[0060] The power structure of an electric vehicle charging station is modeled. The charging station includes multiple charging piles and a local energy storage system. The energy storage system has bidirectional power regulation capabilities for charging and discharging. The power of the energy storage system is subject to the maximum access limit of the power grid. The energy storage charge state is updated in each scheduling cycle and maintained within the preset upper and lower limits. Assume that the scheduling time of the electric vehicle charging station is discretized into a finite set of time steps in Represents the current scheduling period. Suppose there are |C| charging piles forming a set C, and the output power of the i-th charging pile at time t is P t i , the total power of the charging station is:

[0061]

[0062] The power regulation value of the energy storage system is P t S Indicates that P t S >0 means discharge, P t S <0 means charging, and the allowed power range is:

[0063]

[0064] To prevent the charging station from overloading, the total power must not exceed the maximum access power limit of the grid P g :

[0065] P t C +P t S ≤P g

[0066] The state of charge (SOC) of the local energy storage system is updated at each time step according to the following relationship:

[0067]

[0068] Where Δt represents the actual duration of each time step, E max Indicates the maximum energy capacity of the energy storage system. To ensure the life of the equipment, Constrained to the interval Run inside.

[0069] Step S2: Input the future electric vehicle arrival rate prediction results

[0070] At the beginning of each scheduling cycle, the system receives the predicted value of the electric vehicle arrival rate within a future time window, denoted as λ t , which describes the number of electric vehicles expected to arrive at a charging station per unit time. This value can be provided by the traffic prediction module, historical data modeling, or an external interface. The arrival rate prediction is input into the dispatch controller as part of the charging station status, which is used to predict load levels and plan dispatch strategies.

[0071] Step S3: Build a deep reinforcement learning-based scheduling controller and define the state-action structure

[0072] In order to realize the optimal scheduling strategy based on state, the soft policy reinforcement learning algorithm SoftActor-Critic (SAC) is used to build the scheduling controller framework. In the scheduling controller, the input is the state vector s t Including the current energy storage charge state Arrival rate prediction value λ t 、Current electricity price p t, charging pile occupancy rate, current pricing strategy The output is the action vector a t Including the power distribution of each charging pile {P t i}, energy storage system power regulation value P t S And the charging price announced to the public

[0073] Step S4: Design a reward function to guide economic and operational stability optimization

[0074] In order to make the reinforcement learning strategy tend to maximize the benefits during the training process and comply with the system constraints, the comprehensive reward function is constructed as follows:

[0075] R t =Ω t -ω1P exc -ω2P diff +ω3R diff -ω4P swi -ω5P util

[0076] Among them, R t is the reward value at the current moment t, Ω t is the profit of the charging station at the current time t, P exc is the penalty for exceeding the power limit of the power grid, P diff is the energy storage price difference penalty, R diff To provide reasonable arbitrage rewards for energy storage, P swi Penalty for frequent energy storage switching, P utili is the penalty for insufficient energy storage utilization, and ω1 to ω5 are the weight coefficients of various penalties and rewards.

[0077] Charging profit Ω is adjusted according to the energy storage power value P t S The symbols are calculated piecewise:

[0078]

[0079] Among them, P t i represents the power of the i-th charging pile at time t, The user-side charging electricity price set for the site, p t is the basic electricity price on the grid side, p s is the electricity purchase cost of the energy storage system, Δt is the time interval, P t S It is the power regulation value of the energy storage system, which is positive for discharge and negative for charge.

[0080] Grid power over-limit penalty P excThe expression is as follows:

[0081]

[0082] in, is the total charging power, P g is the maximum power limit for grid connection, is the relative limit-crossing ratio, λ dyn =λ base (1+γ·n exceed ) is the dynamic penalty coefficient, λ base is the basic penalty weight, γ is the growth coefficient, n exceed The number of consecutive over-limits.

[0083] The function f(δ) is defined as:

[0084]

[0085] Among them, δ1 and δ2 are two thresholds of the relative over-limit ratio, which are used to adjust the penalty size.

[0086] In order to unify the modeling of energy storage price difference penalties and energy storage reasonable arbitrage incentives, an auxiliary variable is introduced:

[0087] V=|p s -p t |·|P t S |·Δt

[0088] The energy storage price mismatch penalty and reward items are:

[0089]

[0090] The symbolic function is defined as:

[0091]

[0092] Energy storage frequent switching penalty P swi The definition is as follows:

[0093]

[0094] in, is the energy storage power at the previous moment, is the energy storage rated power, n swi For continuous switching count, To activate the indicator function, when the condition C is met swi hour, otherwise The conditions are defined as:

[0095]

[0096] Among them, T P is the energy storage active power threshold, T change is the power change threshold.

[0097] Penalty P for insufficient energy storage utilization utili is defined as:

[0098]

[0099] The above reward function guides charging scheduling behavior through the profit maximization goal. At the same time, it combines penalties and incentives for charging station capacity limitations, energy storage operation rationality, switching frequency and resource utilization, effectively improving the convergence efficiency of the reinforcement learning strategy and the stability of the scheduling method operation. It is suitable for the coordinated scheduling scenario of charging stations with local energy storage systems.

[0100] Step S5: Train reinforcement learning strategy using time-profit priority mechanism

[0101] The time-profit priority experience sampling mechanism is used to improve learning efficiency. For each sample (s i ,a i ,r i ,s i ′), its sampling priority is calculated as follows:

[0102]

[0103] Among them, Pr i represents the comprehensive sampling priority of the i-th transition, which is composed of the profit-based priority Pr i Priority based on temporal continuity By weight ω profit and ω temp Linear weighting is obtained, where ω profit +ω temp =1, κ is the time attenuation coefficient, d i is the time distance between the current time step and transition i; profit priority Pr i And by the transition profit Ω i Calculated, specifically:

[0104] If Ω i >0, then

[0105] Pr i =(Ω i +∈) α

[0106] otherwise

[0107]

[0108] Among them, ∈ is a very small positive number that prevents the sampling probability from being zero, and α∈[0,1] controls the degree of preference of sampling for high-profit transitions.

[0109] Step S6: Deploy the scheduling control strategy and execute the joint scheduling process in real time

[0110] In the actual operation phase of the scheduling method, the trained deep reinforcement learning scheduling strategy is embedded into the scheduling controller, which executes the scheduling decision process in a periodic manner. Whenever a new scheduling time t is entered, the current state information is first collected, including the current energy storage charge state Arrival rate prediction value λ t 、Current electricity price p t , charging pile occupancy rate, current pricing strategy

[0111] Subsequently, the state information is input into the reinforcement learning policy network, and the scheduling controller is based on the learned policy π(a t |s t ) Output the joint control action a of the current scheduling cycle t , including the power distribution of each charging pile {P t i}, energy storage system power regulation value P t S And the charging price announced to the public

[0112] After the action is generated, the dispatch controller immediately performs actual power allocation and price update according to the instruction. t i Control signal to adjust the output power of each pile; the energy storage management module adjusts the output power of each pile according to P t S Instructions to perform corresponding charging and discharging operations; the price management system will The price is dynamically adjusted by the station operation platform. In this process, the control strategy ensures that the operating power of the charging station does not exceed the upper limit of the grid access P. g At the same time, energy storage can be fully utilized as a load regulation buffer to ensure service continuity and avoid grid penalties caused by power surges.

[0113] The dispatch controller updates the energy storage status synchronously during execution The next time step loop begins. Throughout the entire process, the scheduling strategy utilizes high-frequency, refined control outputs to continuously optimize resource allocation, and a reward mechanism embedded with economic feedback guides behavioral decisions toward profit maximization.

Claims

1. A method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning, characterized in that: The steps include: Establish a charging station power model, where the charging station includes multiple charging piles and a local energy storage system with bidirectional power regulation for charging and discharging. The energy storage system power is subject to the maximum access limit of the grid, and the energy storage charge state is updated during each scheduling cycle and maintained within the preset upper and lower limits. Obtain a predicted arrival rate for electric vehicles over a period of time. The arrival rate represents the number of electric vehicles expected to arrive at a charging station per unit time and serves as one of the key input parameters for the charging station status, guiding the dispatch controller to pre-allocate power resources and pricing strategies. Build a dispatch controller based on deep reinforcement learning, taking the energy storage state of charge, the predicted arrival rate, the current electricity price, the charging pile occupancy rate, and the current pricing strategy as state inputs. The output includes a combined action on the power allocation of each charging pile, the energy storage system power adjustment value, and the charging price. Design a reward function for multi-objective optimization. This reward function includes six indicators: charging profit, grid power over-limit penalty, energy storage price mismatch penalty, energy storage reasonable arbitrage reward, energy storage switching frequency penalty, and energy storage under-utilization penalty. This is used to balance economic objectives with operational stability objectives. A priority experience sampling mechanism based on the dual indicators of time correlation and profit contribution is constructed. During the control strategy training process, sampling weights are dynamically assigned based on the economic benefits and time distance of each historical experience, so that the scheduling controller prioritizes learning high-value, highly relevant scheduling samples. The trained control strategy is deployed at the actual charging station. The charging station status is sensed in real time during each scheduling cycle. The scheduling controller calls the control strategy network to generate control actions, and drives the charging pile power output, energy storage power regulation and charging price setting to achieve an integrated closed-loop scheduling operation of charging station power distribution, energy storage system power regulation and electricity price control.

2. The method for coordinated scheduling of local energy storage in electric vehicle charging stations based on deep reinforcement learning according to claim 1, characterized in that: The power regulation value of the energy storage system is an adjustable continuous value, and maximum limit values ​​are set in the charging direction and the discharging direction respectively to ensure the safe operation of the battery system.

3. The method for coordinated scheduling of local energy storage in electric vehicle charging stations based on deep reinforcement learning according to claim 1, characterized in that: The reward function is as follows: R t =Oh t -ω1P exc -ω2P diff +ω3R diff -ω4P swi -ω5P util Among them, R t is the reward value at the current moment t, Ω t is the profit of the charging station at the current time t, P exc P is the penalty for exceeding the power limit of the power grid. diff is the energy storage price difference penalty, R diff To provide reasonable arbitrage rewards for energy storage, P swi Penalty for frequent energy storage switching, P utili is the penalty for insufficient energy storage utilization, ω1 to ω5 are the weight coefficients of various penalties and rewards; Charging profit Ω is adjusted according to the energy storage power value P t S The symbols are calculated piecewise: Among them, P t i represents the power of the i-th charging pile at time t, The user-side charging electricity price set for the site, p t is the basic electricity price on the grid side, p s is the electricity purchase cost of the energy storage system, Δt is the time interval, P t S is the power regulation value of the energy storage system, which is positive for discharge and negative for charge; Grid power over-limit penalty P exc The expression is as follows: in, is the total charging power, P g is the maximum power limit for grid connection, is the relative limit-crossing ratio, λ dyn =λ base (1+γ·n exceed ) is the dynamic penalty coefficient, λ base is the basic penalty weight, γ is the growth coefficient, n exceed The number of consecutive exceeding limits; The function f(δ) is defined as: Among them, δ1 and δ2 are two thresholds of relative over-limit ratio, which are used to adjust the penalty size; In order to unify the modeling of energy storage price difference penalties and energy storage reasonable arbitrage incentives, an auxiliary variable is introduced: V=|p s -p t |·|P t S |·Δt The energy storage price mismatch penalty and reward items are: The symbolic function is defined as: Energy storage frequent switching penalty P swi The definition is as follows: in, is the energy storage power at the previous moment, is the energy storage rated power, n swi For continuous switching count, To activate the indicator function, when the condition C is met swi hour, otherwise The conditions are defined as: Among them, T P is the energy storage active power threshold, T change is the power change threshold; Penalty P for insufficient energy storage utilization utili is defined as:

4. The method for coordinated scheduling of local energy storage in electric vehicle charging stations based on deep reinforcement learning according to claim 1, characterized in that: The specific calculation method of the priority experience sampling mechanism based on the dual indicators of time correlation and profit contribution is as follows: For each sample (s i ,a i ,r i ,s′ i ), its sampling priority is calculated as follows: Among them, Pr i represents the comprehensive sampling priority of the i-th transition, which is composed of the profit-based priority Pr i Priority based on temporal continuity By weight ω profit and ω temp Linear weighting is obtained, where ω profit +ω temp =1, κ is the time attenuation coefficient, d i is the time distance between the current time step and transition i; profit priority Pr i And by the transition profit Ω i Calculated, specifically: If Ω i >0, then Pr i =(Ω i +) α otherwise Among them, ∈ is a very small positive number that prevents the sampling probability from being zero, and α∈[0,1] controls the degree of preference of sampling for high-profit transitions.

5. The method for coordinated scheduling of local energy storage in electric vehicle charging stations based on deep reinforcement learning according to claim 1, characterized in that: The dispatch controller supports a dynamic pricing mechanism. In each dispatch cycle, the charging price is determined based on the arrival rate of electric vehicles, the current electricity price and the load level of the charging station, achieving price elasticity and load guidance, thereby improving user regulation capabilities and operating benefits.

6. The method for coordinated scheduling of local energy storage at electric vehicle charging stations based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning-based dispatch controller introduces future electric vehicle arrival rate prediction information as state input to dynamically adjust energy storage charging and discharging plans and pricing strategies to achieve early response to peak loads and forward-looking scheduling of power resources.

Images