Traffic hub parking lot charging station energy management method based on hierarchical reinforcement learning

Abstract

This invention discloses an energy management method for charging stations in transportation hub parking lots based on hierarchical reinforcement learning, relating to the fields of smart grid and electric vehicle charging control technology. The method includes constructing a hierarchical control model architecture and dividing vehicles into multiple groups based on urgency. Upper-layer agents output recommended power for each group based on the overall load status of the parking lot and the characteristics of each group, using a flexible actor-critic algorithm. Lower-layer electric vehicle agents use the upper-layer recommended power as part of their state, combining their own state and real-time electricity price to output the charging and discharging power of specific vehicles. A multi-objective reward function incorporating electricity cost, range anxiety, and battery loss is constructed for network training. This invention solves the curse of dimensionality problem under large-scale electric vehicle access, balancing user cost, charging satisfaction, and battery health. It also considers the vehicle-to-grid model, improving voltage fluctuations by allowing electric vehicles to discharge during peak grid demand.

Description

Technical Field

[0001] This invention relates to the field of smart grid and electric vehicle charging control technology, and in particular to an energy management method for charging stations in transportation hub parking lots based on hierarchical reinforcement learning. Background Technology

[0002] Global climate change, the energy crisis, and environmental pollution are accelerating the electrification transformation of the transportation sector. As a core component of clean transportation systems, electric vehicles not only effectively reduce fossil fuel consumption but also significantly alleviate air quality deterioration and climate change, especially as the proportion of renewable energy in the electricity supply continues to increase, their advantages in reducing greenhouse gas emissions will become even more pronounced. Furthermore, electric vehicles provide flexibility to the energy system, helping to improve grid stability.

[0003] With the continuous increase in the market penetration rate of electric vehicles, the large-scale deployment of charging infrastructure has become an inevitable requirement. However, the randomness and uncertainty of electric vehicle charging behavior can easily lead to waste of charging resources and fluctuations in grid load, posing significant challenges to the stable operation of the power system. On the one hand, the simultaneous charging of a large number of electric vehicles can easily increase the burden on the distribution network, even exceeding its carrying capacity; on the other hand, traditional charging station facilities face difficulties in battery resource allocation and energy interaction coordination with the grid, especially when charging demand fluctuates. Implementing an orderly charging strategy can help the grid "reduce peak loads and fill valleys," improve equipment utilization, and reduce the costs of power companies. Therefore, coordinating and scheduling the charging and discharging behavior of electric vehicles is crucial for improving the stability of the power system and the satisfaction of electric vehicle owners. In addition, electric vehicles are often idle during parking periods, and this idle time accounts for a significant portion of their life cycle. By applying vehicle-to-grid (V2G) technology, electric vehicles can become mobile energy storage units, feeding excess electricity back to the power system during peak electricity consumption periods in a vehicle-to-grid (V2G) mode to participate in energy market transactions. This behavior can enhance the economic sustainability of the energy system.

[0004] To develop a vehicle-grid coordinated scheduling mechanism that can both guarantee the charging needs of electric vehicle owners and effectively alleviate power grid congestion, scholars have constructed and analyzed mathematical models integrating distribution networks and electric vehicles from two perspectives. From the perspective of transportation hub energy systems, existing research mainly focuses on global energy coordinated scheduling strategies to ensure the safe operation of the power system, covering key areas such as planning, scheduling, uncertainty handling, and assessment. However, these studies often overlook the local interaction needs and real-time response mechanisms that exist in practical applications.

[0005] From the perspective of system operators, existing research mainly focuses on the coordinated optimization of global traffic assignment and power generation scheduling, but neglects the local interaction and dynamic adjustment problems that are widespread in practical applications. To address this issue, researchers have proposed a decentralized two-layer optimization method that simulates the dynamics of traffic flow propagation between different networks by optimizing cross-regional traffic flow. However, such model-based optimization methods face challenges of insufficient modeling accuracy when dealing with scenarios with multiple uncertainties (such as real-time electric vehicle charging demand and dynamic nodal marginal price (LMP)), and often only obtain suboptimal solutions.

[0006] From the perspective of electric vehicle users, research on the coordinated operation of integrated power distribution and transportation networks mainly focuses on choosing the shortest path to a charging station (CS), while neglecting the differences in charging and discharging behavior between different charging stations. Electric vehicles utilize their mobility to charge during periods of low electricity prices and discharge during periods of high electricity prices to gain economic benefits. With the implementation and application of vehicle-to-everything (V2X) technology, electric vehicles can improve the operational flexibility of integrated power distribution and transportation networks while simultaneously achieving economic benefits.

[0007] Traditional charging management methods typically employ rule-based control or convex optimization, but these suffer from poor real-time performance in highly dynamic environments. In recent years, reinforcement learning (RL) has been widely applied to energy management, but it faces two main challenges: (1) Curse of Dimensionality: The number of vehicles in the hub parking lot is large. If a single agent is used for centralized control, the action space will increase exponentially with the number of vehicles, making training difficult. If multiple agents are used for independent control, the non-stationarity of the environment will lead to convergence difficulties.

[0008] (2) Multi-objective conflict: Charging scheduling needs to balance the stability of the grid side (peak shaving and valley filling) and the interests of the user side (low cost, full charge, and protection of battery life). Existing methods often cannot take all objectives into account.

[0009] Therefore, there is an urgent need for a hierarchical energy management strategy that can handle large-scale vehicle access and effectively balance the interests of multiple parties. Summary of the Invention

[0010] To address the aforementioned issues, this application proposes an energy management strategy for charging stations in urban transportation hub parking lots. It employs a hierarchical multi-agent deep reinforcement learning method, which solves the dimensionality problem of large-scale electric vehicle scheduling and achieves multi-objective optimization through a mechanism of planning group power at the upper level and controlling individual vehicle power at the lower level.

[0011] To achieve the above objectives, this invention provides an energy management method for charging stations in transportation hub parking lots based on hierarchical deep reinforcement learning, the specific steps of which are as follows: S1. Construct an electric vehicle charging environment model based on the geographical environment and operation of parking lots at transportation hubs, and statistically analyze vehicle arrival and charging / discharging demand based on the electric vehicle charging environment model. S2. Establish a vehicle grouping mechanism based on urgency. Calculate the urgency value based on vehicle arrival status and charging / discharging demand. Preset urgency value range. Divide vehicles into several groups according to the urgency value range and record the aggregation characteristics of the vehicles. S3. Construct a hierarchical multi-agent deep reinforcement learning control architecture, and process the aggregated features of the vehicle based on the hierarchical multi-agent deep reinforcement learning control architecture to generate energy management strategies for charging stations. The architecture comprises an upper-level agent and multiple lower-level agents corresponding to the number of groups; The upper-layer intelligent agent is used to extract the aggregation features of each group and the normalized value of the current power grid load as the upper-layer state, and output the recommended charging and discharging power range of each group as the upper-layer action. The lower-level intelligent agent maps the upper-level actions to the recommended power values ​​of each group and uses them as part of the lower-level state; the lower-level intelligent agent combines the single-vehicle state information to output the specific charging and discharging power of each electric vehicle in groups.

[0012] Preferably, the specific content of calculating the urgency value based on vehicle arrival status and charging / discharging demand in S2 includes analyzing the remaining parking time based on vehicle arrival status; The vehicle's charging requirements are determined by analyzing charging and discharging demands. The urgency score is calculated based on the vehicle's charging needs and remaining parking time. The expression is: ; in, For electric vehicle capacity, For the desired state of charge, The current state of charge, The remaining berthing time.

[0013] Based on the urgency level U, vehicles are dynamically assigned to eight preset groups, each corresponding to a different urgency level range.

[0014] Preferably, the state space dimension of the upper-layer agent in S3 is N×8, where N is the number of groups; The characteristics of each group include: number of vehicles, maximum remaining time, minimum remaining time, maximum demand gap, minimum demand gap, average remaining time, average demand gap, and current normalized base load; The action space dimension of the upper-level agent is N, and the output value is a continuous value between [-1, 1], which is used to guide the power allocation tendency of each group; The rewards for upper-level agents include load smoothing rewards. The load easing reward It is negatively correlated with the degree to which the total load of the parking lot exceeds the transformer capacity limit.

[0015] Preferably, the state space of the lower-level intelligent agent in S3 includes four dimensions: remaining parking time of the vehicle, power demand gap, recommended power of the group to which it belongs, and current normalized electricity price; The action space dimension of the lower-level agent is 1, and the output value is a continuous value between [-1, 1]. Positive values ​​represent charging, and negative values ​​represent discharging.

[0016] Preferably, the rewards for the lower-level agents in S3 adopt a multi-objective weighted reward function. The expression is: ; in, As a weighting factor for electricity cost rewards, To weight the mileage anxiety reward, Weighting of battery wear rewards. As a reward for electricity cost, Rewards for mileage anxiety Rewards for battery wear; Electricity cost incentive It is negatively correlated with the product of real-time electricity price and actual power output; The Mileage Anxiety Reward It is negatively correlated with the square of the uncharged charge when the vehicle leaves the site or charging is completed, and negatively correlated with the degree of deviation of the actual power from the recommended power during charging. Battery loss reward It is negatively correlated with the magnitude of charging and discharging power and the amplitude of power fluctuation.

[0017] Preferably, the training of the hierarchical multi-agent deep reinforcement learning control architecture employs the SAC algorithm, including: Based on the SAC algorithm, experience replay buffers are established for the upper and lower layers respectively, and the parameters of the upper and lower neural networks are updated alternately. Automatic entropy adjustment is introduced to enhance the exploration capability.

[0018] Preferably, the SAC algorithm includes an Actor network, a Critic network, and a Target Critic network; An automatic entropy adjustment mechanism is introduced during training to dynamically adjust the balance between exploration and utilization by optimizing the log standard deviation (log alpha).

[0019] Data preprocessing steps: The upper and lower state states are normalized online using a Z-Score filter (ZFilter) to eliminate the influence of data with different dimensions on the training of the neural network.

[0020] Preferably, the training process for constructing the hierarchical multi-agent deep reinforcement learning control architecture includes: Step 1, Network Initialization: Construct the upper-layer Actor / Critic network and the lower-layer 8 groups of Actor / Critic networks; Step 2, Experience Replay: Set up the experience replay pool for the upper-level model. and the lower-level model experience replay pool ; The lower-level experience replay pool buffer stores tuples of (state, action, reward, next_state) at the vehicle granularity.

[0021] Step 3, Update Strategy: Every fixed number of steps, sample batch data from the experience replay pool Buffer, calculate the temporal difference TD error to update the value network Critic, and maximize the Q value and entropy to update the Actor policy network. Step 4, Automatic Entropy Adjustment: By optimizing the entropy temperature coefficient The parameters are dynamically adjusted to increase the exploration level and prevent the strategy from getting trapped in local optima too early.

[0022] Preferably, step 3 includes the following: Step 31: The SAC algorithm adopts the reinforcement learning objective of maximizing entropy regularization. To achieve this objective, a network architecture including an Actor network, two Critic networks, and an automatic entropy adjustment module is constructed. Target Strategy The definition of is: ; in, Represents the trajectory distribution under the strategy. As a discount factor, For the reward function, Here is the entropy regularization coefficient. For the entropy of the strategy, This represents the optimal strategy. Let t represent the long-term expected goal, and t represent the current time. This represents the environmental state at time t. This represents the action performed by the agent at time t; Step 32: Optimize the Q-network parameters by minimizing the soft Bellman residual, defining the loss function. for: ; in, For experience replay buffer, For the target Q value, This represents a value assessment network that combines the minimum of two objective networks to mitigate the overestimation problem. ; In the formula, The parameters represent the target Critic network. These are the parameters of the Actor network; Step 33, the Actor network parameters The update process includes using reparameter techniques to update the action. Representing state and noise Functions: ; in, Follows a standard normal distribution. This represents the mean of the policy network. The standard deviation of the policy network is represented. Represents a policy network; Optimize network parameters by minimizing KL divergence; Step 34, the entropy regularization coefficient The automatic adjustment process includes establishing an objective function for entropy automatic adjustment, aiming to maintain the policy at the minimum target entropy during training. The randomness of its loss function Defined as; ; Through calculation about The gradient is calculated and updated to achieve the temperature coefficient. Adaptive adjustment.

[0023] In summary, the energy management method for charging stations in transportation hub parking lots based on hierarchical reinforcement learning of the present invention has the following advantages compared with traditional technologies: 1. Layered architecture reduces dimensionality: The problem of controlling hundreds of vehicles is transformed into the control of 8 groups and the independent control of individual vehicles, which significantly reduces the dimensionality of the state space and action space and improves the scalability of the algorithm; 2. Multi-objective collaborative optimization: The lower-level reward function explicitly designs three indicators: cost, anxiety, and battery degradation. These indicators are then balanced by manually setting weights by experts. This satisfies users' charging needs while reducing costs and battery aging. 3. Grid-friendly: The upper-level intelligent agent focuses on global load control, effectively preventing transformer overload; 4. Strong algorithm robustness: The SAC algorithm and automatic entropy adjustment are adopted, which have better exploration ability and stability compared with traditional DDPG or DQN algorithms; combined with Z-Score normalization, the adaptability to different data distributions is improved.

[0024] The technical method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of energy flow in a transportation hub parking lot according to an embodiment of the present invention; Figure 2 This is an overall flowchart of the method of the present invention; Figure 3 This is a schematic diagram of the hierarchical multi-agent control architecture of the present invention; Figure 4 The figures show the charging and discharging power and state of charge (SOC) variation curves of a typical electric vehicle in this embodiment of the invention. Figure 4 (a) in the figure shows the time-series curve of charging power variation during the parking period of vehicle 1. Figure 4 (b) in the figure shows the curve of battery load of vehicle 1 changing over time. Figure 4 (c) in the figure represents the time-series curve of charging power variation during the parking period of vehicle 2. Figure 4 In the figure, (d) represents the curve of battery load of vehicle 2 changing over time. Figure 4 (e) in the figure represents the time-series curve of charging power variation during the vehicle's 3-day stay. Figure 4 (f) in the figure represents the curve of the battery load of vehicle 3 changing over time; Figure 5 This is the training convergence curve of the lower-layer agent's multi-objective reward and the upper-layer reward in an embodiment of the present invention. Figure 5 In the figure, (a) represents the convergence curve of the total reward of the lower-level network. Figure 5 (b) in the figure represents the cost-reward convergence curve of the lower-layer network. Figure 5 (c) in the figure represents the convergence curve of the loss reward of the lower-layer network. Figure 5 (d) in the figure represents the anxiety-reward convergence curve of the lower-level network; Figure 6 This is the optimized total power load distribution curve of the parking lot throughout the day in an embodiment of the present invention. Detailed Implementation

[0026] The technical method of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0028] Techniques, systems, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the instruction manual.

[0029] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0030] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0031] Example 1 This embodiment proposes an energy management method for charging stations in transportation hub parking lots based on hierarchical reinforcement learning, including the following steps: S1. Environmental Modeling and Data Acquisition: Construct a simulation environment with 96 time steps (24 hours) and statistically analyze the vehicle arrival process and charging / discharging requirements. S2. Dynamic Grouping: Calculate the "urgency" based on the vehicle's remaining parking time and charging demand gap, and divide the vehicle into 8 groups based on the urgency to achieve dimensionality reduction. like Figure 3 As shown, S3, upper-level control (group level): Status: Input the statistical characteristics of 8 groups (number of vehicles, maximum / minimum / average remaining time, maximum / minimum / average demand) and the grid base load.

[0032] Action: Output the normalized recommended power for the 8 groups.

[0033] Incentives: Primarily penalize total load exceeding limits (exceeding transformer capacity) to guide the power grid in peak shaving and valley filling.

[0034] like Figure 3 As shown, S4, lower-level control (vehicle level): Status: Enter the remaining time of the bicycle, the demand gap, the upper-level recommended power of the group, and the current electricity price.

[0035] Action: Outputs the actual charging and discharging power of the bicycle.

[0036] Rewards: A weighted sum of electricity cost, range anxiety, and battery degradation.

[0037] S5. Model Training: Based on the Soft Actor-Critic (SAC) algorithm, experience replay buffers are established for the upper and lower layers respectively. The parameters of the upper and lower neural networks are updated alternately, and automatic entropy adjustment is introduced to enhance the exploration capability.

[0038] Example 2 Integrated transportation hubs occupy a pivotal position in the national transportation network, handling massive passenger and freight volumes daily. The energy management strategies for charging stations in hub parking lots have thus attracted widespread attention from all sectors of society. For example... Figure 1 As shown, this embodiment uses a transportation hub as an example to further illustrate the specific application of the technical method of the present invention. A method for energy management of charging stations in transportation hub parking lots based on hierarchical deep reinforcement learning includes the following steps: Figure 2 As shown: S1. Construct a parking lot environment for electric vehicle charging and discharging.

[0039] S11. Set the time step d=0.25 hours, with 96 time steps throughout the day. The base load and time-of-use electricity price are set according to the actual city data. S12. The vehicle model includes the following attributes: arrival time, departure time, initial SOC, expected SOC, battery capacity (60kWh), maximum charging power (30kW), and maximum discharging power (10kW).

[0040] S13. Vehicle departure data is based on actual data from the Hongqiao Hub West Parking Lot, and is divided into two categories: charging type and discharging type.

[0041] S2. Dynamic grouping based on urgency, defining vehicle urgency: .

[0042] Eight urgency ranges are defined for each group: [-10, -5], [-5, 0], [0, 5], [5,10], [10,15], [15, 20], [20, 25], [25, 30]. Vehicles outside these ranges are grouped into the first and last groups. At each time step, the group a vehicle belongs to is updated based on its real-time status.

[0043] like Figure 3 As shown, S3, the upper-layer intelligent agent, is responsible for global resource allocation: S31. Input: Dimensions are 64 (8 groups × 8 features). Features are normalized using ZFilter. Features include: number of vehicles in each group, remaining time (maximum / minimum / mean), demand gap (maximum / minimum / mean), and normalized value of current grid base load.

[0044] S32. Network Structure: Both the Actor and Critic networks are fully connected neural networks with a hidden layer dimension of 128.

[0045] S33, a control signal with a dimension of 8, corresponding to 8 groups. .

[0046] S34, Reward Function: .

[0047] in For total load, This refers to the transformer capacity.

[0048] like Figure 3 As shown, S4 and the lower-level intelligent agents are responsible for specific execution, with a total of 8 sub-intelligent agents corresponding to 8 groups.

[0049] S41. Mapping Mechanism: Mapping upper-level actions Mapped to the recommended power of the i-th group .

[0050] S42. Input: Dimension 4. Includes: Remaining parking time per vehicle, vehicle power demand gap, recommended power for this group, and current normalized electricity price.

[0051] S42, Output: Dimension 1 Actual power Scaling is based on the vehicle's maximum charging and discharging capacity.

[0052] S43. Multi-objective reward function: (1) Cost: .

[0053] (2) Anxiety: .

[0054] (3) Loss: Punishment for high power and frequent power fluctuations.

[0055] Total Rewards: .

[0056] S5 and SAC algorithm training: S51. Network initialization: Construct the upper-layer Actor / Critic network and the lower-layer 8 groups of Actor / Critic networks.

[0057] S52, Experience Replay: Settings (Capacity 10000) and (Capacity 100,000).

[0058] The lower-level buffer stores (state, action, reward, next_state) tuples at the vehicle level.

[0059] S53. Update strategy: Every fixed number of steps, sample batches from the buffer, calculate the TD error to update the Critic, and maximize the Q value and entropy to update the Actor.

[0060] S54, Automatic Entropy Adjustment: Through optimization The parameters dynamically adjust the exploration level (Alpha value) to prevent the strategy from getting trapped in local optima too early.

[0061] S6. Performance Evaluation: During training, record the total reward, cost component, anxiety component, and loss component for each round. After training, load the optimal model parameters, test on the evaluation set, and output the SOC change curve for each vehicle and the total power curve of the parking lot to verify the peak shaving and valley filling effect and user satisfaction.

[0062] Figure 4 The study demonstrates the temporal changes in charging and discharging power and the evolution of battery state of charge (SOC) of three typical electric vehicles during their stationary periods. The dynamic adjustment of vehicle charging power reflects the real-time optimization of the charging strategy by the intelligent agent. Furthermore, the battery load state eventually approaches or reaches the desired state, proving that the proposed method can formulate a charging and discharging strategy that meets the battery load state based on vehicle status and environmental information, and achieve dynamic optimization of charging power.

[0063] Figure 5 The convergence trend of the multi-objective reward in the lower layer and the total reward in the upper layer during the algorithm training process is shown. As can be seen from the figure, all reward curves fluctuate rapidly in the early stage of training, then gradually stabilize and finally converge to a stable value. This proves that the proposed multi-objective reinforcement learning algorithm has good convergence and stability, and can gradually optimize each sub-objective during the training process, and finally achieve a stable strategy of multi-objective collaborative optimization.

[0064] Figure 6 The graph shows the total power load distribution curve of the parking lot throughout the day. As can be seen from the graph, the power amplitude of the grid is small and the distribution is uniform, which reflects the orderly discharge strategy and avoids the impact on the grid. The total power load curve is smooth, which realizes the peak shaving and valley filling of the load. This proves that the proposed scheduling strategy can effectively optimize the overall power load of the parking lot. Through orderly charging and discharging, it achieves friendly interaction with the grid, meets the charging needs of vehicles, smooths the load fluctuation, and improves the stability of grid operation.

[0065] This invention employs a hierarchical deep reinforcement learning-based energy management method for charging stations in transportation hub parking lots, filling the technological gap in real-time scheduling and multi-objective collaborative optimization of large-scale electric vehicle clusters. It establishes a dynamic grouping mechanism based on urgency and a two-layer control architecture for large-scale vehicles, achieving accurate dimensionality reduction and efficient solution of the high-dimensional state space. Through the Flexible Actor-Critic (SAC) algorithm and automatic entropy adjustment mechanism, it balances grid load fluctuations with user charging demand, clarifying the robustness of hierarchical collaborative control in complex dynamic environments. By leveraging a multi-dimensional weighted reward function, it accurately considers key influencing indicators such as electricity cost, range anxiety, and battery degradation, providing intelligent decision support for personalized charging strategy formulation. This method balances grid operation stability with maximizing user benefits, helping urban hub parking lots optimize energy allocation and reduce operational risks, providing a feasible technical path for the integrated development of smart transportation and the energy internet.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical methods of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical methods of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical methods to deviate from the spirit and scope of the technical methods of the present invention.

Claims

1. A traffic hub parking lot charging station energy management method based on hierarchical reinforcement learning, characterized in that, Includes the following steps: S1. Construct an electric vehicle charging environment model based on the geographical environment and operation of parking lots at transportation hubs, and statistically analyze vehicle arrival and charging / discharging demand based on the electric vehicle charging environment model. S2. Calculate the urgency value based on vehicle arrival status and charging / discharging demand. There is a preset urgency value range. Divide the vehicles into several groups according to the urgency value range and record the aggregation characteristics of the vehicles. S3. Construct a hierarchical multi-agent deep reinforcement learning control architecture, and process the aggregated features of the vehicle based on the hierarchical multi-agent deep reinforcement learning control architecture to generate energy management strategies for charging stations. The architecture comprises an upper-level agent and multiple lower-level agents corresponding to the number of groups; The upper-layer intelligent agent is used to extract the aggregation features of each group and the normalized value of the current power grid load as the upper-layer state, and output the recommended charging and discharging power range of each group as the upper-layer action. The lower-level intelligent agent maps the upper-level actions to the recommended power values ​​of each group and uses them as part of the lower-level state; the lower-level intelligent agent combines the single-vehicle state information to output the specific charging and discharging power of each electric vehicle in groups.

2. The hierarchical reinforcement learning based transportation hub parking lot charging station energy management method according to claim 1, characterized in that, The specific content of S2 in calculating the urgency value based on vehicle arrival status and charging / discharging demand includes analyzing the remaining parking time based on vehicle arrival status. The vehicle's charging requirements are determined by analyzing charging and discharging demands. The urgency value is calculated based on the vehicle's charging needs and the remaining parking time. The expression for the urgency value is: u = 1 - (t / T) ； wherein, is the electric vehicle capacity, is the desired state of charge, is the current state of charge, is the remaining parking time.

3. The hierarchical reinforcement learning based energy management method for traffic hub parking lot charging stations according to claim 2, wherein, The state space dimension of the upper-level agent in S3 is N×8, where N is the number of groups; The characteristics of each group include: number of vehicles, maximum remaining time, minimum remaining time, maximum demand gap, minimum demand gap, average remaining time, average demand gap, and current normalized base load; The action space dimension of the upper-level agent is N, and the output value is a continuous value between [-1, 1], which is used to guide the power allocation tendency of each group; The reward of the upper agent includes a load leveling reward The load leveling reward is negatively correlated with the extent to which the total load of the parking lot exceeds the transformer capacity limit.

4. The hierarchical reinforcement learning based transportation hub parking lot charging station energy management method according to claim 3, characterized in that, The state space of the lower-level intelligent agents in S3 includes four dimensions: remaining parking time of the vehicle, power demand gap, recommended power of the group to which the vehicle belongs, and current normalized electricity price; The action space dimension of the lower-level agent is 1, and the output value is a continuous value between [-1, 1]. Positive values ​​represent charging, and negative values ​​represent discharging.

5. The hierarchical reinforcement learning based transportation hub parking lot charging station energy management method according to claim 4, characterized in that, The reward of the lower intelligent agent in S3 adopts a multi-objective weighted reward function, and the expression of the multi-objective weighted reward function is: ​ ； wherein, is a cost of electricity reward weight, is a range anxiety reward weight, is a battery degradation reward weight, is a cost of electricity reward, is a range anxiety reward, is a battery degradation reward; The electricity cost reward is negatively correlated with the product of real-time electricity price and actual power The range anxiety reward Square negative correlation with the amount of charge left when the vehicle leaves or charging is completed, negative correlation with the degree of deviation from the recommended power with the actual power during charging; The battery loss reward Negative correlation with the size of the charging and discharging power and the power fluctuation amplitude.

6. The hierarchical reinforcement learning based transportation hub parking lot charging station energy management method according to claim 5, characterized in that, The training of the hierarchical multi-agent deep reinforcement learning control architecture employs the SAC algorithm, including: Based on the SAC algorithm, experience replay buffers are established for the upper and lower layers respectively, and the parameters of the upper and lower neural networks are updated alternately. Automatic entropy adjustment is introduced to enhance the exploration capability.

7. The hierarchical reinforcement learning based energy management method for traffic hub parking lot charging stations according to claim 6, wherein, The SAC algorithm includes an Actor network, a Critic network, and a Target Critic network; An automatic entropy adjustment mechanism is introduced during the training process to dynamically adjust the balance between exploration and utilization by optimizing the logarithmic standard deviation.

8. The hierarchical reinforcement learning based transportation hub parking lot charging station energy management method according to claim 7, characterized in that, The training process for constructing the hierarchical multi-agent deep reinforcement learning control architecture includes: Step 1, Network Initialization: Construct the upper-layer Actor / Critic network and the lower-layer 8 sets of Actor / Critic networks; Step 2, Experience Replay: Set upper model experience replay pool and lower model experience replay pool ; The lower-level experience replay pool buffer stores (state, action, reward, next_state) tuples at the single-vehicle granularity; Step 3, Update Strategy: Every fixed number of steps, sample batch data from the experience replay pool, calculate the temporal difference error to update the value network, and maximize the Q value and entropy to update the strategy network; Step 4, Automatic Entropy Adjustment: By optimizing the entropy temperature coefficient Parameters, dynamically adjust the degree of exploration, prevent the strategy from falling into local optimum too early.

9. The hierarchical reinforcement learning based energy management method for traffic hub parking lot charging stations according to claim 8, wherein, Step 3 includes the following: Step 31: The SAC algorithm adopts a reinforcement learning objective of maximizing entropy regularization, including a network architecture of an Actor network, two Critic networks, and an automatic entropy adjustment module. Target policy The expression for the target policy is: ； in, Represents the trajectory distribution under the strategy. As a discount factor, For the reward function, Here is the entropy regularization coefficient. For the entropy of the strategy, Indicate the target strategy, Let t represent the long-term expected goal, and t represent the current time. This represents the environmental state at the current time t. This represents the action performed by agent t at the current time. Step 32: Optimize the Q-network parameters by minimizing the soft Bellman residual, defining the loss function. for: ； wherein, is an experience replay buffer, is a target Q value, denotes a value estimation network; ； In the formula, denote the parameters of the target Critic network, are the Actor network parameters; Step 33, the Actor network parameters The update procedure of the Actor network parameters comprises using a reparameterization trick to transform the action distribution into a function of the state and a noise variable. ； wherein, subordinate to a standard normal distribution, denotes the policy network mean, denotes the policy network standard deviation, denotes the policy network; Optimize network parameters by minimizing KL divergence; Step 34, the entropy regularization coefficient An automatic adjustment process of the entropy regularization coefficient, including establishing an objective function of entropy automatic adjustment, whose loss function is defined as; ； By calculating With respect to the gradient and update, the adaptive adjustment of the temperature coefficient is realized.

Images