Charging station scheduling method based on hierarchical multi-target reinforcement learning

Abstract

The invention discloses a charging station scheduling method based on hierarchical multi-target reinforcement learning, and belongs to the technical field of smart power grids and artificial intelligence, and the method comprises the steps: constructing an electric vehicle charging event generation model for generating an electric vehicle charging event; a multi-time-scale multi-target double-layer reinforcement learning framework is constructed: an upper-layer model takes day as a time scale and takes maximization of long-term cumulative income of a charging station as a target to obtain a current range of a charging pile in each hour of the day; the lower-layer model determines the charging current of each charging pile in each minute on the day by taking minutes as a time scale and taking the current range of the charging pile in each hour on the day as a constraint condition; and generating a plurality of electric vehicle charging events by using the electric vehicle charging event generation model, training the multi-time-scale multi-target double-layer reinforcement learning framework, and applying the trained multi-time-scale multi-target double-layer reinforcement learning framework to actual working conditions to perform charging station scheduling. According to the invention, the problems of credit distribution delay and multi-target tradeoff are solved.

Description

Technical Field

[0001] This invention belongs to the field of smart grid and artificial intelligence technology, and in particular relates to a charging station scheduling method based on hierarchical multi-objective reinforcement learning. Background Technology

[0002] In recent years, the rapid growth of electric vehicles (EVs) has profoundly reshaped the energy consumption landscape, prompting the implementation of grid demand response to maintain grid stability during peak electricity demand periods. For traditional charging stations without V2G capabilities, operators can proactively reduce peak loads to improve operational efficiency, but this opportunity also brings significant operational challenges. Simultaneously, operational strategies must balance multiple operational objectives, including user charging satisfaction, equipment safety, and grid stability. This conflicting benefit transforms the charging station operation and scheduling problem into a complex, multi-timescale, multi-objective dynamic optimization problem.

[0003] Therefore, there is an urgent need for a new technical solution that can effectively decouple decision-making processes at different time scales and adaptively balance multiple conflicting operational objectives, thereby finding the optimal strategy for charging stations in complex dynamic environments. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a charging station scheduling method based on hierarchical multi-objective reinforcement learning, which solves the problems of credit allocation delay and multi-objective trade-offs.

[0005] To achieve the aforementioned objectives, the present invention employs the following technical solution: a charging station scheduling method based on hierarchical multi-objective reinforcement learning, comprising: Obtain a real dataset of electric vehicle charging data, train an adversarial network based on the real dataset, and build an electric vehicle charging event generation model for generating electric vehicle charging events. Constructing a multi-timescale, multi-objective, two-layer reinforcement learning framework: The upper-layer model of the multi-timescale, multi-objective, two-layer reinforcement learning framework adopts the soft actor-critic algorithm. Before the start of each day, the current range of the charging piles for each hour of the day is obtained with the day as the time scale and the goal of maximizing the long-term cumulative revenue of the charging station. The lower-level model of the multi-timescale multi-objective two-layer reinforcement learning framework adopts the TD3 deep reinforcement learning framework based on dynamic utility function, with minutes as the time scale and the current range of charging piles in each hour of the day as the constraint condition to determine the charging current of each charging pile in each minute of the day. A training set is obtained by generating several electric vehicle charging events using an electric vehicle charging event generation model. Based on the training set, a multi-timescale, multi-objective, two-layer reinforcement learning framework is trained. The trained multi-timescale, multi-objective, two-layer reinforcement learning framework is then used for charging station scheduling under actual working conditions.

[0006] Furthermore, the structural expression, loss function, and optimization objective of the electric vehicle charging event generation model are as follows:

[0007]

[0008]

[0009]

[0010]

[0011] in, The result of the discriminator's judgment; It is the sigmoid activation function; This is the weight matrix for the third-level linear transformation; Use the LeakyReLU activation function; This is the weight matrix for the second-level linear transformation; This is the weight matrix for the first-level linear transformation; Electric vehicle charging incident; This is the bias vector for the first-level linear transformation; This is the bias vector for the second-level linear transformation; This is the bias vector for the third-level linear transformation; For electric vehicle charging events generated by the generator based on noise; This is the third level of linear transformation; It is the ReLU activation function; This is the second level of linear transformation; This is the first level of linear transformation; For noise; The discriminator loss function; Generate sample batch size for the generator; Batch size of a real-world dataset for electric vehicle charging; For the first Sample labels for each input sample; For the first One electric vehicle charging event; Input the sample index to the discriminator; The generator loss function; For the first The noise corresponding to each generated sample; Value function; For the distribution of real-world datasets on electric vehicle charging P data Expectations; To distribute the generated electric vehicle charging information P z The expectation.

[0012] Furthermore, the expression for the electric vehicle charging event is:

[0013] in, For the first A charging incident involving an electric vehicle; For the first The model of the electric vehicle; For the first Battery level of the electric vehicle; For the first The arrival time of the electric vehicle; For the first The charging time for a single electric vehicle; For the first The maximum charging power of a vehicle; For the first The charging mode indicator for the electric vehicle includes both fast charging and slow charging.

[0014] Furthermore, the expressions for the state space, action space, and reward function of the upper-layer model are as follows:

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in, This represents the state space of the upper-level model; For the first The state of the sky; For the first The state of the sky; For total incentive; Incentives for electricity sales; Incentives for demand response; This represents the actual daily electricity consumption. Based on power consumption; This represents the normalized difference between power consumption and the reference power consumption. For time steps; It represents the number of minutes in a day, specifically 1440. for Time step The charging power of each charging station; Time scale; for The incentive coefficient for charging services at each time step; For charging station index; This represents the total number of charging stations; This is the demand response incentive coefficient; This provides the action space for the upper-level model. For the first The first day Actions during the hour; This is the minimum current for the charging station; This is the maximum current of the charging pile; This is the reward function for the upper-level model.

[0025] Furthermore, the state The input to the soft actor-critic algorithm is processed using a self-attention mechanism.

[0026] Furthermore, the expressions for the state space, action space, and penalty function of the lower-level model are as follows:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] in, This represents the state space of the lower-level model. for The global state at the time step; for Local state at a time step; This indicates the current time step's position within the day. , and The three-phase current of the power grid; for The total power of all charging stations at the time step; For the first The first day Actions during the hour; For the first The status of each charging station; This represents the total number of charging stations; The symbol represents whether there is a vehicle connection; 1 indicates a vehicle connection, and 0 indicates otherwise. This represents the current capacity of electric vehicle batteries; The capacity of the connected electric vehicles; The maximum charging power of the connected electric vehicle; The current allocated in the previous time step; This provides the action space for the lower-level model. For the first The charging current of each charging station; This is the reward function for the lower-level model; For service incentive weighting; Incentives for service; Weights for grid load stability incentives; To provide incentives for stable grid load; As the incentive weight for demand response execution; Incentives for the degree of performance in responding to demands; As a weight for equipment safety incentives; Incentives for equipment safety; The penalty value is the reward. for The number of charging stations that are charging at the time step; For the first One charging station The actual charging current allocated to each time step; The maximum physically permissible charging current for a single charging station; The weighting factor is the standard deviation. It exhibits local fluctuations; This is the weighting factor for the peak-to-average power ratio; For global peak performance; The standard deviation of the normalized power series; This is the sequence after min-max normalization of the power sequence; From the start of the round to Total charging power sequence at each time step; To calculate the average power; It is a non-zero constant; for Charging power at time step; for The upper-level model of the time step is the first The maximum current limit set for each charging station; for Standard deviation of three-phase current at time step; This represents the maximum phase difference. It is a non-zero constant; for Time step j Phase current of the phase; for The average value of the three-phase current at the time step; for Phase current of phase 1 at time step; for Phase current of phase 2 at time step; for Phase current of phase 3 at time step; Base penalty value; for The total power that the charging station draws from the power grid at each time step; The maximum power that a grid node is allowed to supply; This represents the average overload of a single pile. This represents the total number of charging stations.

[0043] Furthermore, calculations are performed based on the charging current of each charging station. Time step charging power of each charging station :

[0044]

[0045]

[0046]

[0047] in, for The original three-phase power of the electric vehicle at time step; for The percentage of electric vehicle charge at each time step; This represents the initial value of the electric vehicle's battery capacity. This represents the maximum capacity of the electric vehicle battery. Electric vehicles in slow charging mode The original single-phase power at the time step; Electric vehicles in fast charging mode The original three-phase power at the time step; Voltage; Time scale; for The actual charging power of electric vehicles at time step; This refers to the battery capacity of an electric vehicle.

[0048] Furthermore, the TD3 deep reinforcement learning framework based on dynamic utility functions specifically involves introducing a scalar utility value into the calculation of the target Q-value of the Q-network in the TD3 deep reinforcement learning framework:

[0049]

[0050] in, The target Q-value for the Q-network; It is a scalar utility value; Discount factor; This is a termination marker; For the target commentator network, when When =1, it is the first target commentator network, when When =2, it represents the second-target critic network; For the lower-level model The state at +1 time step; This refers to the lower-level actions after adding noise and cropping. It is a neural network with monotonic non-decreasing properties; For the lower-level model Rewards for time steps; Nonlinear preferences for different combinations of objectives in a neural network.

[0051] The beneficial effects of this invention are as follows: by decoupling decision-making across time scales and introducing dynamic preference learning, this invention effectively solves the problems of credit allocation delay and multi-objective trade-offs, and can significantly improve the overall efficiency of operators while taking into account grid stability and equipment safety. By decomposing the complex scheduling problem into upper-layer daily economic planning and lower-layer minute-level real-time control, the problem of credit allocation delay in single-layer models is effectively solved, significantly improving learning efficiency and the global optimality of the strategy. This makes the objectives of each agent layer clearer: the upper layer focuses on maximizing total revenue, while the lower layer focuses on executing macro strategies and optimizing real-time operating indicators. A multi-head self-attention mechanism is introduced into the upper-layer agent, which can dynamically evaluate the importance of information at different times in historical operating data, thereby extracting more insightful features from long-term time-series data. This makes the macro strategies formulated by the upper layer more forward-looking and robust, and better able to adapt to changes in the market environment such as electricity prices and subsidy policies. A dynamically parameterized utility function network is introduced into the lower layer, replacing the fixed, linear objective weighting method in traditional methods. This network can learn and express complex, non-linear dependencies and preferences between multiple objectives, enabling the lower-layer agent to make more intelligent and dynamic optimizations that better align with the overall interests of the operator when pursuing multiple conflicting objectives. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the specific number of vehicles in the training set in an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram illustrating the vehicle arrival time and charging dwell time in an embodiment of the present invention.

[0055] Figure 4 This is a schematic diagram of the overall excitation of the upper layer and the test results of the lower layer in an embodiment of the present invention.

[0056] Figure 5 This is a comparison chart of the training and testing results of the present invention and other methods in the embodiments of the present invention.

[0057] Figure 6 This is a schematic diagram of the lower-level feature parameters in an embodiment of the present invention. Detailed Implementation

[0058] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0059] like Figure 1 As shown, in one embodiment of the present invention, a charging station scheduling method based on hierarchical multi-objective reinforcement learning includes: Obtain a real dataset of electric vehicle charging data, train an adversarial network based on the real dataset, and build an electric vehicle charging event generation model for generating electric vehicle charging events. Constructing a multi-timescale, multi-objective, two-layer reinforcement learning framework: The upper-layer model of the multi-timescale, multi-objective, two-layer reinforcement learning framework adopts the soft actor-critic algorithm. Before the start of each day, the current range of the charging piles for each hour of the day is obtained with the day as the time scale and the goal of maximizing the long-term cumulative revenue of the charging station. The lower-level model of the multi-timescale multi-objective two-layer reinforcement learning framework adopts the TD3 deep reinforcement learning framework based on dynamic utility function, with minutes as the time scale and the current range of charging piles in each hour of the day as the constraint condition to determine the charging current of each charging pile in each minute of the day. A training set is obtained by generating several electric vehicle charging events using an electric vehicle charging event generation model. Based on the training set, a multi-timescale, multi-objective, two-layer reinforcement learning framework is trained. The trained multi-timescale, multi-objective, two-layer reinforcement learning framework is then used for charging station scheduling under actual working conditions.

[0060] In this embodiment, using data from a public dataset as an example, the 730 days of charging events are divided into a training set, a test set, and a validation set in a ratio of 8:1:1. The specific number of vehicles in the training set during training is also included. Figure 2 As shown, the vehicle arrival time and charging dwell time are as follows: Figure 3 As shown.

[0061] from Figure 2 and Figure 3 It can be seen that the number of electric vehicles using three-phase charging is significantly greater than that of small cars using single-phase charging, with the number of electric vehicles from the most numerous brands being three to four times greater than that from the least numerous brands. Furthermore, the majority of electric vehicles in the charging order data arrive between 4 PM and 10 PM, and the actual charging time for most electric vehicles is relatively short, generally less than one hour.

[0062] In this embodiment, a two-layer reinforcement learning method with upper and lower layer coupling is adopted. The upper-layer policy management system pursues the maximization of operator operation efficiency, while the lower-layer charging management system takes into account multiple objectives and adaptively learns the optimal charging strategies for multiple electric vehicles.

[0063] The structural expression, loss function, and optimization objective of the electric vehicle charging event generation model are as follows:

[0064]

[0065]

[0066]

[0067]

[0068] in, The result of the discriminator's judgment; It is the sigmoid activation function; This is the weight matrix for the third-level linear transformation; Use the LeakyReLU activation function; This is the weight matrix for the second-level linear transformation; This is the weight matrix for the first-level linear transformation; Electric vehicle charging incident; This is the bias vector for the first-level linear transformation; This is the bias vector for the second-level linear transformation; This is the bias vector for the third-level linear transformation; For electric vehicle charging events generated by the generator based on noise; This is the third level of linear transformation; It is the ReLU activation function; This is the second level of linear transformation; This is the first level of linear transformation; For noise; The discriminator loss function; Generate sample batch size for the generator; Batch size of a real-world dataset for electric vehicle charging; For the first Sample labels for each input sample; For the first One electric vehicle charging event; Input the sample index to the discriminator; The generator loss function; For the first The noise corresponding to each generated sample; Value function; For the distribution of real-world datasets on electric vehicle charging P data Expectations; To distribute the generated electric vehicle charging information P z The expectation.

[0069] In this embodiment, an adversarial network is used to train a discriminator and a generator. The discriminator aims to maximize its ability to correctly classify real and fake charging event samples, while the generator aims to minimize the discriminator's ability to detect fake charging event samples. The generated data is detected using the KL divergence method to measure the difference between the generated data distribution and the original data distribution, thereby ensuring data quality.

[0070] In this embodiment, the top ten car models sold in the location of the publicly available dataset were collected as training data for this invention. The basic data of these cars are shown in Table 1. Different electric vehicle models have different battery capacities and charging methods. In general, there are more models of large cars with three-phase charging than small cars with single-phase charging.

[0071] Table 1

[0072] The expression for the electric vehicle charging event is:

[0073] in, For the first A charging incident involving an electric vehicle; For the first The model of the electric vehicle; For the first Battery level of the electric vehicle; For the first The arrival time of the electric vehicle; For the first The charging time for a single electric vehicle; For the first The maximum charging power of a vehicle; For the first The charging mode indicator for the electric vehicle includes both fast charging and slow charging.

[0074] The expressions for the state space, action space, and reward function of the upper-layer model are as follows:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] in, This represents the state space of the upper-level model; For the first The state of the sky; For the first The state of the sky; For total incentive; Incentives for electricity sales; Incentives for demand response; This represents the actual daily electricity consumption. Based on power consumption; This represents the normalized difference between power consumption and the reference power consumption. For time steps; It represents the number of minutes in a day, specifically 1440. for Time step The charging power of each charging station; Time scale; for The incentive coefficient for charging services at each time step; For charging station index; This represents the total number of charging stations; This is the demand response incentive coefficient; This provides the action space for the upper-level model. For the first The first day Actions during the hour; This is the minimum current for the charging station; This is the maximum current of the charging pile; This is the reward function for the upper-level model.

[0085] The state The input to the soft actor-critic algorithm is processed using a self-attention mechanism.

[0086] The expressions for the state space, action space, and penalty function of the lower-level model are as follows:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] in, This represents the state space of the lower-level model. for The global state at the time step; for Local state at a time step; This indicates the current time step's position within the day. , and The three-phase current of the power grid; for The total power of all charging stations at the time step; For the first The first day Actions during the hour; For the first The status of each charging station; This represents the total number of charging stations; The symbol represents whether there is a vehicle connection; 1 indicates a vehicle connection, and 0 indicates otherwise. This represents the current capacity of electric vehicle batteries; The capacity of the connected electric vehicles; The maximum charging power of the connected electric vehicle; The current allocated in the previous time step; This provides the action space for the lower-level model. For the first The charging current of each charging station; This is the reward function for the lower-level model; For service incentive weighting; Incentives for service; Weights for grid load stability incentives; To provide incentives for stable grid load; As the incentive weight for demand response execution; Incentives for the degree of performance in responding to demands; As a weight for equipment safety incentives; Incentives for equipment safety; The penalty value is the reward. for The number of charging stations that are charging at the time step; For the first One charging station The actual charging current allocated to each time step; The maximum physically permissible charging current for a single charging station; The weighting factor is the standard deviation. It exhibits local fluctuations; This is the weighting factor for the peak-to-average power ratio; For global peak performance; The standard deviation of the normalized power series; This is the sequence after min-max normalization of the power sequence; From the start of the round to Total charging power sequence at each time step; To calculate the average power; It is a non-zero constant; for Charging power at time step; for The upper-level model of the time step is the first The maximum current limit set for each charging station; for Standard deviation of three-phase current at time step; This represents the maximum phase difference. It is a non-zero constant; for Time stepj Phase current of the phase; for The average value of the three-phase current at the time step; for Phase current of phase 1 at time step; for Phase current of phase 2 at time step; for Phase current of phase 3 at time step; Base penalty value; for The total power that the charging station draws from the power grid at each time step; The maximum power that a grid node is allowed to supply; This represents the average overload of a single pile. This represents the total number of charging stations.

[0103] Calculated based on the charging current of each charging station. Time step charging power of each charging station :

[0104]

[0105]

[0106]

[0107] in, for The original three-phase power of the electric vehicle at time step; for The percentage of electric vehicle charge at each time step; This represents the initial value of the electric vehicle's battery capacity. This represents the maximum capacity of the electric vehicle battery. Electric vehicles in slow charging mode The original single-phase power at the time step; Electric vehicles in fast charging mode The original three-phase power at the time step; Voltage; Time scale; for The actual charging power of electric vehicles at time step; This refers to the battery capacity of an electric vehicle.

[0108] The TD3 deep reinforcement learning framework based on dynamic utility functions specifically involves introducing a scalar utility value into the calculation of the target Q-value of the Q-network within the TD3 deep reinforcement learning framework.

[0109]

[0110] in, The target Q-value for the Q-network; It is a scalar utility value; Discount factor; This is a termination marker; For the target commentator network, when When =1, it is the first target commentator network, when When =2, it represents the second-target critic network; For the lower-level model The state at +1 time step; This refers to the lower-level actions after adding noise and cropping. It is a neural network with monotonic non-decreasing properties; For the lower-level model Rewards for time steps; Nonlinear preferences for different combinations of objectives in a neural network.

[0111] This embodiment simulates scenarios with different subsidy intensities. The test set includes 73 different dates under three different demand response incentives: 0.25, 0.5, and 0.75.

[0112] The Hamlet method's parameter settings are divided into upper and lower layers. The upper layer's multi-head self-attention mechanism's feature extractor processes the historical state sequence of the past 7 days, with an output dimension of 256. Both the policy network and the Q-value network employ multilayer perceptrons containing two hidden layers (256 neurons each). The SAC model is trained using the Adam optimizer with a learning rate of 3e-4 and a discount factor γ of 0.9. The experience replay buffer size is 730, and the batch size is 64. To promote policy exploration, the entropy coefficient is set to auto-adjust. After each training day, the agent performs a 32-gradient update step.

[0113] In the lower layers, both the Actor and Critic networks of TD3 consist of three hidden layers, each containing 960 neurons, using SiLU as the activation function. To improve training stability, layer normalization is integrated into the actor network. Both networks are updated using the Adam optimizer with a learning rate of 1e-4. To balance exploration and application, Ornstein-Uhlenbeck noise is introduced as action noise. A priority experience replay buffer with a capacity of 1,000,000 experiences is used, with 256 experiences sampled from the buffer for training each time, a discount factor of 0.99, a soft update coefficient of 0.005 for the target network, and a policy update frequency of 2.

[0114] An early stopping mechanism based on performance convergence is introduced. If the standard deviation of the daily total return of the upper-layer agent is less than 15.0 for 90 consecutive days, the model is considered to have converged sufficiently and training is terminated early. During training, the model performance is evaluated every 21,600 time steps (15 days) to monitor the learning progress and save the optimal model.

[0115] During the baseline data collection phase, a fixed exploratory macro strategy is adopted, which does not provide strategy guidance for the training of lower layers throughout the 24 hours of the day, maintaining the original state. At the end of the baseline day, the system will calculate and store the average daily performance indicators of this phase, namely the average total excitation and the average total power consumption of the baseline day.

[0116] The hierarchical concurrent learning phase follows a nested structure with "days" as the outer loop and "minutes" as the inner loop. At the start of a day, the upper-layer agent observes its current state, then samples a macroscopic action based on its current policy. Throughout the day, the system enters the lower-layer training loop, performing training at each time step. t The lower-level agent executes a complete "perception-decision-learning" cycle. The environment returns the state and incentives for the next moment and stores them in the lower-level agent's experience buffer; at the end of the day, the environment aggregates the performance data of the entire day and calculates the upper-level agent's performance on the [day / time] [period / time]. d The total incentives obtained for the day and the state for the next day are stored in the experience tuples in the experience buffer of the upper-level agent. In the policy learning phase of the upper-layer agent, the core task is to formulate the daily policy. The SAC algorithm was chosen as the basic framework for training the upper-layer policy, and a feature extractor based on a self-attention mechanism was designed to optimize the state representation method (specifically, processing state data based on a self-attention mechanism, with the processed data participating in SAC's action decision-making, value evaluation, policy optimization, and entropy adjustment processes) to enhance the ability to learn strategic knowledge from historical data. SAC encourages the agent to explore more broadly by introducing a regularization term that maximizes policy entropy into the objective function, thereby effectively preventing the policy from prematurely converging to a local optimum.

[0117] For multi-objective learning of the lower-level agent, a novel TD3 deep reinforcement learning framework based on dynamic utility functions with nonlinear representation capabilities is proposed. A method for parameterizing the utility function using neural networks is introduced, mapping a multi-objective incentive vector containing preferences to a scalar utility value representing overall satisfaction. This scalar utility value includes a neural network with monotonically non-decreasing properties and nonlinear preferences for different combinations of objectives within the neural network (i.e., mapping multiple rewards to a single overall satisfaction scalar through the neural network, with nonlinear preferences equivalent to the weights of each incentive). For the lower-level agent, service incentives and demand response performance incentives are considered more important than grid load stability incentives and equipment safety incentives. The preferences among these objectives are used to construct a set of expert utility functions with the same general trend but different details. The improved TD3 significantly enhances the stability of the learning process.

[0118] To verify whether the method proposed in this invention (referred to as Hmecs in the figure) can effectively address the core challenge of fluctuating demand response incentives, three different demand response incentive scenarios in the test set are used to simulate different subsidy intensities. The total incentive situation obtained from the upper layer is as follows: Figure 4 (a) Figure 4 (b) Figure 4 As shown in (c).

[0119] By comparison, it can be seen that the method proposed in this invention can respond correctly to different demand response stimuli. When the demand response stimuli are high, for example... Figure 4 (c) When the demand response incentive is 0.75 per kilowatt-hour, the agent obtains more demand response subsidies by actively reducing load and power consumption. At this point, the total daily incentive value reaches 734.70, with the highest being 749.44 on the second day. When the demand response incentive is lower, for example... Figure 4(a) When the demand response incentive is 0.25 per kilowatt-hour, the agent focuses on maximizing the power consumption of electric vehicle charging. At this time, the total incentive value per day is 299.60, with the highest being 325.14 on the 18th day. The above results also show that the strategy of the proposed method is relatively stable and does not fluctuate drastically with different test dates.

[0120] The test results related to the lower layer are as follows Figure 4 (d), Figure 4 (e), Figure 4 As shown in (f), where, Figure 4 (d) is the load variation index (LVI), which measures the smoothness of the charging load throughout the day. It is calculated by summing the absolute values ​​of the power differences between consecutive time steps, dividing by the average power over the entire period, and then normalizing. A smaller LVI value indicates a smoother load curve and a relatively smaller rate of power variation. It can be seen that when the demand response stimulus is high, the LVI value is actually higher and more volatile. This is because the power is lower at this time, and fluctuations of the same magnitude will cause an increase in volatility, and the higher the demand response value, the more pronounced this volatility. Figure 4 (e) is the average SOC of electric vehicles. It can be seen that the lower the demand response incentive, the higher the average SOC of electric vehicles, and the lower the average degree of demand response. Figure 4 (f) shows the daily average of the three-phase current at a demand response stimulus of 0.5 kWh. It can be observed that the fluctuations between the three phases are not significant, which fully ensures the safety of the charging station equipment and achieves balanced three-phase load, demonstrating the effectiveness of the proposed method. This verifies that the lower-level multi-objective framework of the proposed method can effectively balance multiple operational constraints such as grid friendliness and equipment safety, resolving the contradictions between multiple objectives.

[0121] To verify the superiority of the proposed hierarchical architecture and its components compared to other hierarchical methods, the performance of the proposed method was compared with several widely used benchmark methods. The upper layers utilize the classic single-objective reinforcement learning algorithms SAC, Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG). These three classic reinforcement learning algorithms each have their own focus: First, DQN innovatively solves the decision-making problem in high-dimensional, discrete action spaces by combining deep learning with Q-learning and utilizing experience replay and the target network for stable training; second, DDPG, as an Actor-Critic algorithm, extends the ideas of DQN to continuous action spaces, directly outputting precise continuous actions using a deterministic policy; and SAC, an Off-Policy Actor-Critic algorithm, greatly encourages exploration by maximizing policy entropy in the objective function, thus exhibiting excellent stability and sample efficiency in complex continuous control tasks. The lower layer uses the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm for comparison. The core of MADDPG is to have the commentator network predict an independent value for each conflicting objective. This allows the algorithm to learn a single, flexible policy, making trade-offs between various task requirements simply by adjusting the preference weights for different objectives during execution, without retraining. The results are as follows... Figure 5 (a), Figure 5 As shown in (b).

[0122] The comparison results of the upper-layer training of the method proposed in this invention are as follows: Figure 5 As shown in (a), the proposed method has the highest total stimulus during training, outperforming the other three methods. Its convergence speed is second only to DDPG+MADDPG, while DDPG+MADDPG converges the fastest, exhibiting a small fluctuation after 150 rounds before rapidly converging. Furthermore, SAC+MADDPG shows the worst training performance, converging only after 766 rounds, while DQN+MADDPG converges the slowest, only reaching convergence after 805 rounds. This is primarily because DQN is a value-based learning method, whose convergence speed is slower than the Actor-Critic-based methods of SAC and DDPG. Figure 5(b) shows the performance results of the four methods on the test set, and the test results correspond to the training results. It can be seen that the method proposed in this invention has the best test performance, with an average of 476.86 activations per day. Although the DQN+MADDPG method converges the slowest, it still achieves 426.61 activations per day on the test set. The DDPG+MADDPG-based method has an average of 362.15 activations per day on the test set. The worst performing method on the test set is SAC+MADDPG, with an average daily activation of only 329.38. This may be due to the mismatch in state representation and feature extraction in the original SAC method, which the method proposed in this invention addresses.

[0123] Some lower-level feature parameters, such as Figure 6 As shown, Figure 6 (a) shows the results of Daily energy consumption. The comparison shows that the method proposed in this invention has the lowest average daily energy consumption, while SAC+MADDPG has the highest average daily energy consumption. This indicates that the method proposed in this invention has a more accurate understanding that under this demand response incentive, more participation in demand response can achieve greater benefits. Figure 6 (b) The results of the Peak-to-Average Power Ratio are shown. The ratio is relatively large in the method proposed in this invention, which may be due to its relatively small Daily energy consumption, making it more sensitive to changes in peak values. However, this does not affect the operation of the charging station. Overall, the method proposed in this invention achieves a better balance between obtaining greater long-term benefits and maintaining stable real-time operation.

Claims

1. A charging station scheduling method based on hierarchical multi-objective reinforcement learning, characterized in that, include: Obtain a real dataset of electric vehicle charging data, train an adversarial network based on the real dataset, and build an electric vehicle charging event generation model for generating electric vehicle charging events. Constructing a multi-timescale, multi-objective, two-layer reinforcement learning framework: The upper-layer model of the multi-timescale, multi-objective, two-layer reinforcement learning framework adopts the soft actor-critic algorithm. Before the start of each day, the current range of the charging piles for each hour of the day is obtained with the day as the time scale and the goal of maximizing the long-term cumulative revenue of the charging station. The lower-level model of the multi-timescale multi-objective two-layer reinforcement learning framework adopts the TD3 deep reinforcement learning framework based on dynamic utility function, with minutes as the time scale and the current range of charging piles in each hour of the day as the constraint condition to determine the charging current of each charging pile in each minute of the day. A training set is obtained by generating several electric vehicle charging events using an electric vehicle charging event generation model. Based on the training set, a multi-timescale, multi-objective, two-layer reinforcement learning framework is trained. The trained multi-timescale, multi-objective, two-layer reinforcement learning framework is then used for charging station scheduling under actual working conditions.

2. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 1, characterized in that, The structural expression, loss function, and optimization objective of the electric vehicle charging event generation model are as follows: in, The result of the discriminator's judgment; It is the sigmoid activation function; This is the weight matrix for the third-level linear transformation; Use the LeakyReLU activation function; This is the weight matrix for the second-level linear transformation; This is the weight matrix for the first-level linear transformation; Electric vehicle charging incident; This is the bias vector for the first-level linear transformation; This is the bias vector for the second-level linear transformation; This is the bias vector for the third-level linear transformation; For electric vehicle charging events generated by the generator based on noise; This is the third level of linear transformation; It is the ReLU activation function; This is the second level of linear transformation; This is the first level of linear transformation; For noise; The discriminator loss function; Generate sample batch size for the generator; Batch size of a real-world dataset for electric vehicle charging; For the first Sample labels for each input sample; For the first One electric vehicle charging event; Input the sample index to the discriminator; The generator loss function; For the first The noise corresponding to each generated sample; It is a value function; For the distribution of real-world datasets on electric vehicle charging P data Expectations; To distribute the generated electric vehicle charging information P z The expectation.

3. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 1, characterized in that, The expression for the electric vehicle charging event is: in, For the first A charging incident involving an electric vehicle; For the first The model of the electric vehicle; For the first Battery level of the electric vehicle; For the first The arrival time of the electric vehicle; For the first The charging time for a single electric vehicle; For the first The maximum charging power of a vehicle; For the first The charging mode indicator for the electric vehicle includes both fast charging and slow charging.

4. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 1, characterized in that, The expressions for the state space, action space, and reward function of the upper-layer model are as follows: in, This represents the state space of the upper-level model; For the first The state of the sky; For the first The state of the sky; For total incentive; Incentives for electricity sales; Incentives for demand response; This represents the actual daily electricity consumption. Based on power consumption; This represents the normalized difference between power consumption and the reference power consumption. For time steps; It represents the number of minutes in a day, specifically 1440. for Time step The charging power of each charging station; Time scale; for The incentive coefficient for charging services at each time step; For charging station index; This represents the total number of charging stations; This is the demand response incentive coefficient; This provides the action space for the upper-level model. For the first The first day Actions during the hour; This is the minimum current for the charging station; This is the maximum current of the charging pile; This is the reward function for the upper-level model.

5. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 4, characterized in that, The state The input to the soft actor-critic algorithm is processed using a self-attention mechanism.

6. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 1, characterized in that, The expressions for the state space, action space, and penalty function of the lower-level model are as follows: in, This represents the state space of the lower-level model. for The global state at the time step; for Local state at a time step; This indicates the current time step's position within the day. , and The three-phase current of the power grid; for The total power of all charging stations at the time step; For the first The first day Actions during the hour; For the first The status of each charging station; This represents the total number of charging stations; The symbol represents whether there is a vehicle connection; 1 indicates a vehicle connection, and 0 indicates otherwise. This represents the current capacity of electric vehicle batteries; The capacity of the connected electric vehicles; The maximum charging power of the connected electric vehicle; The current allocated in the previous time step; This provides the action space for the lower-level model. For the first The charging current of each charging station; This is the reward function for the lower-level model; For service incentive weighting; Incentives for service; Weights for grid load stability incentives; To provide incentives for stable grid load; As the incentive weight for demand response execution; Incentives for the degree of performance in responding to demands; As a weight for equipment safety incentives; Incentives for equipment safety; The penalty value is the reward. for The number of charging stations that are charging at the time step; For the first One charging station The actual charging current allocated to each time step; The maximum physically permissible charging current for a single charging station; The weighting factor is the standard deviation. It exhibits local fluctuations; This is the weighting factor for the peak-to-average power ratio; For global peak performance; The standard deviation of the normalized power series; This is the sequence after min-max normalization of the power sequence; From the start of the round to Total charging power sequence at each time step; To calculate the average power; It is a non-zero constant; for Charging power at time step; for The upper-level model of the time step is the first The maximum current limit set for each charging station; for Standard deviation of three-phase current at time step; This represents the maximum phase difference. It is a non-zero constant; for Time step j Phase current of the phase; for The average value of the three-phase current at the time step; for Phase current of phase 1 at time step; for Phase current of phase 2 at time step; for Phase current of phase 3 at time step; Base penalty value; for The total power that the charging station draws from the power grid at each time step; The maximum power that a grid node is allowed to supply; This represents the average overload of a single pile. This represents the total number of charging stations.

7. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 6, characterized in that, Calculated based on the charging current of each charging station. charging power of each charging station in time step : in, for The original three-phase power of the electric vehicle at time step; for The percentage of electric vehicle charge at each time step; This represents the initial value of the electric vehicle's battery capacity. This represents the maximum capacity of the electric vehicle battery. Electric vehicles in slow charging mode The original single-phase power at the time step; Electric vehicles in fast charging mode The original three-phase power at the time step; Voltage; Time scale; for The actual charging power of electric vehicles at time step; This refers to the battery capacity of an electric vehicle.

8. The charging station scheduling method based on hierarchical multi-objective reinforcement learning according to claim 1, characterized in that, The TD3 deep reinforcement learning framework based on dynamic utility functions specifically involves introducing a scalar utility value into the calculation of the target Q-value of the Q-network within the TD3 deep reinforcement learning framework. in, The target Q-value for the Q-network; It is a scalar utility value; Discount factor; This is a termination marker; For the target commentator network, when When =1, it is the first target commentator network, when When =2, it represents the second-target critic network; For the lower-level model The state at +1 time step; This refers to the lower-level actions after adding noise and cropping. It is a neural network with monotonic non-decreasing properties; For the lower-level model Rewards for time steps; Nonlinear preferences for different combinations of objectives in a neural network.

Images