A method for optimizing electric vehicle charging scheduling based on a stakberg game

Abstract

The application discloses a kind of electric vehicle charging scheduling optimization methods based on Stackelberg game, comprising: the system model of photovoltaic energy storage charging station is constructed, and energy management strategy is defined;The interactive relationship between photovoltaic energy storage charging station and multiple electric vehicles is modeled as Stackelberg Markov game;For leader agent and each follower agent, construct agent network;Each time step is by leader agent first to make charging price and energy storage scheduling strategy and broadcast, and each follower agent decides the charging power of itself in parallel accordingly;After all agents execute action, experience data is stored in local experience replay pool;Each agent samples experience data to update agent network;Until the strategy of all agents converges, the optimal charging station dynamic charging price and energy storage scheduling strategy and the optimal charging strategy of electric vehicle are obtained, the maximization of photovoltaic energy storage charging station operating income is realized.The application can realize the maximization of photovoltaic energy storage charging station operating income.

Description

Technical Field

[0001] This invention relates to the field of photovoltaic-storage charging station technology, and in particular to an electric vehicle charging scheduling optimization method based on Stackelberg game theory. Background Technology

[0002] Electric vehicles (EVs), as a significant alternative to traditional gasoline-powered vehicles, are experiencing unprecedented market penetration. However, the uncontrolled grid connection of numerous EVs, especially the concentrated charging during peak hours, poses a serious challenge to the stable operation of the power grid, potentially leading to issues such as localized voltage exceedances and transformer overloads. To address this challenge, smart charging stations equipped with photovoltaic (PV) and energy storage systems (ESS) are considered key infrastructure for improving grid flexibility and promoting the local consumption of renewable energy. Through EV charging scheduling, these charging stations can not only reduce their dependence on the main grid and alleviate the grid load caused by EV charging scheduling, but also participate in the energy market as independent economic entities, achieving operational profitability through low-charge, high-discharge methods of energy storage systems and meeting EV charging demands. Therefore, designing an efficient EV charging scheduling strategy for PV energy storage charging stations to maximize their operational revenue has become a hot topic of common concern in academia and industry.

[0003] Traditional charging scheduling strategies typically rely on precise mathematical modeling methods, such as linear programming and mixed-integer programming. Ideally, these model-driven methods can solve for and find the theoretically global optimum, providing a theoretical upper limit to the system's performance. However, these methods require highly accurate modeling of the complex real world. This strong dependence on precise models makes it difficult to accurately model complex, time-varying real-world environments with high uncertainty and dynamic changes, such as photovoltaic power output and electric vehicle user behavior, thus hindering robust optimal decision-making.

[0004] To overcome the aforementioned limitations, Deep Reinforcement Learning (DRL), which does not require a pre-built accurate environment model, has shown great potential. However, early DRL algorithms, such as DDPG, often suffer from instability in the learning process when dealing with continuous action space problems due to overestimation of the value function. To address this, researchers proposed the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm to stabilize the training process. However, the single-agent framework cannot characterize the dynamic game relationship between multiple independent decision-makers in a system, such as the relationship between a charging station and various EV users, and it lacks scalability due to the rapid expansion of the state-action space. Therefore, using Multi-Agent Reinforcement Learning (MARL) to model this problem is a more natural and effective approach. However, when solving problems involving simultaneous decision-making by multiple agents, the non-stationarity of the environment caused by the continuous changes in the policies of other agents becomes a challenge. Therefore, some studies have introduced the Centralized Training with Decentralized Execution (CTDE) framework. Its core idea is to introduce a central Critic network with access to global information during the training phase to guide and stabilize the learning process. However, the CTDE framework has not completely escaped the constraints of centralization. During the training phase, each agent still needs to upload its own private observation data to the central node, which can lead to communication bottlenecks and privacy protection issues in large-scale systems.

[0005] To address the aforementioned challenges, the Distributed Training with Distributed Execution (DTDE) framework is considered an ideal solution. In DTDE, all agents train and execute independently locally, interacting only by exchanging necessary action information, thus fundamentally solving privacy, communication, and scalability issues. However, most existing DTDE algorithms, such as independent learners, typically assume an equal and simultaneous decision-making relationship between agents. This model suffers severe distortion when describing the asymmetric, hierarchical relationship between charging service providers and a large number of consumers, potentially leading to learned policies that are ineffective in the real world. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose an electric vehicle charging scheduling optimization method based on Stackelberg game theory, which maximizes the operating revenue of photovoltaic energy storage charging stations by optimizing electric vehicle charging scheduling strategies and dynamic pricing of charging stations.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows:

[0008] This invention provides an electric vehicle charging scheduling optimization method based on Stackelberg game theory, comprising the following steps:

[0009] Step 1: Construct a system model of the photovoltaic-storage charging station and define the energy management strategy within the system model to enable interaction between the photovoltaic-storage charging station and multiple electric vehicles;

[0010] Step 2: Model the interaction between the photovoltaic energy storage charging station and multiple electric vehicles as a Stackelberg Markov game, with the photovoltaic energy storage charging station as the leader agent and each electric vehicle as a follower agent.

[0011] Step 3: Construct independent agent networks for the leader agent and each follower agent. Each agent network includes a main Actor network, two main Critic networks, and a corresponding target network.

[0012] Step 4: During the training process, at each time step, the leader agent first formulates the charging price and energy storage scheduling strategy and broadcasts it. Then, each follower agent makes its own charging power decision in parallel based on this.

[0013] Step 5: After all agents perform actions, they store their respective interaction experience data in their independent local experience replay pools. Step 6: Each agent samples experience data from its local experience replay pool based on the priority of the temporal differential error, and uses the sampled experience data to update the main Critic network, the main Actor network, and the target network.

[0014] Step 7: Repeat steps 4 to 6 until the strategies of all agents converge, obtain the optimal dynamic charging price and energy storage scheduling strategy for charging stations and the optimal charging strategy for electric vehicles, and maximize the operating revenue of photovoltaic-storage charging stations.

[0015] Furthermore, step 1 specifically includes:

[0016] Step 11: Construct a system model for the photovoltaic-energy storage charging station. The system model includes an external power grid, the photovoltaic power generation system, an energy storage system, charging piles, and electric vehicles; specifically:

[0017] In terms of energy flow, the external power grid and photovoltaic power generation system are connected to the energy storage system, which supplies power to electric vehicles connected to the charging pile. In terms of information flow, the photovoltaic-energy storage charging station establishes communication connections with the external power grid, photovoltaic power generation system, energy storage system and electric vehicles respectively, to collect operating status information and issue control commands, thereby realizing centralized management of energy flow within the system model.

[0018] Step 12: Define the energy flow priority within the system model as follows: Prioritize using electricity generated by the photovoltaic power generation system to meet the charging needs of electric vehicles; store excess electricity in the energy storage system; supplement any shortfall using electricity from the energy storage system or by purchasing electricity from the external grid; specifically including:

[0019] (1) In photovoltaic power generation systems, At any given moment, the actual photovoltaic power consumed by the photovoltaic-storage charging station Depends on the output power of the photovoltaic power generation system The real-time electricity demand within the photovoltaic-storage charging station is allocated according to the following logic: the photovoltaic power generation system will prioritize meeting the charging needs of electric vehicles; if there is still a surplus, it will be used to charge the energy storage system. This relationship is expressed as follows:

[0020]

[0021] in, It is a photovoltaic power generation system in Output power at any moment i An index for electric vehicles. M This represents the total number of electric vehicles. Indicates the first i The charging power of an electric vehicle at time t. Are all electric vehicles in Total charging power at any given time It is an energy storage system in The charging power at any given time, when the energy storage system is charging. >0, otherwise =0;

[0022] (2) In energy storage systems, this includes energy balance and state update as well as battery degradation costs;

[0023] 1) Energy balance and state renewal:

[0024] The state of charge (SOC) of the energy storage system at each time step The internal changes are determined by the charging and discharging power of the energy storage system. Decide, State of charge at time t By energy storage system State of charge at time t Updated to:

[0025]

[0026] in, It is an energy storage system in The charging and discharging power at any given time It is the total capacity of the energy storage system, in units of... , It is the charge / discharge efficiency coefficient, and when charging, it is the charging efficiency. The discharge efficiency is the efficiency during discharge. ;

[0027] at the same time, Its operation must meet the upper and lower limit constraints:

[0028]

[0029] in, This represents the state of charge of the energy storage system at time t. The lower bound of the constraint, This represents the state of charge of the energy storage system at time t. The upper limit of the constraint condition;

[0030] 2) Battery degradation costs:

[0031] Quantify the losses caused by each charge and discharge cycle of the energy storage system into degradation costs. A linear model is used, assuming that it is proportional to the energy throughput of the energy storage system:

[0032]

[0033] in, It is the degradation cost coefficient of the energy storage system, representing the economic loss caused by each kilowatt-hour of charging and discharging.

[0034] Step 13: Transform the energy scheduling problem into a constrained objective function optimization problem. The objective function includes the total revenue of the photovoltaic-storage-charging station and the total cost to electric vehicle users, and considers the penalty term for electric vehicle user dissatisfaction.

[0035] The expression is as follows:

[0036]

[0037] in, Let represent the objective function of the photovoltaic-storage charging station; where, Let T represent the total revenue of the photovoltaic-storage-charging station, and let T represent the set of time periods. This represents the charging price offered to electric vehicles by the photovoltaic-storage charging station at time t. This represents the electricity price of the external power grid at time t. This represents the total power purchased from the external power grid at time t; It is a penalty item for electric vehicle user dissatisfaction. It is a penalty function that applies when the electric vehicle is at its expected departure time. The desired state of charge was not achieved. At that time, the penalty function will give a negative reward. It is the coefficient of dissatisfaction among electric vehicle users;

[0038] Let the objective function of electric vehicle users be denoted as , where This represents the total cost for electric vehicle users;

[0039] express The lower limit, express The upper limit, Indicates that the energy storage system is in Charge and discharge power at any time The lower limit, Indicates that the energy storage system is in Charge and discharge power at any time The upper limit.

[0040] Furthermore, in step 2, the Stackelberg Markov game is formally defined as a tuple. Its specific components are as follows:

[0041] (1) The set of participants in the game is ,in, Representing the sole leader intelligent agent, namely the photovoltaic energy storage charging station; Representing the A follower intelligent agent, namely an electric vehicle; , i Represents the index of the follower agent and i It is a positive integer. M This represents the total number of follower agents;

[0042] (2) The decision-making order S is the leader agent. First, take action based on its state. From its action space Select and publish its actions Then all the follower intelligent agents After observing the actions of the leader agent, simultaneously and independently, based on their respective states... From its action space Choose the optimal action for each. ;

[0043] (3) In action space A, the leader agent Action space It is a compact set consisting of charging price and the charging and discharging power of the energy storage system; the action space of each follower agent. It is a compact set consisting of charging power;

[0044] (4) In the utility function U, the optimization objective is to maximize the operational revenue of the leader agent and minimize the charging cost of the follower agents, that is, to maximize the utility function of the leader agent and the utility function of the follower agents; specifically as follows:

[0045] The utility function of each follower agent is: The optimization problem is:

[0046] ;

[0047] The utility function of the leader agent is Each follower intelligent agent i action Every optimization problem has at least one optimal solution. Therefore, the utility function The optimization problem is:

[0048]

[0049] in, Represents each follower agent i action The corresponding optimal solution, It is to follow

[0050] Intelligent agent The optimal set of reactions is defined as:

[0051] ;

[0052] in, For any Both are non-empty sets;

[0053] (5) T represents the set of times, where t ∈ T for each decision stage.

[0054] Furthermore, in step 2, the photovoltaic-storage-charging station acts as the leader intelligent agent, and each electric vehicle acts as a follower intelligent agent; specifically including:

[0055] (1) The photovoltaic-storage charging station is taken as the leader intelligent agent, and the leader intelligent agent is modeled as follows:

[0056] Status: In At any given moment, the state of the leader agent Defined as:

[0057]

[0058] in, It refers to the current moment; Is it the external power grid? t Electricity price at any given time; It is a photovoltaic power generation system in Output power at any given moment; It is an energy storage system in State of charge at time t, It is all electric vehicles at the previous moment Total charging power;

[0059] Action: The action at time t is defined as follows:

[0060]

[0061] in, It is the charging price that the photovoltaic and energy storage charging station gives to electric vehicles at time t; For energy storage systems in The charging and discharging power at any given moment;

[0062] Rewards: Rewards for the leader agent The calculation method is total revenue minus total cost, as shown in the following formula:

[0063]

[0064]

[0065] in, This represents the total charge level of all electric vehicles. Indicates in Electricity purchased from the external power grid at all times; This represents the degradation cost coefficient of the energy storage system; This indicates the degradation cost of the energy storage system;

[0066] (2) Each electric vehicle is treated as a follower agent, and the follower agent is modeled as follows:

[0067] State: Follower agent Its decision-making is based on its own state and the charging price information received from the leader agent. state of time Defined as:

[0068]

[0069] in, It is the current moment. It is an electric car i The state of charge of the battery at time t. It is an electric car i The target battery charge state; It is an electric car i Expected departure time;

[0070] Action: Follower agent Actions at time t That is, the first i The charging power of an electric vehicle at time t, this action Limited by the maximum charging power of the electric vehicle itself;

[0071]

[0072] in, Indicates electric vehicles The upper limit of charging power;

[0073] Reward: Follower agent reward function Designed as a utility function consisting of negative cost terms, the goal is to maximize this utility function:

[0074] .

[0075] Furthermore, in step 3, each agent network includes a main Actor network, two main Critic networks, and a corresponding target network. Specifically, the main Actor network is used to output actions based on the current state, taking the agent's local state as input and outputting its actions; the two main Critic networks are used to evaluate the Q-value of the state-action pair, taking the state and action as input and outputting the evaluated Q-value; the target network includes a target Actor network and two target Critic networks, whose network structures are the same as the main Actor network and the main Critic network, respectively.

[0076] Furthermore, step 4 specifically includes:

[0077] Step 41: At each time step, the leader agent outputs the charging price and energy storage system charging and discharging power as actions through its main Actor network based on its local state information, and broadcasts the charging price to all follower agents.

[0078] Step 42: After observing the charging price, each follower agent outputs charging power as an action in parallel and independently through its own main Actor network, based on its local state information.

[0079] Step 5 specifically includes:

[0080] Step 51: Configure an independent local experience replay pool for each agent;

[0081] Step 52: After all agents perform actions, they store their respective interaction experience data in the corresponding local experience replay pool. The experience data includes tuples of state, action, reward, and next state.

[0082] Furthermore, in step 6, each agent samples experience data from the local experience replay pool based on the priority of the temporal difference error, specifically including:

[0083] (1) In priority experience playback, the magnitude of the time difference error is used as an indicator to measure the importance of an experience. The time difference error This represents the difference between the agent's current prediction of the value of a state-action pair and the estimate obtained from the environment. The formula for calculating this difference is:

[0084]

[0085] in, It is the agent's prediction of the value of the current state-action pair. It utilizes the current reward and the maximum Q value of the next state The calculated estimated value; Discount factor;

[0086] (2) For the first in the local experience replay pool j The experience, its priority Defined as:

[0087]

[0088] in, This indicates that the first j The empirical time-series difference error, the first j The absolute value of the empirical time-series difference error Used as the first j The initial priority of each experience, It is a constant that guarantees all experiences can be sampled;

[0089] (3) No. j Sampling probability of a piece of experience Defined as:

[0090] in, This is the total number of experience points in the local experience replay pool. It is a hyperparameter between 0 and 1, when When = 0, all empirical sampling probabilities are equal, and uniform random sampling is performed; when When =1, sampling is performed entirely according to priority;

[0091] (4) Calculate a sampling weight for each sampled experience. The sampling weight is then multiplied by the loss of that sample when calculating the loss function. The formula for calculating the sampling weight is:

[0092]

[0093] in, It is a hyperparameter used to correct for bias, when When =1, the deviation is completely eliminated.

[0094] Furthermore, step 6 updates the main Critic network using sampled empirical data; specifically, it includes:

[0095] (1) The target value is calculated by pruning double Q learning and target policy smoothing techniques; specifically, the smaller output value of the two target Critic networks is used as the basis for the target value, and bounded noise is added to the action output of the target Actor network during the calculation.

[0096] (2) The main Critic network is updated by minimizing the mean squared error loss between the predicted value and the target value of the main Critic network;

[0097] The specific update process of the main Critic network for both follower and leader agents is as follows:

[0098] 1) For the main Critic network update of the follower agent:

[0099] For any follower agent, its two main Critic networks Its input is only its own local state. and the actions one takes ;

[0100] Follower agent The k Loss function of the principal Critic network For target value Compared with the current main Critic network for a given state-action pair Predicted value The mean square error between them is expressed as:

[0101]

[0102] in, k Indicates the index of the main Critic network. Represents follower intelligent agents The k Parameters of the main Critic network, Represents follower intelligent agents The local experience replay pool Indicates the replay pool of local experience The expected value of an empirically derived random sample. The target value is represented as follows:

[0103]

[0104] in, For follower intelligent agents The reward function, Indicates the following intelligent agent The The Q-value function output by the target Critic network. As a discount factor, Representative follower intelligent agent In state The target Actor network, It is smooth noise added to the target action, and the noise distribution is as follows: The tilde (~) indicates compliance, and the clip (.) function represents the clipping function. Represents a normal distribution. It is a random noise parameter;

[0105] 2) Update of the leader agent's main Critic network:

[0106] Leader agent Two main Critic networks The input needs to include its own local state. One's own actions And the actions of all the follower agents it can observe. Leader agent The k Loss function of the principal Critic network Defined as:

[0107]

[0108] in, Represents the leader intelligent agent The k Parameters of the main Critic network, Indicates the expected value. Represents the leader intelligent agent The k Individual Critic Network; Indicates the target value;

[0109] Target value The calculation method is as follows:

[0110]

[0111] in, Represents the leader intelligent agent The k A target Critic network, Represents the leader intelligent agent The next state, Represents the leader intelligent agent The next move, This indicates the actions of all follower agents in the next moment.

[0112] Furthermore, step 6 involves updating the main Actor network using sampled empirical data; specifically, this includes:

[0113] (1) Set the update frequency of the main Actor network to be lower than that of the main Critic network;

[0114] (2) The main Actor network is updated by the policy gradient ascent method, wherein the policy gradient is the product of the gradient of the main Critic network output with respect to the action and the gradient of the main Actor network output with respect to the parameter.

[0115] The specific update process of the main Actor network for both follower and leader agents is as follows:

[0116] 1) For updates to the main Actor network of follower agents:

[0117] After introducing Gaussian noise, for the follower agent Its main Actor network generates actions using the following formula. :

[0118]

[0119] in, It is a follower intelligent agent exist Gaussian noise at time t, It is a follower intelligent agent The main Actor network, whose parameters are: It updates by maximizing the output of its main Critic network; the update method involves adjusting parameters along the direction of the policy gradient. It can be expressed by the following formula:

[0120]

[0121] in, Describe the objective function Parameters of the main Actor network policy gradient, Indicates the replay pool of local experience Mid-sampled state Expected value; Represents follower intelligent agents i The main Critic network, Represents follower intelligent agents i The output of the main Critic network is relative to the input action. The gradient; Represents follower intelligent agents i The main Actor network, This indicates that the output of the main Actor network is relative to the parameters. The gradient;

[0122] 2) Update of the leader agent's main actor network:

[0123] Leader agent Main Actor Network Its parameters are Its policy gradient Defined as:

[0124]

[0125] in, Describe the objective function Parameters of the main Actor network policy gradient, Indicates the expected value. The main Critic network represents the leader agent. The output of the main Critic network represents the leader agent's action relative to the leader agent's own action. The gradient; The main Actor network represents the leader agent. This represents the output of the main Actor network relative to the leader agent's own parameters. The gradient.

[0126] Furthermore, in step 6, the target network is updated using the sampled empirical data, and the update rule is as follows:

[0127]

[0128] in, The parameters representing the target network. Parameters representing the main network, The soft update coefficient is between 0 and 1, which allows the parameters of the target network to slowly track the parameters of the main network.

[0129] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0130] This invention considers the complex environmental factors in the photovoltaic-storage charging station scenario and employs a multi-agent deep reinforcement learning method. This eliminates the need for precise environmental modeling and offers good scalability. To overcome the limitations of a central processing unit (CPU) and the privacy issues associated with CPU reliance, a Stackelberg game framework is adopted. This master-slave game framework aligns with the hierarchical structure of the charging station, avoiding dependence on a CPU and the resulting communication and privacy problems. A privacy-preserving leader-follower game model is constructed: the interaction between the photovoltaic-storage charging station and electric vehicles is modeled as a Stackelberg Markov game. The photovoltaic-storage charging station, as the leader agent, sets the charging price, and the electric vehicles, as follower agents, respond to the price. This model more closely resembles real-world business scenarios, and its hierarchical decision structure protects user privacy because the leader does not need to access the followers' internal private information. A Stackelberg-based multi-agent twin delayed-deep deterministic policy gradient (SMATD3) algorithm is designed to solve the joint dynamic pricing and energy dispatch optimization problem in photovoltaic-storage charging stations. This method applies the stable learning mechanism of MATD3 to the distributed framework of Stackelberg games, enabling each agent to learn efficiently and autonomously. Furthermore, to further improve learning efficiency, a Prioritized Experience Replay (PER) pool is introduced, significantly improving data utilization efficiency and accelerating the algorithm's convergence speed. This invention adapts well to complex dynamic environments, exhibits good scalability, maximizes the operational revenue of photovoltaic-storage charging stations by optimizing electric vehicle charging scheduling strategies and dynamic pricing for charging stations, and can also achieve peak shaving and valley filling by utilizing energy storage systems to reduce grid load. Attached Figure Description

[0131] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0132] Figure 1 This is an execution flowchart of an electric vehicle charging scheduling optimization method based on Stackelberg game provided in an embodiment of the present invention.

[0133] Figure 2 This is a schematic diagram of the energy flow and information flow direction of the system model provided in the embodiment of the present invention.

[0134] Figure 3 This is a diagram of the SMATD3 algorithm framework provided in an embodiment of the present invention.

[0135] Figure 4 This is a flowchart of the policy update process for an agent network provided in an embodiment of the present invention. Detailed Implementation

[0136] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0137] Please see Figures 1-4 The present invention provides an electric vehicle charging scheduling optimization method based on Stackelberg game, comprising the following steps:

[0138] Step 1: Construct a system model of the photovoltaic energy storage charging station (PV energy storage charging station) and define the energy management strategy within the system model to realize the interaction between the PV energy storage charging station and multiple electric vehicles;

[0139] In this embodiment, step 1 specifically includes:

[0140] Step 11: Construct a system model of the photovoltaic-storage charging station. The system model includes an external power grid, a photovoltaic-storage charging station, a photovoltaic power generation system, an energy storage system, charging piles, and electric vehicles.

[0141] As shown in Figure 2, the system model centers on an electric vehicle charging station (EVCS) equipped with a photovoltaic power generation system and an energy storage system, clearly demonstrating its hierarchical interaction with the external power grid and electric vehicle users. Within the entire energy ecosystem, the EVCS interacts with the external power grid through unidirectional energy flow and bidirectional information flow. That is, the EVCS can purchase electricity from the external power grid but cannot sell electricity back to it. Furthermore, the EVCS can decide whether to purchase electricity from the external power grid based on real-time electricity price information published by the external power grid and the charging needs of electric vehicle (EV) users.

[0142] Specifically:

[0143] In terms of energy flow, the external power grid and photovoltaic power generation system are connected to the energy storage system, which supplies power to electric vehicles connected to the charging pile. In terms of information flow, the photovoltaic-energy storage charging station establishes communication connections with the external power grid, photovoltaic power generation system, energy storage system and electric vehicles respectively, to collect operating status information and issue control commands, thereby realizing centralized management of energy flow within the system model.

[0144] Step 12: Define the energy flow priority within the system model as follows: Prioritize using electricity generated by the photovoltaic power generation system to meet the charging needs of electric vehicles; store excess electricity in the energy storage system; and supplement any shortfall using electricity from the energy storage system or by purchasing electricity from the external grid. This means the following scenarios exist:

[0145] Scenario 1: When the electricity generated by the photovoltaic power generation system is greater than the electricity required for charging the electric vehicle, the electricity generated by the photovoltaic power generation system is used to power the electric vehicle through the energy storage system. After charging is completed, the remaining electricity in the photovoltaic power generation system is used to charge the energy storage system, and the excess electricity is stored in the energy storage system.

[0146] Scenario 2: When the electricity generated by the photovoltaic power generation system is equal to the electricity required for charging the electric vehicle, the electricity generated by the photovoltaic power generation system is used to power the electric vehicle through the energy storage system, and charging is completed;

[0147] Scenario 3: When the electricity generated by the photovoltaic power generation system is less than the electricity required for charging the electric vehicle, the electricity generated by the photovoltaic power generation system is first used to power the electric vehicle through the energy storage system. At this time, the charging is not fully completed. If you want to fully charge the vehicle, you can choose to use the electricity in the energy storage system or purchase electricity from the external grid to supplement the remaining electricity that needs to be charged, based on the purchase price of electricity from the external grid and the amount of electricity stored in the energy storage system.

[0148] Within a photovoltaic-storage charging station, energy dispatch and distribution are central. Its electricity primarily comes from real-time generation by the local photovoltaic power generation system and electricity purchased from the external grid. The energy storage system (ESS) plays a crucial energy buffering role. It can absorb surplus photovoltaic power from the local system and proactively purchase and store electricity during periods of low external grid electricity prices. When EV users have charging needs, the charging station will comprehensively dispatch the photovoltaic power generation system, the ESS, and decide whether to purchase electricity from the external grid to meet the charging load. Specifically, by utilizing pre-stored low-cost electricity to charge EVs during peak external grid load periods, the EVCS not only reduces its own operating costs but also achieves peak shaving and valley filling of the external grid load, thus avoiding exacerbating the load pressure on the external grid. Regarding energy dispatch priorities, the EVCS will prioritize the use of electricity generated by the photovoltaic power generation system, storing excess electricity in the ESS to minimize curtailment and promote local consumption of renewable energy. Secondly, it will comprehensively consider the current external grid load and electricity prices to decide whether to use electricity stored in the ESS or purchase electricity from the external grid to meet charging needs.

[0149] Specifically, it includes:

[0150] (1) In photovoltaic power generation systems, photovoltaic-storage charging stations will prioritize the use of clean energy generated by local photovoltaic power generation systems to maximize the utilization rate of renewable energy and reduce operating costs. At any given moment, the actual photovoltaic power consumed by the photovoltaic-storage charging station Depends on the output power of the photovoltaic power generation system The real-time electricity demand within the photovoltaic-storage charging station is allocated according to the following logic: the photovoltaic power generation system will prioritize meeting the charging needs of electric vehicles; if there is still a surplus, it will be used to charge the energy storage system. This relationship is expressed as follows:

[0151]

[0152] in, It is a photovoltaic power generation system in Output power at any moment i An index for electric vehicles. M This represents the total number of electric vehicles. Indicates the first i The charging power of an electric vehicle at time t. Are all electric vehicles in Total charging power at any given time It is an energy storage system in The charging power at any given time, when the energy storage system is charging. >0, otherwise =0;

[0153] (2) In energy storage systems, this includes energy balance and state update as well as battery degradation costs;

[0154] 1) Energy balance and state renewal:

[0155] The State of Charge (SOC) of an energy storage system at each time step The internal changes are determined by the charging and discharging power of the energy storage system. Decide, State of charge at time t By energy storage system State of charge at time t Updated to:

[0156]

[0157] in, It is an energy storage system in The charging and discharging power at any given time It is the total capacity of the energy storage system, in units of... , It is the charge / discharge efficiency coefficient, and when charging, it is the charging efficiency. The discharge efficiency is the efficiency during discharge. ,exist The update formula usually uses Appearing in the form of;

[0158] at the same time, Its operation must meet the upper and lower limit constraints:

[0159]

[0160] in, This represents the state of charge of the energy storage system at time t. The lower bound of the constraint, This represents the state of charge of the energy storage system at time t. The upper limit of the constraint condition;

[0161] 2) Battery degradation costs:

[0162] Each charge and discharge cycle of an energy storage system causes some loss to its battery life. This loss can be quantified as a degradation cost. A linear model is used, assuming that it is proportional to the energy throughput of the energy storage system:

[0163]

[0164] in, It is the degradation cost coefficient of the energy storage system, representing the economic loss caused by each kilowatt-hour of charging and discharging.

[0165] Step 13: Transform the energy scheduling problem into a constrained objective function optimization problem. The objective function includes the total revenue of the photovoltaic-storage-charging station and the total cost to electric vehicle users, and considers the penalty term for electric vehicle user dissatisfaction.

[0166] The expression is as follows:

[0167]

[0168] in, Let represent the objective function of the photovoltaic-storage charging station; where, Let T represent the total revenue of the photovoltaic-storage-charging station, and let T represent the set of time periods. This represents the charging price offered to electric vehicles by the photovoltaic-storage charging station at time t. This represents the electricity price of the external power grid at time t. This represents the total power purchased from the external power grid at time t; It is a penalty item for electric vehicle user dissatisfaction. It is a penalty function that applies when the electric vehicle is at its expected departure time. The desired state of charge was not achieved. At that time, the penalty function will give a negative reward. It is the coefficient of dissatisfaction among electric vehicle users;

[0169] Let the objective function of electric vehicle users be denoted as , where This represents the total cost for electric vehicle users;

[0170] express The lower limit, express The upper limit, Indicates that the energy storage system is in Charge and discharge power at any time The lower limit, Indicates that the energy storage system is in Charge and discharge power at any time The upper limit.

[0171] Step 2: Model the interaction between the photovoltaic energy storage charging station and multiple electric vehicles as a Stackelberg Markov game, with the photovoltaic energy storage charging station as the leader agent and each electric vehicle as a follower agent.

[0172] In this embodiment, the Stackelberg Markov game in step 2 is formally defined as a tuple. Its specific components are as follows:

[0173] (1) The set of players (P) in the game is as follows: ,in, Representing the sole leader intelligent agent, namely the photovoltaic energy storage charging station; Representing the A follower intelligent agent, namely an electric vehicle; , i Represents the index of the follower agent and i It is a positive integer. M This represents the total number of follower agents;

[0174] (2) The decision sequence S (Sequence of Moves) is the leader agent. First, take action based on its state. From its action space Select and publish its actions Then all the follower intelligent agents After observing the actions of the leader agent, simultaneously and independently, based on their respective states... From its action space Choose the optimal action for each. ;

[0175] (3) In Action Spaces A, the leader agent Action space It is a compact set (bounded and closed set) consisting of charging prices and the charging and discharging power of the energy storage system; the action space of each follower agent. It is a compact set consisting of charging power;

[0176] (4) In the utility function U, the optimization objective is to maximize the operational benefits of the leader agent and minimize the charging costs of the follower agents, that is, to maximize the utility function of the leader agent and the utility function of the follower agents; specifically as follows:

[0177] The utility function of each follower agent is: The optimization problem is:

[0178] ;

[0179] Among them, utility function It is about Continuous functions;

[0180] The utility function of the leader agent is When making decisions, the leader agent takes into account the actions of the follower agents. Each follower agent... i action Every optimization problem has at least one optimal solution. Therefore, utility function The optimization problem is:

[0181]

[0182] in, Represents each follower agent i action The corresponding optimal solution, It is to follow

[0183] Intelligent agent The optimal set of reactions is defined as:

[0184] ;

[0185] in, For any Both are non-empty sets;

[0186] (5) T represents the set of times, where t ∈ T for each decision stage.

[0187] Stackelberg equilibrium is a refined Nash equilibrium concept in sequential game theory, namely, Subgame Perfect Nash Equilibrium (SPNE). This invention uses backward induction to prove its existence.

[0188] Definition 1: Stackelberg Equilibrium

[0189] use Let's represent the Stackelberg game described above, where a strategy group... The Stackelberg equilibrium constituting this game applies to any leader agent. and follower intelligent agents All should satisfy:

[0190]

[0191]

[0192] in, In addition to follower intelligent agents The set of actions of all follower intelligent agents other than [the other agent].

[0193] Theorem 1: In the Stackelberg game as defined above In this context, there exists at least one Stackelberg equilibrium.

[0194] prove:

[0195] For any given action by the leader agent First, we analyze the decision-making problem of follower agents. Since each follower agent... utility function It is independent of the actions of other follower agents, therefore The subgame of a follower agent can be decomposed into: This is an independent optimization problem.

[0196] According to Weierstrass's extreme value theorem, a continuous function defined on a compact set must attain its maximum and minimum values. Therefore, for any action given by the leader agent... Each follower intelligent agent All optimization problems have at least one optimal solution. Therefore, the follower intelligent agent Optimal reaction set For any Both are non-empty sets.

[0197] Furthermore, according to Bell's maximum value theorem, due to the utility function of the follower agent... It is continuous, and its action space If it is a fixed compact set, then the optimal response of the follower agent corresponds to... It is an upper semi-continuous set-valued function.

[0198] Furthermore, if the utility function of the leader agent... Regarding the actions of all intelligent agents They are all continuous, and follow the optimal response of the agent. If the upper half is continuous, then the objective function of the leader agent, after substituting the optimal response, is... It is also an upper semi-continuous function.

[0199] Action space of the leader agent It is also a compact set. A semi-continuous function defined on a compact set must attain its maximum value. Therefore, the optimization problem of the leader agent also has at least one optimal solution. .

[0200] By backward induction, it is proved that in the second phase of the game, for any possible action of the leader agent, all follower agents have an optimal response strategy. In the first phase of the game, considering the optimal responses of the follower agents, the leader agent also has an optimal action strategy that maximizes its own utility. .

[0201] Therefore, an optimal strategy by the leader agent And the optimal response of all follower agents to this strategy. (in The strategy combination formed together It is a subgame perfect Nash equilibrium of this game, namely the Stackelberg equilibrium.

[0202] In this embodiment, in step 2, the photovoltaic-storage-charging station acts as the leader intelligent agent, and each electric vehicle acts as a follower intelligent agent; specifically, it includes:

[0203] (1) The photovoltaic-storage charging station is taken as the leader intelligent agent, and the leader intelligent agent is modeled as follows:

[0204] Status: In At any given moment, the state of the leader agent Defined as:

[0205]

[0206] in, It refers to the current moment; Is it the external power grid? t Electricity price at any given time; It is a photovoltaic power generation system in Output power at any given moment; It is an energy storage system in State of charge at time t, It is all electric vehicles at the previous moment The total charging power; this information represents the market's response to the previous price and is a key basis for leaders to adjust their current strategies;

[0207] Action: A leader's decision-making is twofold, encompassing both external pricing and internal energy management. The action at time t is defined as:

[0208]

[0209] in, It is the charging price that the photovoltaic and energy storage charging station gives to electric vehicles at time t; For energy storage systems in The charging and discharging power at any given moment;

[0210] Rewards: Rewards for the leader agent Related to its operating profit, the goal is to maximize that The value is calculated by subtracting total cost from total revenue, as shown in the following formula:

[0211]

[0212]

[0213] in, This represents the total charge level of all electric vehicles. Indicates in Electricity purchased from the external power grid at all times; This represents the degradation cost coefficient of the energy storage system; This indicates the degradation cost of the energy storage system;

[0214] (2) Each electric vehicle is treated as a follower agent, and the follower agent is modeled as follows:

[0215] State: Follower agent Its decision-making is based on its own state and the charging price information received from the leader agent. state of time Defined as:

[0216]

[0217] in, It is the current moment. It is an electric car i The state of charge of the battery at time t. It is an electric car i The target battery charge state; It is an electric car i Expected departure time;

[0218] Action: Follower agent Actions at time t That is, the first i The charging power of an electric vehicle at time t, this action Limited by the maximum charging power of the electric vehicle itself;

[0219]

[0220] in, Indicates electric vehicles The upper limit of charging power;

[0221] Reward: Follower agent reward function Designed as a utility function consisting of negative cost terms, the goal is to maximize this utility function:

[0222] .

[0223] Step 3: Construct independent agent networks for the leader agent and each follower agent. Each agent network includes a main Actor network, two main Critic networks, and a corresponding target network.

[0224] In this embodiment, each agent network in step 3 includes a main Actor network, two main Critic networks, and a corresponding target network. Specifically, the main Actor network is used to output actions based on the current state, taking the agent's local state as input and outputting its actions; the two main Critic networks are used to evaluate the Q-value of the state-action pair, taking the state and action as input and outputting the evaluated Q-value; the target network includes a target Actor network and two target Critic networks, whose network structures are the same as the main Actor network and the main Critic network, respectively.

[0225] Step 4: During the training process, at each time step, the leader agent first formulates the charging price and energy storage scheduling strategy and broadcasts it. Then, each follower agent makes its own charging power decision in parallel based on this.

[0226] In this embodiment, step 4 specifically includes:

[0227] Step 41: At each time step, the leader agent outputs the charging price and energy storage system charging and discharging power as actions through its main Actor network based on its local state information, and broadcasts the charging price to all follower agents.

[0228] Step 42: After observing the charging price, each follower agent outputs charging power as an action in parallel and independently through its own main Actor network, based on its local state information.

[0229] Step 5: After all agents have performed their actions, they store their respective interaction experience data in their independent local experience replay pools.

[0230] In this embodiment, step 5 specifically includes:

[0231] Step 51: Configure an independent local experience replay pool for each agent;

[0232] Step 52: After all agents perform actions, they store their respective interaction experience data in the corresponding local experience replay pool. The experience data includes tuples of state, action, reward, and next state.

[0233] Step 6: Based on the priority of the temporal difference error, each agent samples experience data from the local experience replay pool, uses the SMATD3 algorithm, and updates the main Critic network, main Actor network, and target network using the sampled experience data.

[0234] The SMATD3 algorithm framework is as follows: Figure 3 As shown. During the training phase, the agent trains its local network offline using only locally observed information and the actions of interacting agents. During the execution phase, the agent executes actions online using only locally observed information and the actions of interacting agents. No information exchange is required between follower agents in either phase. Information exchange between the leader and follower agents also only requires exchanging action information, without sharing private locally observed data. Furthermore, as... Figure 3 As shown, the proposed method schedules electric vehicle charging through the interaction between the agent and the environment, without requiring precise modeling.

[0235] In this embodiment, step 6, where each agent samples experience data from the local experience replay pool based on the priority of the temporal difference error, specifically includes:

[0236] During training, the SMATD3 algorithm uniformly and randomly samples a batch of experiences from the local experience replay pool to update the neural network. This uniform sampling method is simple and effective, in which all experiences are equally important for learning. The core idea of ​​PER (Priority Local Experience Replay Pool) is to break the assumption of uniform sampling, allowing the agent to learn more frequently from experiences that differ significantly from the prediction and contain more information, thereby accelerating the learning process.

[0237] (1) In priority experience playback, the magnitude of the time difference error (TD-Error) is used as an indicator to measure the importance of an experience. This represents the difference between the agent's current prediction of the value of a state-action pair and the estimate obtained from the environment. The formula for calculating this difference is:

[0238]

[0239] in, It is the agent's prediction of the value of the current state-action pair. It utilizes the current reward and the maximum Q value of the next state The calculated estimated value; Discount factor;

[0240] A large absolute value of TD-Error indicates a significant discrepancy between the agent's prediction and the actual result. This suggests that the agent's understanding of the situation is incorrect or insufficient, and also indicates that the experience contains a large amount of new information. A small absolute value of TD-Error means that the agent's prediction is very accurate, it has a good understanding of the situation, and there is little to learn from it.

[0241] Therefore, the absolute value of TD-Error This is used as the initial priority for that experience. However, if the highest priority experience is greedily selected for learning each time, the agent will always learn only a few high-priority experiences, which can easily lead to overfitting on these samples and loss of generalization ability. Furthermore, low-priority experiences may never be sampled, which is detrimental to maintaining the stability and comprehensiveness of the learning.

[0242] (2) For the first in the local experience replay pool j The experience, its priority Defined as:

[0243]

[0244] in, This indicates that the first j The empirical time-series difference error, the first j The absolute value of the empirical time-series difference error Used as the first j The initial priority of each experience, It is a constant that guarantees all experiences can be sampled;

[0245] (3) No. j Sampling probability of a piece of experience Defined as:

[0246] in, This is the total number of experience points in the local experience replay pool. It is a hyperparameter between 0 and 1, which determines the importance of the priority; when When = 0, all empirical sampling probabilities are equal, and uniform random sampling is performed; when When =1, sampling is performed entirely according to priority; by adjusting... A trade-off can be struck between purely greedy priority sampling and uniform sampling.

[0247] (4) Because the sampling distribution of the data has been changed, the data is no longer uniformly distributed, which introduces bias. Since the model samples too many high-priority samples, the update direction may be biased. To correct this bias, PER introduces importance sampling weights.

[0248] The specific approach is to calculate a sampling weight for each sampled experience. The sampling weight is then multiplied by the loss of that sample when calculating the loss function. This ensures that although the sampling process is biased to improve efficiency, the gradient ultimately used to update the neural network is an unbiased estimate of the true expected gradient. The formula for calculating the sampling weight is:

[0249]

[0250] in, This is a hyperparameter used to correct for bias, typically starting with a small value and gradually annealing to 1 during training. It determines the extent to which the effects of non-uniform sampling are fully compensated. When When =1, the deviation is completely eliminated. By reducing the influence of high-priority samples in gradient updates while increasing the influence of low-priority samples, the learning bias caused by priority sampling is corrected, thereby ensuring that the algorithm can still converge to the optimal policy stably and accurately while pursuing high efficiency.

[0251] The SMATD3 algorithm is employed for agent network updates. The core of SMATD3 lies in the fact that each agent utilizes the MATD3 network update mechanism to achieve stable and efficient learning within a distributed framework. Compared to earlier algorithms such as MADDPG, MATD3 employs a pruned double-Q learning approach. When calculating the target Q-value, MATD3 selects the smaller Q-value from two independent main Critic networks for updating. This conservative strategy effectively decouples the selection and calculation of the target Q-value, breaking the cumulative cycle of overestimation and resulting in more accurate and robust value assessment. Furthermore, to improve training stability during network updates, two key techniques are introduced: delayed updates and target policy smoothing. In MATD3, the update frequency of the main Actor network is lower than that of the main Critic network. Typically, the main Critic network updates multiple times before the main Actor network updates once. This delay allows the main Critic network sufficient time to converge to a more accurate value estimate, thus providing a more stable and reliable gradient signal for policy updates and avoiding policy oscillations caused by frequent fluctuations in the value function. The target policy smoothing mechanism, MATD3, adds a small amount of bounded noise to the action output of the target Actor network when calculating the target Q-value. This smooths the Q-function value, resulting in a smoother evaluation value for similar actions. This effectively prevents the policy network from overfitting to certain peak regions of the value function, thereby enhancing the algorithm's generalization ability and robustness.

[0252] Therefore, through the synergistic effect of the aforementioned multiple stabilization mechanisms, MATD3 significantly improves the stability of the learning process and the performance of the final policy, demonstrating a clear advantage over algorithms such as MADDPG. The specific algorithm flow is as follows: Figure 4 As shown.

[0253] The SMATD3 algorithm abandons the traditional CTDE framework and instead designs a network update mechanism for each agent that performs distributed training locally. The core of this mechanism is that each agent's main Critic network can only access the information it can obtain under the Stackelberg game structure.

[0254] In this embodiment, step 6 updates the main Critic network using sampled empirical data; specifically, it includes:

[0255] (1) The target value is calculated by pruning double Q learning and target policy smoothing techniques; specifically, the smaller output value of the two target Critic networks is used as the basis for the target value, and bounded noise is added to the action output of the target Actor network during the calculation.

[0256] (2) The main Critic network is updated by minimizing the mean squared error loss between the predicted value and the target value of the main Critic network;

[0257] The specific update process of the main Critic network for both follower and leader agents is as follows:

[0258] 1) For the main Critic network update of the follower agent:

[0259] For any follower agent, its goal is to maximize its own utility given the charging price provided by the leader agent, and its network updates are entirely based on its local information. Two main Critic networks Its input is only its own local state. and the actions one takes ;

[0260] Follower agent The k Loss function of the principal Critic network For target value Compared with the current main Critic network for a given state-action pair Predicted value The mean square error between them is expressed as:

[0261]

[0262] in, k Indicates the index of the main Critic network. Represents follower intelligent agents The k Parameters of the main Critic network, Represents follower intelligent agents The local experience replay pool Indicates the replay pool of local experience The expected value of an empirically derived random sample. The target value is represented as follows:

[0263]

[0264] in, For follower intelligent agents The reward function, Indicates the following intelligent agent The The Q-value function output by the target Critic network. As a discount factor, Representative follower intelligent agent In state The target Actor network, It is smooth noise added to the target action, and the noise distribution is as follows: The tilde (~) indicates compliance, and the clip (.) function represents the clipping function. Represents a normal distribution. It is a random noise parameter;

[0265] 2) Update of the leader agent's main Critic network:

[0266] The decisions of the leader agent affect all follower agents, and its gains depend on the responses of all follower agents. Therefore, its network updates need to take into account the action information of all agents.

[0267] Leader agent Two main Critic networks The input needs to include its own local state. One's own actions And the actions of all the follower agents it can observe. Leader agent The k Loss function of the principal Critic network Defined as:

[0268]

[0269] in, Represents the leader intelligent agent The k Parameters of the main Critic network, Indicates the expected value. Represents the leader intelligent agent The k Individual Critic Network; Indicates the target value;

[0270] Target value The calculation method is as follows:

[0271]

[0272] in, Represents the leader intelligent agent The k A target Critic network, Represents the leader intelligent agent The next state, Represents the leader intelligent agent The next move, This indicates the actions of all follower agents in the next moment.

[0273] In this embodiment, step 6 updates the main Actor network using sampled empirical data; specifically, it includes:

[0274] (1) Set the update frequency of the main Actor network to be lower than that of the main Critic network;

[0275] (2) The main Actor network is updated by the policy gradient ascent method, wherein the policy gradient is the product of the gradient of the main Critic network output with respect to the action and the gradient of the main Actor network output with respect to the parameter.

[0276] The specific update process of the main Actor network for both follower and leader agents is as follows:

[0277] 1) For updates to the main Actor network of follower agents:

[0278] Because agents lack sufficient understanding of their environment in the early stages of learning, Gaussian noise is introduced to enhance their exploration of the environment. After introducing Gaussian noise, for follower agents... Its main Actor network generates actions using the following formula. :

[0279]

[0280] in, It is a follower intelligent agent exist Gaussian noise at time t, It is a follower intelligent agent The main Actor network, whose parameters are: It updates by maximizing the output of its main Critic network; the update method involves adjusting parameters along the direction of the policy gradient. It can be expressed by the following formula:

[0281]

[0282] in, Describe the objective function Parameters of the main Actor network policy gradient, Indicates the replay pool of local experience Mid-sampled state Expected value; Represents follower intelligent agents i The main Critic network, Represents follower intelligent agents i The output of the main Critic network is relative to the input action. The gradient; Represents follower intelligent agents i The main Actor network, This indicates that the output of the main Actor network is relative to the parameters. The gradient;

[0283] 2) Update of the leader agent's main actor network:

[0284] Leader agent Main Actor Network Its parameters are Its policy gradient Defined as:

[0285]

[0286] in, Describe the objective function Parameters of the main Actor network policy gradient, Indicates the expected value. The main Critic network represents the leader agent. The output of the main Critic network represents the leader agent's action relative to the leader agent's own action. The gradient; The main Actor network represents the leader agent. This represents the output of the main Actor network relative to the leader agent's own parameters. The gradient.

[0287] In this embodiment, step 6 updates the target network using sampled empirical data to stabilize the entire training process and avoid errors caused by the target value. , The learning oscillations caused by sharp fluctuations result in the target network parameters being updated using the same soft update method for all agents. This method allows the target network parameters to slowly track the main network parameters, with the update rule being:

[0288]

[0289] in, Parameters representing the target network (including the target Actor network and the target Critic network), Parameters representing the main network (including the main Actor network and the main Critic network), The soft update coefficient is between 0 and 1, which allows the parameters of the target network to slowly track the parameters of the main network.

[0290] Step 7: Repeat steps 4 to 6 until the strategies of all agents converge, obtain the optimal dynamic charging price and energy storage scheduling strategy for charging stations and the optimal charging strategy for electric vehicles, and maximize the operating revenue of photovoltaic-storage charging stations.

[0291] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for electric vehicle charging scheduling optimization based on Stackelberg game, characterized in that, Comprising the following steps: Step 1, constructing a system model of the light storage charging station, and defining the energy management strategy inside the system model to realize the interaction between the light storage charging station and multiple electric vehicles; Step 2, modeling the interaction between the optical storage charging station and a plurality of electric vehicles as a Stackelberg Markov game, the optical storage charging station as a leader agent, and each of the electric vehicles as a follower agent; the Stackelberg Markov game is formally defined as a tuple The specific components are as follows: (1) the set of participants of the game is wherein, represents the unique leader agent, i.e. the light storage charging station; represents the i-th follower agent, i.e. the electric vehicle; , , i denotes the index of the follower agent and i is a positive integer, M denotes the total number of follower agents; (2) the decision order S is leader agent First, the leader agent selects and announces its action from its action space according to its state , then all follower agents simultaneously and independently select their optimal actions from their action spaces according to their states after observing the action of the leader agent. (3) In the action space A, the leader agent 's action space is a compact set consisting of charging price and charging / discharging power of the energy storage system; the action space of each follower agent is a compact set consisting of charging power; (4) In the utility function U, specifically as follows: The utility function of each follower agent is The optimization problem for each follower agent is: ； The utility function of the leader agent is where each follower agent i optimizes the action The optimization problem of the utility function has at least one optimal solution and the optimization problem of the utility function is wherein, represents the action of each follower agent i corresponding optimal solution, is the follower​ agent the optimal set of responses of the agent, defined as: ； wherein For any is a non-empty set; (5) T represents the set of time, for each decision-making stage time t, t∈T; Step 3, constructing an independent agent network for the leader agent and each follower agent respectively, each agent network including a main Actor network, two main Critic networks and corresponding target networks; Step 4, in the training process, the leader agent first formulates the charging price and energy storage scheduling strategy at each time step and broadcasts it, and then each follower agent makes a decision on its own charging power accordingly; Step 5, after all agents perform actions, store their interaction experience data in their independent local experience replay pools; Step 6, each agent samples experience data from the local experience replay pool based on the priority of the time difference error, and updates the main Critic network, the main Actor network and the target network using the sampled experience data; Step 7, repeat steps 4 to 6 until the strategies of all agents converge, obtaining the optimal dynamic charging price and energy storage scheduling strategy of the charging station and the optimal charging strategy of the electric vehicle, and maximizing the operating income of the light storage charging station.

2. The method of claim 1, wherein the Stackelberg game-based electric vehicle charging scheduling optimization method is characterized by, The step 1 specifically comprises: Step 11, constructing a system model of the light storage charging station, the system model comprising an external power grid, a light storage charging station, a photovoltaic power generation system, an energy storage system, a charging pile and an electric vehicle; specifically: In terms of energy flow, the external power grid and the photovoltaic power generation system are connected to the energy storage system, which supplies power to the electric vehicles connected to the charging piles; in terms of information flow, the light storage charging station is communicatively connected with the external power grid, the photovoltaic power generation system, the energy storage system and the electric vehicles, for collecting operating state information and issuing control instructions, thereby realizing centralized management of the energy flow inside the system model; Step 12, defining the energy flow priority inside the system model as follows: preferentially using the electricity generated by the photovoltaic power generation system to meet the charging demand of the electric vehicles, storing the excess electricity in the energy storage system, and supplementing the insufficient part by using the electricity in the energy storage system or purchasing electricity from the external power grid; specifically comprising: (1) In the photovoltaic power generation system, in the moment, the actual photovoltaic power absorbed by the light storage charging station depends on the output power of the photovoltaic power generation system and the real-time electricity demand in the light storage charging station, and the distribution logic is as follows: the photovoltaic power generation system will first meet the charging demand of the electric vehicle, and if there is still surplus, it will be used to charge the energy storage system, and the relationship is represented as: wherein, is the output power of the photovoltaic power generation system at time instant, i is the index of the electric vehicle, M is the total number of electric vehicles, denotes the charging power of the i th electric vehicle at time instant t, is the total charging power of all electric vehicles at time instant, is the charging power of the energy storage system at time instant, when the energy storage system is charging, > 0, otherwise is 0; (2) In the energy storage system, including energy balance and state update as well as battery degradation cost; 1) Energy balance and state update: The state of charge SOC of the energy storage system is updated at each time step by the state of charge of the energy storage system at the time instant t+1, by the state of charge of the energy storage system at the time instant t+1, by the state of charge of the energy storage system at the time instant t+1, in, It is an energy storage system in The charging and discharging power at any given time It is the total capacity of the energy storage system, in units of... , It is the charge / discharge efficiency coefficient, and when charging, it is the charging efficiency. The discharge efficiency is the efficiency during discharge. ; At the same time, Upper and lower constraints on its operation must be met: wherein, SoC(t) represents the state of charge of the energy storage system at time t lower bound of the constraint, SoC(t) represents the state of charge of the energy storage system at time t upper bound of the constraint; 2) Battery degradation cost: Quantifying the amount of degradation caused by each charge and discharge of the energy storage system as a degradation cost and adopting a linear model that is proportional to the energy throughput of the energy storage system: wherein, is the degradation cost coefficient of the energy storage system, representing the economic loss brought by each charging and discharging degree. Step 13, converting the energy scheduling problem into a constrained objective function optimization problem, the objective function including the total revenue of the light storage charging station and the total cost of the electric vehicle users, and considering the penalty term of the dissatisfaction degree of the electric vehicle users; The expression is as follows: in, Let represent the objective function of the photovoltaic-storage charging station; where, Let T represent the total revenue of the photovoltaic-storage-charging station, and let T represent the set of time periods. This represents the charging price offered to electric vehicles by the photovoltaic-storage charging station at time t. This represents the electricity price of the external power grid at time t. This represents the total power purchased from the external power grid at time t; It is a penalty item for electric vehicle user dissatisfaction. It is a penalty function that applies when the electric vehicle is at its expected departure time. The desired state of charge was not achieved. At that time, the penalty function will give a negative reward. It is the coefficient of dissatisfaction among electric vehicle users; represents an objective function of an electric vehicle user, wherein, represents a total cost of an electric vehicle user; express The lower limit, express The upper limit, Indicates that the energy storage system is in Charge and discharge power at any time The lower limit, Indicates that the energy storage system is in Charge and discharge power at any time The upper limit.

3. The method of claim 1, wherein the Stackelberg game is based on a charging cost of the electric vehicle and a charging cost of the charging station. In the utility function U, the optimization goal is to maximize the operating income of the leader agent and minimize the charging cost of the follower agent, that is, to maximize the utility function of the leader agent and to maximize the utility function of the follower agent. 4.The electric vehicle charging scheduling optimization method based on Stackelberg game of claim 1, wherein, In step 2, the light storage charging station is the leader agent, and each electric vehicle is the follower agent; specifically comprising: (1) The light storage charging station is taken as a leader agent, and the leader agent is modeled as follows: State: In the state of the leader agent at time is defined as: in, It is the current moment; Is it the external power grid? t Electricity price at any given time; It is a photovoltaic power generation system in Output power at any given moment; It is an energy storage system in State of charge at time t, It is all electric vehicles at the previous moment Total charging power; Action: The action at time t is defined as: wherein, is the charging price given by the optical storage charging station to the electric vehicle at time t; is the charging and discharging power of the energy storage system at time t. Rewards: The reward for the leader agent which is calculated as total revenue minus total cost, as shown in the following equation: wherein, represents the charging power of all electric vehicles; represents the power purchased from the external power grid at the time point t; represents the power purchased from the external power grid at the time point t; represents the degradation cost coefficient of the energy storage system; represents the degradation cost of the energy storage system; (2) Each electric vehicle is taken as a follower agent, and the follower agent is modeled as follows: State: Follower agent The decision of the follower agent depends on its own state and the charging price information received from the leader agent, which is at the moment of time defined as: wherein, is the current time, is an electric vehicle i is the state of charge of the battery at time t, is an electric vehicle i is the target state of charge of the battery; is an electric vehicle i is the expected departure time of the electric vehicle; Action: Follower agent Action at time t , the charging power of the i-th electric vehicle at time t, is limited by the maximum charging power of the electric vehicle itself; i ​​ wherein, represents the upper limit of the charging power of an electric vehicle ; Reward: follower agent Reward function The reward function of the follower agent is designed as a utility function composed of a negative cost term, which is to be maximized: 。 5. The method of claim 1, wherein, The network of each agent in step 3 includes a main Actor network, two main Critic networks and corresponding target networks, and specifically includes: the main Actor network is used to output an action according to a current state, taking the local state of the agent as input, and outputting the action thereof; the two main Critic networks are used to evaluate the Q value of a state-action pair, taking the state and action as input, and outputting the evaluated Q value; the target network includes a target Actor network and two target Critic networks, and the network structures thereof are the same as those of the main Actor network and the main Critic network.

6. The method of claim 1, wherein, Step 4 specifically includes: Step 41: At each time step, the leader agent outputs a charging price and a storage system charging and discharging power as an action through its main Actor network according to its local state information, and broadcasts the charging price to all follower agents; Step 42: After observing the charging price, each follower agent outputs a charging power as an action through its own main Actor network in parallel and independently according to its own local state information; Step 5 specifically includes: Step 51: An independent local experience replay pool is configured for each agent; Step 52: After all agents perform the action, the interaction experience data of each agent is stored in the corresponding local experience replay pool, and the experience data includes a tuple of state, action, reward and next state.

7. The method of claim 1, wherein the Stackelberg game is based on a charging cost of the EVs and a charging cost of the charging stations. In step 6, each agent samples experience data from the local experience replay pool based on the priority of the time difference error, and specifically includes: (1) In the priority experience replay, the size of the temporal difference error is used as an indicator to measure the importance of an experience, and the temporal difference error represents the difference between the current prediction value of the agent for a certain state-action pair and the estimated value obtained through the environment, and its calculation formula is: wherein, is the agent's prediction of the value of the current state-action pair, is the estimate computed using the current reward and the maximum Q-value of the next state ; and is a discount factor. (2) For the i-th experience in the local experience replay pool, the priority p(i) is defined as: j p(i) = p(i - 1) + a * (r - p(i - 1)) where a is a constant, and r is the reward of wherein, represents the time-difference error of the i-th experience j represents the time-difference error of the i-th experience j represents the absolute value of the time-difference error of the i-th experience is used as the initial priority of the i-th experience, j is a constant that guarantees that all experiences can be sampled, is a constant that guarantees that all experiences can be sampled, (3) Article j the sampling probability of the experienced is defined as: wherein, is the total number of experiences in the local experience replay pool, is a hyper-parameter between 0 and 1, when = 0, the sampling probability of all experiences is equal, uniform random sampling is performed; when = 1, sampling is performed completely according to the priority. (4) calculating a sampling weight for each sampled experience and multiplying the loss of the sample by the sampling weight when calculating the loss function, the formula of the sampling weight is: wherein, is a hyperparameter for correcting the bias, and when = 1, the bias is completely eliminated.

8. The method of claim 1, wherein, In step 6, the main Critic network is updated using the sampled experience data; specifically including: (1) The clipped double Q learning and target policy smoothing technology are used to calculate the target value; specifically, the output value with a smaller value in the two target Critic networks is used as the basis of the target value, and a bounded noise is added to the action output by the target Actor network during calculation; (2) The main Critic network is updated by minimizing the mean square error loss between the predicted value of the main Critic network and the target value; Wherein, the main Critic network update process of the follower agent and the leader agent is as follows: 1) For the main Critic network update of the follower agent: For any one follower agent, the inputs to its two main Critic networks are only its own local state and the action it takes itself ; Follower agent The k Loss function of the principal Critic network For target value Compared with the current main Critic network for a given state-action pair Predicted value The mean square error between them is expressed as: wherein, k denotes the index of the master Critic network, denotes the follower agent 's k th master Critic network's parameters, denotes the follower agent 's local experience replay buffer, denotes the expected value of a randomly sampled experience from the local experience replay buffer , denotes the target value, which is computed as follows: in, For follower intelligent agents The reward function, Indicates the following intelligent agent The The Q-value function output by the target Critic network. As a discount factor, Representative follower intelligent agent In state The target Actor network, It is smooth noise added to the target action, and the noise distribution is as follows: The tilde (~) indicates compliance, and the clip (.) function represents the clipping function. Represents a normal distribution. It is a random noise parameter; 2) The main Critic network update of the leader agent: Leader agent Two main Critic networks The input needs to include its own local state. One's own actions And the actions of all the follower agents it can observe. Leader agent The k Loss function of the principal Critic network Defined as: wherein, denotes the parameters of the i-th main Critic network of the leader agent , k denotes the parameters of the i-th main Critic network of the leader agent , denotes the parameters of the i-th main Critic network of the leader agent , k denotes the parameters of the i-th main Critic network of the leader agent denotes the target value; Target value The calculation is as follows: wherein, represents a leader agent a first k critic network, represents a next state of a leader agent a second critic network, represents a next action of a leader agent represents actions of all follower agents at the next time.

9. The method of claim 1, wherein, In step 6, the main Actor network is updated using the sampled experience data; specifically including: (1) The update frequency of the main Actor network is set to be lower than that of the main Critic network; (2) The main Actor network is updated by the policy gradient ascent method, and the policy gradient is composed of the product of the gradient of the output of the main Critic network with respect to the action and the gradient of the output of the main Actor network with respect to the parameters; Wherein, the main Actor network update process of the follower agent and the leader agent is as follows: 1) For the main Actor network update of the follower agent: After introducing Gaussian noise, for the follower agent whose main Actor network generates actions by the following formula : wherein, is the follower agent at time t, is a Gaussian noise, is the follower agent with parameters is updated by maximizing the output of its master Critic network; its update is a parameter adjustment in the direction of the policy gradient, which is expressed by the following equation: wherein, represents the target function regarding the parameters of the main Actor network the policy gradient, represents the expected value of a state sampled from the local experience replay pool ; represents the output of the main Critic network for the follower agent ; i represents the gradient of the output of the main Critic network for the follower agent with respect to the input action i ; represents the main Actor network for the follower agent ; i represents the gradient of the output of the main Actor network for the follower agent with respect to the parameters ; 2) The main Actor network update of the leader agent: leader agent of the main actor network with parameters whose policy gradient is defined as: wherein, represents the target function regarding the parameters of the main Actor network the policy gradient, represents the expected value, represents the main Critic network of the leader agent, represents the gradient of the output of the main Critic network of the leader agent with respect to the leader agent's own action ; represents the main Actor network of the leader agent, represents the gradient of the output of the main Actor network of the leader agent with respect to the leader agent's own parameters .

10. The method of claim 1, wherein, The target network is updated by using the sampled experience data in step 6, and the update rule is: wherein, parameters representative of the target network, parameters representative of the master network, is a soft update coefficient between 0 and 1, such that the parameters of the target network slowly track the parameters of the master network.

Images