Real-time optimization dispatching method, system, terminal and medium for distributed electric vehicles

Abstract

The present invention provides a real-time optimization scheduling method and system for distributed electric vehicles, establishes a real-time optimization scheduling model for combining electric vehicle charging and power grid unit output; improves the model solving efficiency through an electric vehicle cluster energy boundary equivalent model and an optimized power redistribution strategy; constructs the proposed optimization scheduling model into a Markov decision process; and solves it using a reinforcement learning algorithm; applies the trained intelligent agent to the real-time optimization scheduling of distributed electric vehicles, optimizes the unit output and electric vehicle charging power in each time period in real time, and realizes the economic operation of the power grid. A corresponding terminal and medium are also provided. On the one hand, the present invention does not need to rely on accurate model prediction, and the continuous action space can ensure the flexibility of regulation and control, and the solution speed is fast; on the other hand, the SAC algorithm is preferably used as a reinforcement learning algorithm, which has the characteristics of maximizing entropy, can widely explore the environment, and still has strong robustness under environmental changes.

Description

Technical Field

[0001] The present invention relates to the field of power system optimization and dispatching, and in particular to a method, system, terminal and medium for real-time optimization and dispatching of distributed electric vehicles based on reinforcement learning. Background Art

[0002] With the general trend of increasingly tight oil resources, traditional fuel vehicles that rely heavily on oil are facing increasing pressure in development. As a new type of transportation, electric vehicles (EVs) have the characteristics of low pollution and high energy utilization. In the situation of energy shortage and environmental degradation, the number of electric vehicles is growing rapidly. At the same time, considering that electric vehicles have the dual attributes of controllable loads and energy storage units, if their mobile energy storage characteristics can be fully utilized and their huge potential for participating in power grid dispatching can be tapped, on the one hand, the adverse effects of disorderly charging on the power grid can be eliminated, and on the other hand, support for power grid operation can be provided through demand response (DR) services.

[0003] Considering that there are a large number of decentralized electric vehicles (i.e., electric vehicles charged through home charging piles) that are difficult to manage, the power grid can manage and optimize the charging process of electric vehicles through electric vehicle aggregators (EV Aggregators). At present, many electric vehicles have participated in the research on power grid optimization and dispatching, which can generally be divided into day-ahead optimization and real-time optimization. In the real-time stage, the randomness of the charging behavior of a large number of decentralized electric vehicles is strong and difficult to accurately predict. The complexity and uncertainty of the power system also increase, which brings great challenges to the operation optimization in the real-time stage. At the same time, with the continuous development of artificial intelligence technology, it has also been maturely applied in the power system. In this context, the advantages of reinforcement learning algorithms that do not require model building and are driven by data have gradually become significant. Some machine learning algorithms such as reinforcement learning have been widely used in the real-time optimization and dispatching of electric vehicles.

[0004] After searching, we found:

[0005] The Chinese invention patent application with publication number CN111313449A, "A cluster electric vehicle power optimization management method based on machine learning", discloses an online power control method for cluster electric vehicles based on long short-term memory neural network (LSTM). The method selects the optimal LSTM network for charging piles with different historical charging data, and verifies the advantage of this method in solution time over the mixed integer linear programming method. However, this method is not suitable for situations involving a large number of distributed electric vehicles.

[0006] The Chinese invention patent application with publication number CN109193721A, "An Optimization Method for Charging and Discharging Strategies of Electric Vehicles Based on Reinforcement Learning", aims to maximize the frequency modulation benefits of vehicle owners, and discloses a charging and discharging strategy for electric vehicles participating in frequency modulation based on reinforcement learning, which can quickly respond to frequency modulation signals in a very short time. However, the willingness of vehicle owners to participate is not taken into account when dispatching vehicles, and the Q-learning algorithm used can only process discrete charging and discharging power, and cannot continuously adjust the power.

[0007] The Chinese invention patent application with publication number CN111934335A, "A method for optimizing charging behavior of clustered electric vehicles based on deep reinforcement learning", discloses a distributed optimization algorithm for charging behavior based on reinforcement learning. Under the premise of ensuring the optimization speed and flexibility of charging power adjustment, this method reduces user expenses and achieves the effect of peak load reduction and valley filling of power grid load. However, compared with the centralized algorithm, the distributed algorithm only considers the benefits of the car owner, which may cause another load peak during the valley period at night, thereby causing a burden on the power grid.

[0008] The Chinese invention patent application with publication number CN111429038A, "A real-time random optimization scheduling method for active distribution networks based on reinforcement learning", discloses a random optimization scheduling method for electric vehicle clusters based on the time series difference TD(1) strategy iteration algorithm, and uses approximate value functions to solve the real-time random optimization scheduling problem of active distribution networks. It effectively improves the economy of distribution network operation, realizes "peak shaving and valley filling", and still has good results when random factors change drastically. In order to avoid the curse of dimensionality in centralized optimization, this method proposes an equivalent electric vehicle cluster model, but does not explain how the power is distributed after cluster optimization.

[0009] Therefore, it is urgent to find a decentralized electric vehicle optimization scheduling method that takes into account the interests of both car owners and power grids, has high solution efficiency, and adapts to real-time scheduling requirements. Currently, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not been collected. Summary of the invention

[0010] In view of the above-mentioned deficiencies in the prior art, the present invention provides a real-time optimization scheduling method, system, terminal and medium for distributed electric vehicles based on reinforcement learning.

[0011] According to one aspect of the present invention, a method for real-time optimization scheduling of distributed electric vehicles is provided, comprising:

[0012] Establish a real-time optimization dispatch model that combines electric vehicle charging and grid unit output;

[0013] Electric vehicles with the same charging end time are divided into the same cluster, an equivalent electric vehicle cluster energy boundary model is established, and the power redistribution strategy is optimized;

[0014] The optimization problem of cluster electric vehicle charging and power grid unit output is constructed as a Markov decision process to determine the state space quantity, action space quantity and reward corresponding to each moment;

[0015] The Markov decision process is solved by using a reinforcement learning algorithm, and a reinforcement learning agent is trained to adapt to the uncertainty of charging load in the real-time stage;

[0016] The reinforcement learning agent is applied to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period.

[0017] Preferably, the establishment of a real-time optimization dispatching model combining electric vehicle charging and power grid unit output includes:

[0018] Taking the lowest unit fuel cost as the optimization goal of the real-time optimization scheduling model, the optimization objective function is:

[0019]

[0020] Among them, N G is the number of generating units, T is the number of time periods in an optimization cycle, and p i,t is the power generation of unit i at time t; u i,t It is a binary number determined according to the on / off plan of the day before. It is 1 when the unit is in operation and 0 when the unit is in shutdown. i CB i and CC i is the power generation cost parameter of unit i;

[0021] The constraints of the real-time optimization scheduling model are set as follows:

[0022]

[0023] p i,t -p i,t-1 ≤u i,t-1 P up,i (3)

[0024] p i,t-1 -p i,t ≤u i,t-1 P down,i (4)

[0025]

[0026] Wherein, formula (2) is the power generation power p of unit i i,t Upper and lower bound constraints, where and They represent the lower and upper limits of the power of unit i at time t, which are not only limited by the unit's own capacity, but also affected by the unit's climbing constraints. Formulas (3) and (4) are the unit's climbing constraints, where P up,i and P down,i are the maximum ramp-up rate and the maximum ramp-down rate of unit i respectively; Formula (5) represents the system power balance, where P D,t Including the predicted conventional load and non-flexible EV load, N EV represents the total number of flexible electric vehicles participating in the dispatch, represents the charging power of electric vehicle k participating in the dispatch, P loss,t is the network loss of the system at time t;

[0027] The relevant constraints for a single electric vehicle participating in the dispatch are set as follows:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Among them, formula (6) represents the electric vehicle charging energy constraint, where represents the power of electric vehicle k at the end of time period t, η represents the charging efficiency, and Δt represents the unit time period of the optimization cycle; Formula (7) and Formula (8) represent the upper and lower limit constraints of energy and charging power of electric vehicle k at time t respectively; Formula (9) ensures that the EV is fully charged when leaving, and the upper and lower limits of power are equal to the required power, where t end represents the time when electric car k leaves, represents the amount of electricity required by electric vehicle k when it leaves; Equations (10) and (11) further restrict the lower and upper limits of the charging power of electric vehicle k at time t, respectively. represents the maximum charging power limited by the charging pile or the battery itself; Formula (10) represents the power at the previous moment Lower than the current minimum power limit When the charging power lower limit will be forced to increase; Formula (11) represents the power at the previous moment Approaching the current battery limit When the vehicle leaves the vehicle, the upper limit of the charging power will be forced to be lowered; a charging energy boundary model for a single vehicle is introduced, where the upper bound of energy means that the vehicle is charged at the maximum power at the beginning, and maintains the maximum energy after being fully charged until leaving; the lower bound of energy means that the vehicle delays charging, and continues to charge at the maximum power in the first few time periods before leaving and is fully charged at the time of leaving; the vehicle's power change curve can only fluctuate within the upper and lower bounds of energy;

[0035] At this point, through the optimization objective formula (1) and the constraint condition formulas (2) to (11), the real-time optimization scheduling model for the combined electric vehicle charging and power grid unit output is obtained.

[0036] Preferably, the method of dividing electric vehicles with the same charging end time into the same cluster, establishing an equivalent electric vehicle cluster energy boundary model, and optimizing the power redistribution strategy includes:

[0037] The electric vehicles with the same departure time are regarded as the same cluster, and the charging energy boundary models of individual vehicles are superimposed to obtain the energy boundary model of the electric vehicle cluster. The constraints related to the individual electric vehicles are replaced with the constraints related to the cluster, then:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] Among them, equations (12) to (15) represent the lower limit constraint, upper limit constraint, lower limit constraint and upper limit constraint of charging power of electric vehicle cluster m at time t, respectively. m represents the number of electric vehicles in cluster m, and They represent the lower and upper limits of the power consumption of the cluster respectively. and They represent the lower and upper limits of the cluster’s charging power respectively; Formula (16) indicates that the cluster power is equal to the sum of the powers of all the vehicles in the cluster. is the total charging power of cluster m; Equation (17) represents the charging energy constraint of electric vehicles, is the total energy of cluster m;

[0045] After establishing the optimization variables and related constraints in cluster units, they are brought into the real-time optimization scheduling model for solution to obtain the optimal charging power of the cluster in the current time period. The power Assign to the vehicles in the cluster, update the upper and lower limits of the power and power of each vehicle, and then update the upper and lower limits of the power and power of the cluster again;

[0046] According to the energy buffer consistency algorithm, the optimal power redistribution strategy at each moment is formulated as follows:

[0047]

[0048]

[0049] Among them, λ m represents the energy buffer factor of cluster m, It means optimizing power through the allocated single car; distributing power according to the consistency of energy buffer factor, so that the power status of different cars gradually tends to be consistent and fully charged when leaving; when and When equal, is 0;

[0050] Solve the linear equations by equations (18) and (19) to obtain all To get According to formula (20), the actual charging power is obtained.

[0051]

[0052] The charging power Used to indicate whether the power exceeds the limit; if there is a power limit, the sum of the optimized power of a single car after verification is not equal to the optimized power of the cluster. At this time, the single car that has exceeded the limit will no longer participate in the next round of allocation. The upper or lower limit of the power will be directly assigned, and the total power will be subtracted from the allocated power to obtain the remaining power, which will be allocated again according to the energy buffer consistency algorithm until there is no power remaining.

[0053] Preferably, the optimization problem of cluster electric vehicle charging and power grid unit output is constructed as a Markov decision process to determine the state space quantity, action space quantity and reward corresponding to each moment, including:

[0054] Construct the Markov decision process to maximize the cumulative reward G t for:

[0055]

[0056] The maximum cumulative reward G t Describes the total reward from time t until the end of the round; where R t and R t+1 Represent the rewards at the current moment and the next moment respectively, γ represents the discount factor, which determines the impact of future reward values ​​on the present, and is generally a number between 0 and 1;

[0057] Establish the V value and Q value in the Markov decision process, where the V value represents the reward and expected value in state S until the end of the round, and the Q value represents the reward and expected value after selecting action a until the end of the round, then:

[0058] V π (s) = E π [G t |S t =s] (22)

[0059] Q π (s,a)=E π [G t |S t =s,a t =a] (23)

[0060] Among them, the strategy π(a|s) represents the mapping from state s to action a, S t and a t Respectively represent the state and action taken at time t;

[0061] The Bellman equation for V and Q is shown below:

[0062]

[0063]

[0064] Among them, s' represents the state at the next moment, a' represents the action at the next moment, Represents the current reward obtained under state s and action a;

[0065] The state space S of the Markov decision process is defined as:

[0066]

[0067] in, and are the lower and upper output limits of each unit respectively. and are the lower and upper limits of the power consumption of the cluster respectively. and are the lower and upper limits of the charging power of the cluster, respectively, D,tis the grid load;

[0068] Define the action space a of the Markov decision process at time t as a t :

[0069]

[0070] Among them, p i,t is the unit output size, Charging power for the cluster;

[0071] Unit output size p i,t Or the charging power of the cluster If it exceeds its own upper and lower limit constraints, then t Make the following adjustments:

[0072]

[0073] in, Indicates the actual action value after adjustment;

[0074] Define the reward function R of the Markov decision process at time t as R t :

[0075] R t =R 1t +R 2t

[0076]

[0077] Among them, c is the penalty coefficient greater than 0, N m is the number of clusters.

[0078] Preferably, the method of solving the Markov decision process using a reinforcement learning algorithm and training a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage includes:

[0079] Using a maximum entropy-based deep reinforcement learning algorithm, defining the goal of the maximum entropy-based deep reinforcement learning algorithm, including reward and entropy, then:

[0080]

[0081] H(π(·|s'))=-E a logπ(a'|s') (31)

[0082] Among them, π *Represents the optimal strategy, H represents the entropy function, α is the temperature parameter, and represents the importance of the entropy function relative to the reward function. By embedding entropy into the calculation, the Soft Bellman equation and Soft Bellman update equation under the deep reinforcement learning algorithm based on maximum entropy are obtained as Equation (32) and Equation (33), respectively:

[0083]

[0084]

[0085] Therefore, from formula (33), the conversion formula of V value and Q value under the deep reinforcement learning algorithm based on maximum entropy can be obtained as follows:

[0086]

[0087] The maximum entropy-based deep reinforcement learning algorithm updates the strategy by using KL divergence, as shown in formula (35):

[0088]

[0089] Where π' represents the next stage strategy, the denominator Z(s t ) is used for normalization;

[0090] Assuming that the charging power is continuously adjustable, the deep reinforcement learning algorithm based on maximum entropy is used to iteratively solve the real-time optimization scheduling problem of electric vehicle charging and unit output in the continuous action space and state space to train the intelligent agent;

[0091] The maximum entropy-based deep reinforcement learning algorithm is based on the Actor-Critic framework, and uses two neural networks as the policy network and the value network respectively, which are used to fit the policy π and Q value, and update the parameters of the policy network and the value network respectively according to formula (34) and formula (35); wherein the policy network is used to output the probability distribution of the action, which is mainly represented by a Gaussian distribution; the value network is used to evaluate the quality of the action, which is mainly composed of one or more fully connected neural layers, and finally outputs the estimated Q value; after obtaining the value network output, the policy network continuously modifies the strategy according to the value network valuation, thereby obtaining a reinforcement learning agent;

[0092] The strategy network and value network have the same structure, but different parameter update frequencies.

[0093] Preferably, the reinforcement learning agent is applied to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period, including:

[0094] Obtaining charging information of each newly connected electric vehicle, and transmitting the charging information to the reinforcement learning agent;

[0095] The reinforcement learning agent updates the state variables according to the charging information, and outputs the optimal charging power of the cluster in the current period through the maximum entropy deep reinforcement learning algorithm according to the reward function R of the Markov decision process; according to the optimized power redistribution strategy, the reinforcement learning agent distributes the optimized power to each car in the cluster, and obtains the optimal charging strategy for each electric vehicle cluster in the current period;

[0096] Enter the next state and repeat the above two steps until an optimization cycle is completed.

[0097] Preferably, the method further comprises:

[0098] The feasibility and efficiency of the real-time optimization scheduling method for distributed electric vehicles are verified through examples.

[0099] According to another aspect of the present invention, a distributed electric vehicle real-time optimization scheduling system is provided, comprising:

[0100] The optimization dispatch model module establishes a real-time optimization dispatch model combining electric vehicle charging and power grid unit output;

[0101] Cluster optimization allocation module, which divides electric vehicles with the same charging end time into the same cluster, establishes an equivalent electric vehicle cluster energy boundary model and optimizes the power redistribution strategy;

[0102] Decision-making building module: This module constructs the optimization problem of cluster electric vehicle charging and grid unit output as a Markov decision process, and determines the state space quantity, action space quantity and reward corresponding to each moment;

[0103] A reinforcement learning agent module, which uses a reinforcement learning algorithm to solve the Markov decision process and trains a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage;

[0104] The optimal strategy module applies the reinforcement learning agent to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period.

[0105] According to a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-described methods or to run the above-described system when executing the program.

[0106] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the above-mentioned methods, or to run the above-mentioned system.

[0107] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0108] The real-time optimization scheduling method, system, terminal and medium for decentralized electric vehicles provided by the present invention are based on centralized optimization and take aggregator management as the organizational model. Taking into account the problem of dimensionality curse caused by using a single electric vehicle as a variable, an EV cluster equivalent model and an optimized power redistribution method are proposed based on the energy boundary model of a single vehicle, which effectively reduces the variable dimension and improves the model solving efficiency, making it more suitable for actual conditions.

[0109] Compared with traditional large-scale operations optimization, the distributed electric vehicle real-time optimization scheduling method, system, terminal and medium provided by the present invention are based on reinforcement learning, do not need to rely on accurate models, and do not need to accurately predict the uncertain factors in the system (the actual situation cannot be accurately predicted), such as electric vehicle charging methods and grid loads. Even in the case of environmental changes, the near-optimal electric vehicle charging power and unit output plan can be calculated at a faster speed, achieving the effect of saving grid operation costs, and considering demand response subsidies, it can also improve the benefits of aggregators and car owners.

[0110] The real-time optimization scheduling method, system, terminal and medium of the distributed electric vehicle provided by the present invention, and the specific application of the maximum entropy-based deep reinforcement learning algorithm (SAC algorithm) can realize continuous adjustment of charging power instead of discrete adjustment, and is more flexible. Compared with other reinforcement learning algorithms, SAC has the characteristic of Maximum Entropy, which enables the intelligent body to explore the environment to a greater extent, which is not only conducive to the intelligent body learning complex new tasks, but also has strong anti-interference ability and better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0112] Figure 1 The present invention is a flowchart of a method for real-time optimization and scheduling of distributed electric vehicles in one embodiment of the present invention.

[0113] Figure 2 This is a flow chart of a method for real-time optimization and scheduling of distributed electric vehicles in a preferred embodiment of the present invention.

[0114] Figure 3 This is a schematic diagram of reinforcement learning in a preferred embodiment of the present invention.

[0115] Figure 4 This is a flow chart of the SAC algorithm in a preferred embodiment of the present invention.

[0116] Figure 5 Schematic diagram of the components of a distributed electric vehicle real-time optimization scheduling system in one embodiment of the present invention. DETAILED DESCRIPTION

[0117] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0118] An embodiment of the present invention provides a real-time optimization scheduling method for distributed electric vehicles. The method addresses the defects existing in the prior art, avoids the problem of "curse of dimensionality", ensures that the centralized optimization scheduling involving a large number of distributed electric vehicles can be efficiently solved, and realizes flexible adjustment and optimization speed of the charging power of electric vehicles under the reinforcement learning algorithm, so as to provide a distributed electric vehicle optimization scheduling method that adapts to real-time scheduling needs.

[0119] The technical problems to be solved by the distributed electric vehicle real-time optimization scheduling method provided in this embodiment are mainly reflected in the following points:

[0120] (1) Quantify the actual benefits of electric vehicles participating in demand response to the power grid. Considering that demand response strategies such as time-of-use electricity prices guide electric vehicle users to charge in an orderly manner, which will bring another evening charging peak, this paper introduces a power grid unit economic dispatch model, combines unit output optimization dispatch with electric vehicle charging optimization dispatch, and quantifies the economic benefits of electric vehicles to the power grid.

[0121] (2) Centralized optimization of a large number of decentralized electric vehicles will cause the curse of dimensionality, which will seriously affect the solution efficiency or even make it impossible to solve. To solve this problem, it is necessary to reduce the dimension of the optimization scheduling problem of large-scale decentralized electric vehicles. This problem can be solved by dividing the electric vehicles into clusters to reduce the dimension of the optimization variables. The way the clusters are divided also affects the effect of the model.

[0122] (3) Electric vehicle cluster model, including how to obtain the cluster equivalent model and how to redistribute the optimized power. The boundary conditions of the cluster equivalent model must ensure that it does not exceed the actual effect that can be achieved after the superposition of multiple vehicles; after obtaining the optimal cluster power, the optimized power allocated to each vehicle should also ensure that it does not exceed the boundary conditions of a single vehicle. Therefore, it is very necessary to establish an accurate and reasonable cluster equivalent model and an optimized power redistribution model.

[0123] (4) In the real-time stage, the uncertainty and randomness of electric vehicle charging behavior are high. The advantages of reinforcement learning algorithms that do not require model building and are instead driven by data are gradually becoming more significant. However, the effectiveness of reinforcement learning algorithms in performing optimization scheduling is closely related to the selection of specific algorithms. While ensuring the optimization speed, the feasibility of the model and the flexibility of power regulation must also be guaranteed.

[0124] Figure 1 A flow chart of a real-time optimization scheduling method for distributed electric vehicles provided in one embodiment of the present invention.

[0125] like Figure 1 As shown, the real-time optimization scheduling method for distributed electric vehicles provided in this embodiment may include the following steps:

[0126] S100, establishes a real-time optimization dispatch model combining electric vehicle charging and grid unit output;

[0127] S200, dividing electric vehicles with the same charging end time into the same cluster, establishing an equivalent electric vehicle cluster energy boundary model and optimizing the power redistribution strategy; (In this step, all electric vehicles are divided into several clusters, and in subsequent steps, the control of a single electric vehicle may no longer be considered, but only the control of these clusters may be considered. Establishing an equivalent electric vehicle cluster means that the charging model of the electric vehicle is equivalent before and after the cluster is divided);

[0128] S300, constructs the optimization problem of cluster electric vehicle charging and grid unit output as a Markov decision process, and determines the state space quantity, action space quantity and reward corresponding to each moment;

[0129] S400, solving the Markov decision process using a reinforcement learning algorithm, and training a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage;

[0130] S500, applying the reinforcement learning agent to real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period.

[0131] In this embodiment, the following steps may also be included:

[0132] S600, verifying the feasibility and efficiency of the real-time optimization scheduling method for distributed electric vehicles through examples.

[0133] The real-time optimization and scheduling method for distributed electric vehicles provided by the above-mentioned embodiment of the present invention can not only efficiently handle the "curse of dimensionality" problem that occurs in centralized optimization through electric vehicle cluster division, but also make full use of the reinforcement learning algorithm to achieve flexible and adjustable action space under the premise of ensuring the optimization speed, thereby reducing the overall operation cost of the power grid. This real-time optimization and scheduling method is based on the power unit startup and shutdown plan in the power grid that has been determined a few days ago. In the real-time stage, the power grid optimizes the economic output of the unit and the charging of large-scale electric vehicles. This method optimizes the scheduling of electric vehicle charging and unit output plans without relying on the model, has a fast solution speed and good solution optimality, and meets the needs of real-time scheduling of the power grid.

[0134] The technical solution provided by the above embodiment of the present invention is further described in detail below in conjunction with the accompanying drawings and preferred embodiments. The step numbers in the embodiments of the present invention are only set for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0135] Figure 2 A flow chart of a real-time optimization scheduling method for distributed electric vehicles provided in a preferred embodiment of the present invention.

[0136] like Figure 2 As shown, the real-time optimization scheduling method for distributed electric vehicles provided in the preferred embodiment may include the following steps:

[0137] Step 1: Taking the minimum unit fuel cost as the optimization goal, a real-time optimization scheduling model combining single electric vehicle charging and grid unit output is established.

[0138] Step 2: To reduce the variable dimension, electric vehicles with the same charging end time are divided into the same cluster, and an equivalent cluster energy boundary model and an optimized power redistribution method are proposed.

[0139] Step 3: Construct the optimization problem of cluster electric vehicle charging and unit output into a Markov decision process to determine the state space quantity, action space quantity and reward corresponding to each moment.

[0140] Step 4: Use the maximum entropy-based deep reinforcement learning algorithm Soft Actor-Critic (SAC) to solve the Markov decision process, and obtain a reinforcement learning agent (Agent) that can adapt to the uncertainty of charging load in the real-time stage through a large number of scenario training.

[0141] Step 5: Apply the trained intelligent agent to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period, and verify the feasibility and efficiency of the real-time optimization scheduling method of electric vehicles through examples.

[0142] As a preferred embodiment, step 1: establishing a real-time optimization scheduling model for combining electric vehicle charging and power grid unit output may include the following steps:

[0143] Step 1.1: Considering the huge number of electric vehicles scattered in the city and the difficulty in management, in order to improve the efficiency of managing large-scale decentralized electric vehicles in real time, the power grid generally introduces electric vehicle aggregators as information and energy coordinators between the power grid and electric vehicle users, and indirectly manages electric vehicles through electric vehicle aggregators. Aggregators are generally car companies or charging pile companies. In order to make full use of the flexibility of electric vehicle charging to support the operation of the power grid and reduce the impact of large-scale charging on the power grid, the power grid sends demand response invitations to car owners through aggregators to dispatch the charging process.

[0144] Aggregators divide electric vehicles into flexible EVs and inflexible EVs based on whether electric vehicle users are willing to accept scheduling and whether they have adjustable charging capabilities. Flexible EVs refer to EVs that are willing to accept scheduling and have adjustable capabilities (i.e., the stay time is greater than the time required for full charging), and the rest are inflexible EVs. For inflexible EVs, aggregators are only an electricity provider. Once connected to the grid, they charge at maximum power until they are fully charged. On the one hand, aggregators upload real-time vehicle information (including flexible EVs and inflexible EVs) to the grid, and on the other hand, they control vehicles according to the control signals of the grid.

[0145] Step 1.2: After obtaining the unit start-up and shutdown plan determined by the day-ahead optimization, the grid level needs to coordinate the unit output and electric vehicle charging, and make full use of the flexibility of charging to reduce the overall fuel cost of the grid units and improve the economic efficiency of grid operation. The optimization objectives of the joint real-time optimization scheduling model of electric vehicle charging and grid unit output are as follows:

[0146]

[0147] The objective function is to minimize the fuel cost of the generator sets in the system, where N G is the number of generating units, T is the number of time periods in an optimization cycle, and p i,t is the power generation of unit i at time t; u i,t It is a binary number determined according to the on / off plan of the day before. It is 1 when the unit is in operation and 0 when the unit is in shutdown. i CB i and CC i is the power generation cost parameter of unit i.

[0148] The optimization model includes the following constraints:

[0149]

[0150] p i,t -p i,t-1 ≤u i,t-1 P up,i (3)

[0151] p i,t-1 -p i,t ≤u i,t-1 P down,i (4)

[0152]

[0153] Wherein formula (2) is the power generation power p of unit i i,t Upper and lower bound constraints, where and They represent the lower and upper limits of the power of unit i at time t, which are not only limited by the unit's own capacity, but also affected by the ramp constraint. Constraints (3) and (4) are the unit ramp constraints, where P up,i and P down,i are the maximum ramp-up rate and the maximum ramp-down rate of unit i, respectively. Constraint (5) represents the system power balance, where P D,t Including the predicted conventional load and non-flexible EV load, N EV represents the total number of flexible electric vehicles participating in the dispatch, represents the charging power of electric vehicle k participating in the dispatch, P loss,t is the network loss of the system at time t.

[0154] The following are the constraints related to a single electric vehicle involved in the dispatch:

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161] Formula (6) represents the electric vehicle charging energy constraint, where represents the power of electric vehicle k at the end of time period t, η represents the charging efficiency, and Δt represents the unit time period of the optimization cycle. Equations (7) and (8) represent the upper and lower energy constraints and charging power upper and lower constraints of electric vehicle k at time t, respectively. Equation (9) ensures that the EV is fully charged when leaving, and the upper and lower limits of the power are equal to the required power, where t end represents the time when electric car k leaves, represents the amount of electricity required by electric vehicle k when it leaves. Formula (10) and Formula (11) further restrict the lower limit and upper limit of charging power of electric vehicle k at time t, respectively. represents the maximum charging power limited by the charging pile or the battery itself. Formula (10) represents the power at the previous moment Lower than the current minimum power limit When the charging power lower limit is forced to increase; Formula (11) represents the power at the previous moment Close to the current power limit , the upper limit of charging power will be forced to decrease. Therefore, a charging energy boundary model for a single car can be introduced: the upper bound of energy means that the car is charged at the maximum power at the beginning, and maintains the maximum energy after being fully charged until leaving. The lower bound of energy means that the car delays charging, and continues to charge at the maximum power in the first few time periods before leaving and is fully charged at the time of leaving. In order to ensure that the vehicle reaches the required power when leaving, the vehicle's power change curve can only fluctuate within the upper and lower energy boundaries.

[0162] At this point, the objective function of the joint real-time optimization scheduling model for electric vehicle charging and power grid unit output is formula (1), and the constraints are formulas (2)-(11).

[0163] As a preferred embodiment, step 2: establishing an electric vehicle cluster charging energy boundary model and optimizing power redistribution method may include the following steps:

[0164] Step 2.1: There are a large number of electric vehicles scattered in the city. If each electric vehicle is used as an optimization variable for centralized optimization, it will cause the problem of "dimensionality curse", seriously affecting the efficiency of the solution, or even making it impossible to solve. To avoid this problem, the concept of electric vehicle cluster is introduced - electric vehicles with the same departure time are regarded as the same cluster, and the energy boundary model of a single vehicle is superimposed to obtain the energy boundary model of the cluster, and the relevant constraints (6)-(11) of a single vehicle in the optimization model of step 1 are replaced by the EV cluster constraints (12)-(17), as follows:

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Formula (12) to (15) respectively represent the lower limit constraint, upper limit constraint, lower limit constraint and upper limit constraint of charging power of electric vehicle cluster m at time t. m represents the number of electric vehicles in cluster m, and They represent the lower and upper limits of the power consumption of the cluster respectively. and They represent the lower and upper limits of the cluster’s charging power, respectively. Formula (16) indicates that the cluster power is equal to the sum of the powers of all the vehicles in the cluster. is the total charging power of cluster m. Equation (17) represents the electric vehicle charging energy constraint, is the total energy of cluster m.

[0172] In real-time optimization, it is impossible to accurately predict in advance which vehicles will be connected in the next period of time. That is, the energy boundary of the cluster needs to be updated in real time based on the latest information of electric vehicles connected to the cluster, and the updated information needs to be brought into the model for calculation.

[0173] After the electric vehicle cluster optimization model is established, the scale of cluster-related variables is only related to the number of time periods within the optimization cycle (for example, if the optimization cycle is 24 hours and the optimization time step is 15 minutes, a total of 96 clusters are required), but has nothing to do with the actual number of electric vehicles, which can effectively improve the model solution speed.

[0174] Step 2.2: After establishing the optimization variables and related constraints in cluster units, bring them into the real-time optimization scheduling model for solution, and you will get the optimal charging power of the cluster in the current time period. The power is distributed to the vehicles in the cluster. After the upper and lower limits of the power and power of a single vehicle are updated, the upper and lower limits of the power and power of the cluster are updated again. Since the power of the vehicles in the cluster varies when they start charging, the charging requirements are also different. If the power is simply evenly distributed, some vehicles with small charging requirements may be fully charged too early, or vehicles with large charging requirements may be fully charged later. If the owner picks up the car in advance, there is a risk of not being fully charged. To address this issue, in order to ensure that electric vehicles in the same cluster have relatively close energy states, the optimized power allocation method at each moment is formulated according to the energy buffer consistency algorithm as follows:

[0175]

[0176]

[0177] In formula (18), λ m represents the energy buffer factor of cluster m, It means that the power is optimized by the allocated single electric vehicle. The energy buffer factor of electric vehicles in a cluster is the same at the same time, but the energy buffer factor of the same cluster may be different at different times. This is because the power status of the cluster is different at different times. The power is allocated according to the consistency of the energy buffer factor. If the current power is far from the required power, the allocated power is larger, otherwise it is smaller. This ensures that the power status of different vehicles gradually tends to be consistent and can be fully charged when leaving. In particular, when and When equal, is 0. The linear equations can be solved by equations (18) and (19) to obtain all To get It is also necessary to verify according to formula (20) to obtain the actual charging power setting Prevent over-limit.

[0178]

[0179] If there is power that exceeds the limit, the sum of the power of a single vehicle after verification may not be equal to the optimized power of the cluster. In the case where there is still power remaining after the cluster power is allocated, the processing method is as follows: the single EV that has exceeded the limit will no longer participate in the next round of allocation, and the upper or lower limit of the power will be directly assigned. The total power minus the allocated power will be used to obtain the remaining power, and it will be allocated again according to the energy buffer consistency algorithm until there is no power remaining.

[0180] As a preferred embodiment, step 3: constructing the optimization problem of cluster electric vehicle charging and unit output into a Markov decision process may include the following steps:

[0181] Step 3.1: If Figure 3As shown in the figure, in reinforcement learning, the agent learns the ability to solve problems autonomously through interaction with the environment. The sequence problem to be solved can generally be represented by a Markov decision process (MDP), which means that the next state of the system is determined only by the current state and has nothing to do with the past state. In the Markov decision process, there are three important elements: S (State), a (Action) and R (Reward). The state S represents the environmental characteristics observed by the agent in the process of interacting with the environment, the action a represents the behavior taken by the agent, the actions generated by the agent and the changes in the environment itself will affect the state S, and the reward R represents the feedback from the environment to the agent. The better the feedback, the more rewards, and the agent will get as much reward as possible. Therefore, the whole process of reinforcement learning is: the agent observes the state S in the environment, inputs the state S into the agent, the agent calculates the next action a, and after outputting the action a, the environment will feedback the reward R to the agent and enter another state. In the real world, many problems cannot be expressed by a clear model, and the transition probability between different states is unknown. Reinforcement learning to solve such problems is called a non-model-based reinforcement learning method. The goal of the agent is to maximize the cumulative reward, the cumulative return G t Describes the total reward from time t until the end of the round, G t The specific expression is as follows:

[0182]

[0183] Where R t and R t+1 They represent the rewards at the current moment and the next moment respectively, and γ represents the discount factor, which determines the impact of future rewards on the present. It is generally a number between 0 and 1. k It can be seen that the farther the reward value is from the present moment, the smaller its impact on the present moment.

[0184] In order to help the agent make decisions, there are value functions and action-value functions in MDP, which are used to measure the value of state S and action a respectively, which can be called V value and Q value. V value represents the expected value of the reward and of the agent in state S until the end of the round, and Q value represents the expected value of the reward and after selecting action a until the end of the round. Then:

[0185] V π (s) = E π [G t |S t =s] (22)

[0186] Q π(s,a)=E π [G t |S t =s,a t =a] (23)

[0187] Among them, the strategy π(a|s) represents the mapping from state s to action a, S t and a t Respectively represent the state and action taken at time t;

[0188] The Bellman equation for V and Q is as follows:

[0189]

[0190]

[0191] Among them, s' represents the state at the next moment, a' represents the action at the next moment, Represents the current reward obtained under state s and action a;

[0192] The goal of reinforcement learning is to maximize the cumulative reward by optimizing the strategy, which is also the maximum value function. The Bellman optimal equation is the V value and Q value under the optimal strategy.

[0193] Step 3.2: Define the state space S of the Markov decision process. While the state space reflects environmental changes, the information contained cannot be redundant, otherwise it will affect the judgment of the intelligent agent. In the real-time scheduling of electric vehicles, the uncertainty of the environment is reflected in the difficulty in accurately predicting the charging behavior of electric vehicles that have not yet been connected, including the charging start time, charging end time and required power. In combination with the electric vehicle cluster model proposed above, the system state space is defined as follows:

[0194]

[0195] S mainly includes the upper and lower limits of each unit output and The upper and lower limits of electric vehicle cluster energy and Upper and lower limits of electric vehicle cluster power and and the grid load P D,t .

[0196] Step 3.3: Define the action space a of the Markov decision process. In the real-time scheduling of electric vehicles, the main dispatchable variables include the unit output size p i,t and charging power of EV cluster Therefore, the action space a of the system at time t is defined t as follows:

[0197]

[0198] If the unit output or EV cluster charging power calculated by the agent exceeds its own upper and lower limit constraints, it is necessary to adjust a according to formula (28): t Adjust to be between the upper and lower limits, where Indicates the actual action value after adjustment.

[0199]

[0200] Step 3.4: Define the reward function R of the Markov decision process. The reward represents the environment's score on the agent's behavior, and the agent's goal is to maximize the cumulative reward. In the real-time scheduling of electric vehicles, the objective function (1) is to minimize the fuel cost. In combination with the characteristics of reinforcement learning, the inverse of (1) is added to the reward expression. In addition, in order to ensure that the equality constraint of (5) holds, a power imbalance penalty term is added to the reward function, so the reward function R at time t is t The expression is as follows:

[0201] R t =R 1t +R 2t

[0202]

[0203] Where c is the penalty coefficient greater than 0, N m is the number of EV clusters.

[0204] As a preferred embodiment, step 4, using the maximum entropy-based deep reinforcement learning algorithm Soft Actor-Critic (SAC) to solve, can include the following steps:

[0205] Step 4.1: SAC is an off-policy reinforcement learning algorithm based on Maximum Entropy. SAC is based on the Actor-Critic framework. The goal of the agent is to maximize the cumulative reward and entropy (the greater the entropy, the greater the randomness of random events), that is, to complete the task as randomly as possible, avoiding repeatedly choosing several actions with high Q values ​​and falling into suboptimal solutions, thereby increasing the training speed.

[0206] The objectives of the SAC algorithm include reward and entropy, as follows:

[0207]

[0208] H(π(·|s'))=-E a logπ(a'|s') (31)

[0209] Among them, π * Represents the optimal strategy, H represents the entropy function, α is the temperature parameter, and represents the importance of the entropy function relative to the reward function. By embedding entropy into the calculation, the Soft Bellman equation and Soft Bellman update equation under the deep reinforcement learning algorithm based on maximum entropy are obtained as Equation (32) and Equation (33), respectively:

[0210]

[0211]

[0212] The conversion formula for V value and Q value under the SAC algorithm is as follows:

[0213]

[0214] The maximum entropy-based reinforcement learning algorithm updates the strategy by using KL divergence, as shown in formula (35):

[0215]

[0216] Where π' represents the next stage strategy, the denominator Z(s t ) is used for normalization;

[0217] Based on the above, the SAC strategy update steps are as follows:

[0218] (1) Keep the strategy unchanged and update the Q value according to the Soft Bellman update equation (34) until convergence;

[0219] (2) Update the strategy according to formula (35).

[0220] Step 4.2: Use SAC to iteratively solve the real-time optimization scheduling problem of electric vehicles and units and train the intelligent agent. Under the assumption that the charging power is continuously adjustable, that is, in the continuous action space and state space, the SAC algorithm is used to solve the optimization problem. The SAC algorithm is based on the Actor-Critic framework and uses two neural networks as the value network and the policy network to fit the Q value and the policy π, and updates the neural network parameters according to equations (34) and (35). The policy network is used to output the probability distribution of the action, which is generally represented by the mean and variance of the Gaussian distribution; the value network is used to evaluate the quality of the action, which is generally composed of several fully connected neural layers, and finally outputs the estimated Q value. After obtaining the output of the value network, the policy network continuously modifies the strategy according to the valuation of the value network, so that the performance of the intelligent agent is getting better and better. At the same time, because the output of one neural network is used to correct another neural network, it is easy to have difficulty in convergence. In order to solve this problem, reinforcement learning usually constructs two sets of neural networks with the same structure but different parameter update frequencies to promote effective learning. In the process of training neural networks, an experience replay pool is often used to store a certain amount of experience data. During training, some data is randomly extracted in batches from the experience replay pool to train the neural network.

[0221] like Figure 4 As shown, the specific steps of the SAC algorithm to train the agent are as follows:

[0222] (1) Initialize the parameters of the policy network (φ), the value network (θ1) and the target value network (θ2): Clear the experience replay pool (D);

[0223] (2) Get the status s of the current environment t , and obtain action a according to the distribution sampling output by the policy network φ t ;

[0224] (3) Output action a to the environment t , and get the state s at the next moment t+1 and rewards And the data Deposit into the experience replay pool:

[0225] (4) Repeat steps (2) and (3) until the amount of data in the experience replay pool is greater than the required batch size, take out batches of data from the experience replay pool for neural network update, and proceed to step (5);

[0226] (5) Update the Q value according to formula (36):

[0227]

[0228] (6) Update the strategy network φ according to formula (37):

[0229]

[0230] (7) Automatically adjust the temperature parameter α. When exploring a new area, a larger temperature coefficient should be adopted to allow the agent to explore more space. When the optimal solution is about to be determined, the temperature coefficient should be reduced to accelerate the convergence of the algorithm.

[0231] (8) Update the parameters of the value network (θ1) and the target value network (θ2):

[0232]

[0233] Where τ is the update rate parameter. The larger τ is, the greater the actual Q value θ is. i Value network parameters The faster the transmission.

[0234] (9) Repeat the above steps (2) to (8) until the algorithm converges and forms the trained θ1, θ2 and φ.

[0235] As a preferred embodiment, step 5: applying the trained agent to the real-time optimization scheduling of distributed electric vehicles may include the following steps:

[0236] Step 5.1: Deploy the trained agent on the charging management platform of each aggregator. The charging management platform is the receiver of vehicle information and grid information, and also the sender of charging control signals, with efficient storage and computing functions. Whenever a new electric vehicle is connected, the charging management platform will record the charging information of the electric vehicle (departure time, required power) and transmit the information to the agent.

[0237] Step 5.2: The agent updates the state variables such as the upper and lower charging boundaries of the EV cluster according to the newly received real-time information, and outputs the optimal charging power of the cluster in the current period based on the objective function (29) and the SAC reinforcement learning algorithm. According to the power redistribution algorithm, the agent distributes the optimized power to each vehicle in the cluster, which is then sent to each intelligent remote-controlled charging pile by the charging management platform, and the intelligent remote-controlled charging pile executes the charging plan.

[0238] Step 5.3: Enter the next state and repeat steps 5.1 and 5.2 until an optimization cycle is completed.

[0239] Figure 5 A schematic diagram of the components of a distributed electric vehicle real-time optimization and scheduling system provided in one embodiment of the present invention.

[0240] like Figure 5 As shown, the distributed electric vehicle real-time optimization scheduling system provided by this embodiment may include: an optimization scheduling model module, a cluster optimization allocation module, a decision building module, a reinforcement learning agent module and an optimal strategy module. Among them:

[0241] The optimization dispatch model module establishes a real-time optimization dispatch model combining electric vehicle charging and power grid unit output;

[0242] Cluster optimization allocation module, which divides electric vehicles with the same charging end time into the same cluster, establishes an equivalent electric vehicle cluster energy boundary model and optimizes the power redistribution strategy;

[0243] Decision-making building module: This module constructs the optimization problem of cluster electric vehicle charging and grid unit output as a Markov decision process, and determines the state space quantity, action space quantity and reward corresponding to each moment;

[0244] A reinforcement learning agent module, which uses a reinforcement learning algorithm to solve the Markov decision process and trains a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage;

[0245] The optimal strategy module applies the reinforcement learning agent to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period.

[0246] An embodiment of the present invention provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any of the methods in the above embodiments, or to run the system in the above embodiments.

[0247] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0248] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0249] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0250] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0251] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute the method described in any one of the above embodiments, or to run the system described in the above embodiments.

[0252] The real-time optimization scheduling method, system, terminal and medium of distributed electric vehicles based on reinforcement learning provided by the above-mentioned embodiments of the present invention first solve the problem of too high variable dimension in the optimization model through the EV cluster equivalent model and the optimization power redistribution method, and then construct the optimization problem as a Markov decision process, and solve it using the reinforcement learning algorithm (preferably a deep learning algorithm based on maximum entropy (Soft Actor-Critic, SAC)). Compared with traditional large-scale operations optimization, the real-time optimization scheduling using the reinforcement learning algorithm does not need to rely on an accurate model, and can calculate the near-optimal electric vehicle charging power and unit output plan at a faster speed. This method fully taps the flexibility of electric vehicles, realizes the economic operation of the power grid, and can also improve the benefits of aggregators and car owners when considering demand response subsidies. And the Maximum Entropy feature of SAC allows the intelligent body to explore the environment to a greater extent, which is not only conducive to the intelligent body learning complex new tasks, but also has strong anti-interference ability and better robustness.

[0253] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0254] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0255] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A real-time optimization scheduling method for distributed electric vehicles, characterized in that: include: Establish a real-time optimization dispatch model that combines electric vehicle charging and grid unit output; Electric vehicles with the same charging end time are divided into the same cluster, an equivalent electric vehicle cluster energy boundary model is established, and the power redistribution strategy is optimized; The optimization problem of cluster electric vehicle charging and power grid unit output is constructed as a Markov decision process to determine the state space quantity, action space quantity and reward corresponding to each moment; The Markov decision process is solved by using a reinforcement learning algorithm, and a reinforcement learning agent is trained to adapt to the uncertainty of charging load in the real-time stage; Applying the reinforcement learning agent to real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period; The real-time optimization dispatching model for combining electric vehicle charging and power grid unit output is established, including: Taking the lowest unit fuel cost as the optimization goal of the real-time optimization scheduling model, the optimization objective function is: Among them, N G is the number of generating units, T is the number of time periods in an optimization cycle, and p i,t is the power generation of unit i at time t; u i,t It is a binary number determined according to the on / off plan of the day before. It is 1 when the unit is in operation and 0 when the unit is in shutdown. i CB i and CC i is the power generation cost parameter of unit i; The constraints of the real-time optimization scheduling model are set as follows: p i,t -p i,t-1 ≤u i,t-1 P up,i (3) p i,t-1 -p i,t ≤u i,t-1 P down,i (4) Wherein, formula (2) is the power generation power p of unit i i,t Upper and lower bound constraints, where and They represent the lower and upper limits of the power of unit i at time t, which are not only limited by the unit's own capacity, but also affected by the unit's climbing constraints. Formulas (3) and (4) are the unit's climbing constraints, where P up,i and P down,i are the maximum ramp-up rate and the maximum ramp-down rate of unit i respectively; Formula (5) represents the system power balance, where P D,t Including the predicted conventional load and non-flexible EV load, N EV represents the total number of flexible electric vehicles participating in the dispatch, represents the charging power of electric vehicle k participating in the dispatch, P loss,t is the network loss of the system at time t; The relevant constraints for a single electric vehicle participating in the dispatch are set as follows: Among them, formula (6) represents the electric vehicle charging energy constraint, where represents the power of electric vehicle k at the end of time period t, η represents the charging efficiency, and Δt represents the unit time period of the optimization cycle; Formula (7) and Formula (8) represent the upper and lower limit constraints of energy and charging power of electric vehicle k at time t respectively; Formula (9) ensures that the EV is fully charged when leaving, and the upper and lower limits of power are equal to the required power, where t end represents the time when electric car k leaves, represents the amount of electricity required by electric vehicle k when it leaves; Equations (10) and (11) further restrict the lower and upper limits of the charging power of electric vehicle k at time t, respectively. represents the maximum charging power limited by the charging pile or the battery itself; Formula (10) represents the power at the previous moment Lower than the current minimum power limit When the charging power lower limit will be forced to increase; Formula (11) represents the power at the previous moment Approaching the current battery limit When the vehicle leaves the vehicle, the upper limit of the charging power will be forced to be lowered; a charging energy boundary model for a single vehicle is introduced, where the upper bound of energy means that the vehicle is charged at the maximum power at the beginning, and maintains the maximum energy after being fully charged until leaving; the lower bound of energy means that the vehicle delays charging, and continues to charge at the maximum power in the first few time periods before leaving and is fully charged at the time of leaving; the vehicle's power change curve can only fluctuate within the upper and lower bounds of energy; At this point, through the optimization objective formula (1) and the constraint condition formulas (2) to (11), the real-time optimization scheduling model for the combined electric vehicle charging and power grid unit output is obtained; The method of dividing electric vehicles with the same charging end time into the same cluster, establishing an equivalent electric vehicle cluster energy boundary model, and optimizing the power redistribution strategy includes: The electric vehicles with the same departure time are regarded as the same cluster, and the charging energy boundary models of individual vehicles are superimposed to obtain the energy boundary model of the electric vehicle cluster. The constraints related to the individual electric vehicles are replaced with the constraints related to the cluster, then: Among them, equations (12) to (15) represent the lower limit constraint, upper limit constraint, lower limit constraint and upper limit constraint of charging power of electric vehicle cluster m at time t, respectively. m represents the number of electric vehicles in cluster m, and They represent the lower and upper limits of the power consumption of the cluster respectively. and They represent the lower and upper limits of the cluster’s charging power respectively; Formula (16) indicates that the cluster power is equal to the sum of the powers of all the vehicles in the cluster. is the total charging power of cluster m; Formula (17) represents the electric vehicle charging energy constraint, is the total energy of cluster m; After establishing the optimization variables and related constraints in cluster units, they are brought into the real-time optimization scheduling model for solution to obtain the optimal charging power of the cluster in the current time period. The power Assign to the vehicles in the cluster, update the upper and lower limits of the power and power of each vehicle, and then update the upper and lower limits of the power and power of the cluster again; According to the energy buffer consistency algorithm, the optimal power redistribution strategy at each moment is formulated as follows: Among them, λ m represents the energy buffer factor of cluster m, It means optimizing power through the allocated single car; distributing power according to the consistency of energy buffer factor, so that the power status of different cars gradually tends to be consistent and fully charged when leaving; when and When equal, is 0; Solve the linear equations by equations (18) and (19) to obtain all To get According to formula (20), the actual charging power is obtained. The charging power Used to indicate whether the power exceeds the limit; if there is a power that exceeds the limit, the sum of the optimized power of a single car after verification is not equal to the optimized power of the cluster. At this time, the single car that has exceeded the limit will no longer participate in the next round of allocation. The upper or lower limit of the power is directly assigned, and the total power is subtracted from the allocated power to get the remaining power, and it is allocated again according to the energy buffer consistency algorithm until there is no remaining power; The reinforcement learning agent is applied to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period, including: Obtaining charging information of each newly connected electric vehicle, and transmitting the charging information to the reinforcement learning agent; The reinforcement learning agent updates the state variables according to the charging information, and outputs the optimal charging power of the cluster in the current period through the maximum entropy deep reinforcement learning algorithm according to the reward function R of the Markov decision process; according to the optimized power redistribution strategy, the reinforcement learning agent distributes the optimized power to each car in the cluster, and obtains the optimal charging strategy for each electric vehicle cluster in the current period; Enter the next state and repeat the above two steps until an optimization cycle is completed.

2. The real-time optimization scheduling method for distributed electric vehicles according to claim 1 is characterized in that: The optimization problem of cluster electric vehicle charging and power grid unit output is constructed as a Markov decision process to determine the state space quantity, action space quantity and reward corresponding to each moment, including: Construct the Markov decision process to maximize the cumulative reward G t for: The maximum cumulative reward G t Describes the total reward from time t until the end of the round; where R t and R t+1 Represent the rewards at the current moment and the next moment respectively, γ represents the discount factor, which determines the impact of future reward values ​​on the present, and its value is a number between 0 and 1; Establish the V value and Q value in the Markov decision process, where the V value represents the reward and expected value in state S until the end of the round, and the Q value represents the reward and expected value after selecting action a until the end of the round, then: V π (s)=E π [G t |S t =s] (22) Q π (s,a)=E π [G t |S t =s,a t =a] (23) Among them, the strategy π(a|s) represents the mapping from state s to action a, S t and a t Respectively represent the state and action taken at time t; The Bellman equation for V and Q is shown below: Among them, s' represents the state at the next moment, a' represents the action at the next moment, Represents the current reward obtained under state s and action a; The state space S of the Markov decision process is defined as: in, and are the lower and upper output limits of each unit respectively. and are the lower and upper limits of the cluster’s power consumption, respectively. and are the lower and upper limits of the charging power of the cluster, respectively, D,t is the grid load; Define the action space a of the Markov decision process at time t as a t : Among them, p i,t is the unit output size, Charging power for the cluster; Unit output size p i,t Or the charging power of the cluster If it exceeds its own upper and lower limit constraints, then t Make the following adjustments: in, Indicates the actual action value after adjustment; Define the reward function R of the Markov decision process at time t as R t : Among them, c is the penalty coefficient greater than 0, N m is the number of clusters.

3. The real-time optimization scheduling method for distributed electric vehicles according to claim 2 is characterized in that: The method of solving the Markov decision process by using a reinforcement learning algorithm and training a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage includes: Using a maximum entropy-based deep reinforcement learning algorithm, defining the goal of the maximum entropy-based deep reinforcement learning algorithm, including reward and entropy, then: H(π(·|s'))=-E a logπ(a'|s') (31) Among them, π * Represents the optimal strategy, H represents the entropy function, α is the temperature parameter, and represents the importance of the entropy function relative to the reward function. By embedding entropy into the calculation, the Soft Bellman equation and Soft Bellman update equation under the deep reinforcement learning algorithm based on maximum entropy are obtained as Equation (32) and Equation (33), respectively: Therefore, from formula (33), the conversion formula of V value and Q value under the deep reinforcement learning algorithm based on maximum entropy can be obtained as follows: The maximum entropy-based deep reinforcement learning algorithm updates the strategy through KL divergence, as shown in formula (35): Where π' represents the next stage strategy, the denominator Z(s t ) is used for normalization; Assuming that the charging power is continuously adjustable, the deep reinforcement learning algorithm based on maximum entropy is used to iteratively solve the real-time optimization scheduling problem of electric vehicle charging and unit output in the continuous action space and state space to train the intelligent agent; The maximum entropy-based deep reinforcement learning algorithm is based on the Actor-Critic framework, and uses two neural networks as the policy network and the value network respectively, which are used to fit the policy π and Q value, and update the parameters of the policy network and the value network respectively according to formula (34) and formula (35); wherein the policy network is used to output the probability distribution of the action, which is represented by a Gaussian distribution; the value network is used to evaluate the quality of the action, which is composed of one or more fully connected neural layers, and finally outputs the estimated Q value; after obtaining the value network output, the policy network continuously modifies the strategy according to the value network valuation, thereby obtaining a reinforcement learning agent; The strategy network and value network have the same structure, but different parameter update frequencies.

4. The real-time optimization scheduling method for distributed electric vehicles according to any one of claims 1 to 3, characterized in that: Also includes: The feasibility and efficiency of the real-time optimization scheduling method for distributed electric vehicles are verified through examples.

5. A distributed electric vehicle real-time optimization and dispatching system, characterized in that: include: The optimization dispatch model module establishes a real-time optimization dispatch model combining electric vehicle charging and power grid unit output; Cluster optimization allocation module, which divides electric vehicles with the same charging end time into the same cluster, establishes an equivalent electric vehicle cluster energy boundary model and optimizes the power redistribution strategy; Decision-making building module: This module constructs the optimization problem of cluster electric vehicle charging and grid unit output as a Markov decision process, and determines the state space quantity, action space quantity and reward corresponding to each moment; A reinforcement learning agent module, which uses a reinforcement learning algorithm to solve the Markov decision process and trains a reinforcement learning agent that adapts to the uncertainty of charging load in the real-time stage; An optimal strategy module, which applies the reinforcement learning agent to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period; The real-time optimization dispatching model for combining electric vehicle charging and power grid unit output is established, including: Taking the lowest unit fuel cost as the optimization goal of the real-time optimization scheduling model, the optimization objective function is: Among them, N G is the number of generating units, T is the number of time periods in an optimization cycle, and p i,t is the power generation of unit i at time t; u i,t It is a binary number determined according to the on / off plan of the day before. It is 1 when the unit is in operation and 0 when the unit is in shutdown. i CB i and CC i is the power generation cost parameter of unit i; The constraints of the real-time optimization scheduling model are set as follows: p i,t -p i,t-1 ≤u i,t-1 P up,i (3) p i,t-1 -p i,t ≤u i,t-1 P down,i (4) Wherein, formula (2) is the power generation power p of unit i i,t Upper and lower bound constraints, where and They represent the lower and upper limits of the power of unit i at time t, which are not only limited by the unit's own capacity, but also affected by the unit's climbing constraints. Formulas (3) and (4) are the unit's climbing constraints, where P up,i and P down,i are the maximum ramp-up rate and the maximum ramp-down rate of unit i respectively; Formula (5) represents the system power balance, where P D,t Including the predicted conventional load and non-flexible EV load, N EV represents the total number of flexible electric vehicles participating in the dispatch, represents the charging power of electric vehicle k participating in the dispatch, P loss,t is the network loss of the system at time t; The relevant constraints for a single electric vehicle participating in the dispatch are set as follows: Among them, formula (6) represents the electric vehicle charging energy constraint, where represents the power of electric vehicle k at the end of time period t, η represents the charging efficiency, and Δt represents the unit time period of the optimization cycle; Formula (7) and Formula (8) represent the upper and lower limit constraints of energy and charging power of electric vehicle k at time t respectively; Formula (9) ensures that the EV is fully charged when leaving, and the upper and lower limits of power are equal to the required power, where t end represents the time when electric car k leaves, represents the amount of electricity required by electric vehicle k when it leaves; Equations (10) and (11) further restrict the lower and upper limits of the charging power of electric vehicle k at time t, respectively. represents the maximum charging power limited by the charging pile or the battery itself; Formula (10) represents the power at the previous moment Lower than the current minimum power limit When the charging power lower limit will be forced to increase; Formula (11) represents the power at the previous moment Approaching the current battery limit When the vehicle leaves the vehicle, the upper limit of the charging power will be forced to be lowered; a charging energy boundary model for a single vehicle is introduced, where the upper bound of energy means that the vehicle is charged at the maximum power at the beginning, and maintains the maximum energy after being fully charged until leaving; the lower bound of energy means that the vehicle delays charging, and continues to charge at the maximum power in the first few time periods before leaving and is fully charged at the time of leaving; the vehicle's power change curve can only fluctuate within the upper and lower bounds of energy; At this point, through the optimization objective formula (1) and the constraint condition formulas (2) to (11), the real-time optimization scheduling model for the combined electric vehicle charging and power grid unit output is obtained; The method of dividing electric vehicles with the same charging end time into the same cluster, establishing an equivalent electric vehicle cluster energy boundary model, and optimizing the power redistribution strategy includes: The electric vehicles with the same departure time are regarded as the same cluster, and the charging energy boundary models of individual vehicles are superimposed to obtain the energy boundary model of the electric vehicle cluster. The constraints related to the individual electric vehicles are replaced with the constraints related to the cluster, then: Among them, equations (12) to (15) represent the lower limit constraint, upper limit constraint, lower limit constraint and upper limit constraint of charging power of electric vehicle cluster m at time t, respectively. m represents the number of electric vehicles in cluster m, and They represent the lower and upper limits of the power consumption of the cluster respectively. and They represent the lower and upper limits of the cluster’s charging power respectively; Formula (16) indicates that the cluster power is equal to the sum of the powers of all the vehicles in the cluster. is the total charging power of cluster m; Formula (17) represents the electric vehicle charging energy constraint, is the total energy of cluster m; After establishing the optimization variables and related constraints in cluster units, they are brought into the real-time optimization scheduling model for solution to obtain the optimal charging power of the cluster in the current time period. The power Assign to the vehicles in the cluster, update the upper and lower limits of the power and power of each vehicle, and then update the upper and lower limits of the power and power of the cluster again; According to the energy buffer consistency algorithm, the optimal power redistribution strategy at each moment is formulated as follows: Among them, λ m represents the energy buffer factor of cluster m, It means optimizing power through the allocated single car; distributing power according to the consistency of energy buffer factor, so that the power status of different cars gradually tends to be consistent and fully charged when leaving; when and When equal, is 0; Solve the linear equations by equations (18) and (19) to obtain all To get According to formula (20), the actual charging power is obtained. The charging power Used to indicate whether the power exceeds the limit; if there is a power that exceeds the limit, the sum of the optimized power of a single car after verification is not equal to the optimized power of the cluster. At this time, the single car that has exceeded the limit will no longer participate in the next round of allocation. The upper or lower limit of the power is directly assigned, and the total power is subtracted from the allocated power to get the remaining power, and it is allocated again according to the energy buffer consistency algorithm until there is no remaining power; The reinforcement learning agent is applied to the real-time optimization scheduling of distributed electric vehicles to obtain the optimal charging strategy for each electric vehicle cluster in each time period, including: Obtaining charging information of each newly connected electric vehicle, and transmitting the charging information to the reinforcement learning agent; The reinforcement learning agent updates the state variables according to the charging information, and outputs the optimal charging power of the cluster in the current period through the maximum entropy deep reinforcement learning algorithm according to the reward function R of the Markov decision process; according to the optimized power redistribution strategy, the reinforcement learning agent distributes the optimized power to each car in the cluster, and obtains the optimal charging strategy for each electric vehicle cluster in the current period; Enter the next state and repeat the above two steps until an optimization cycle is completed.

6. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 4, or to run the system described in claim 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 4, or to run the system described in claim 5.

Images