Electric power dispatching method with participation of electric vehicles

Abstract

The invention discloses a power dispatching method with participation of an electric vehicle, and belongs to the technical field of power dispatching, and the construction process of the method comprises the steps: constructing a traffic network-micro-grid coupling network model, and determining the target of power dispatching: minimizing the dispatching cost of the electric vehicle, and achieving the maximum load recovery of a micro-grid group; constructing a partially observable Markov decision process according to the constructed model and the power dispatching target; according to the constructed partially observable Markov decision process, an AD-MADDPG algorithm is designed; the AD-MADDPG algorithm is used for solving a power dispatching method which achieves a power dispatching target according to the constructed model; the AD-MADDPG algorithm is an improved algorithm in which an attention mechanism and a multi-explorer-single learner framework are combined on the basis of a DDPG algorithm. According to the invention, the capability of recovering operation of the power distribution system in emergency can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of electric power dispatching, and in particular relates to an electric power dispatching method involving electric vehicles. Background Art

[0002] As the global climate continues to warm, the occurrence of high-risk, low-probability extreme events poses a great challenge to the stable operation of the power system. In order to mitigate the impact of extreme events, improving the operational resilience of the power system (i.e., the ability of the power system to adapt to the ever-changing complex environment, withstand intentional attacks, accidents or naturally occurring extreme events and recover quickly) has become a research hotspot in recent years.

[0003] Multi-Microgrids (MMGs), a new type of intensive power network composed of distributed generation, energy storage system (ESS), demand response strategy and flexible load resources, with independent management, protection and control capabilities, is one of the solutions that are currently attracting much attention.

[0004] When a regional fault or power outage occurs in the large power grid, the microgrid group MMGs can operate in off-grid mode, deploying modular power system resources and utilizing distributed new energy, ESS and other resources to maintain continuous power supply to local end users. With the rapid development of digitalization and decentralization, electric vehicles (EVs) have been widely used in power systems due to their significant mobility and flexibility advantages. On the one hand, as one of the most common mobile power sources, a large number of electric vehicles can be summoned and deployed to appropriate locations in a short period of time to provide continuous power supply support for critical loads to enhance the resilience of the distribution network system. On the other hand, as one of the demand-side response technologies, electric vehicles can provide various auxiliary services, such as supply and demand balance and carbon emission reduction.

[0005] Among the existing studies on improving the operational resilience of distribution networks using electric vehicles, there is a scheme that models the optimal decision-making problem for the resilience recovery of distribution network systems as a mixed integer second-order cone model, and proposes an EVs incentive mechanism based on asymmetric Nash negotiation to encourage electric vehicle clusters to participate in the resilience recovery scheduling of distribution networks. There are also schemes that consider the cooperation between electric vehicle fleet operators and distribution system operators to guide available electric vehicles to damaged areas, maintain continuous power supply to critical loads, and improve the recovery capacity of distribution systems in emergency situations. A large number of extreme scenarios were generated by using Monte Carlo simulation technology, and the coordinated interactive optimization problem between EVs and distribution networks was modeled as a stochastic model based on mixed integer linear programming.

[0006] However, the above two solutions usually face a large computational burden to deal with environmental uncertainties through scenario-based stochastic programming or robust optimization methods, which may reduce the model's ability to respond to environmental changes. In addition, in the case of incomplete information between EVs and distribution networks, problem solving based on model methods usually cannot guarantee the accuracy of the solution.

[0007] As a model-free method, reinforcement learning provides a new perspective for solving sequential decision optimization problems. Without prior knowledge, the agent does not need to obtain the mathematical models and technical parameters in the model-based optimization method. It gradually learns the optimal control decision by gaining experience through iterative interaction with the environment. In addition, as an online learning method, reinforcement learning can effectively utilize the increasing data when interacting with the environment, capture the uncertainty of the system, and adapt to various state conditions. The reinforcement learning strategy that has converged in training can be directly deployed in the actual scene and output actions within a few milliseconds.

[0008] Among the existing reinforcement learning algorithms, a spatiotemporal network model of coupled transportation network-distribution network is constructed to solve the elastic load recovery problem of distribution network system considering mobile energy storage system (MESS). A reinforcement learning method based on Multi-agent double deep Q-networks (MADDQN) is proposed to optimize the routing and charging and discharging scheduling of MESS and improve the operation elasticity of the distribution network system.

[0009] In the existing reinforcement learning algorithms, the mobility and flexibility of EVs are used to improve the resilience of the power system. A hybrid parameter sharing proximal policy optimization (H-PSPPO) algorithm based on hybrid action space and parameter sharing is proposed to learn the optimal path selection and charging and discharging scheduling strategies of electric vehicles.

[0010] However, although the above two reinforcement learning methods improve the adaptability of the model, they simplify the movement process of EVs in the traffic network into discrete direction selection and do not consider the sequence characteristics of EVs at multiple time steps. In addition, the above reinforcement learning algorithms all adopt a multi-agent training framework and use parameter sharing technology to reduce the training complexity of the model, but they do not consider the mutual cooperation between multiple EVs. Summary of the invention

[0011] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for electric vehicle-involved power dispatching, which is used to improve the ability of a power distribution system to resume operation in an emergency.

[0012] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0013] In a first aspect, the present invention provides a method for dispatching electric vehicles, comprising:

[0014] S1: Construct a transportation network-microgrid coupling network model and determine the goal of power dispatching: minimize the dispatching cost of electric vehicles to achieve maximum load recovery of the microgrid group;

[0015] S2: construct a partially observable Markov decision process based on the model of step S1 and the power dispatch target;

[0016] S3: Design the AD-MADDPG algorithm based on the partially observable Markov decision process in step S2;

[0017] The AD-MADDPG algorithm is used to solve the power dispatching method that achieves the power dispatching target according to the model constructed in step S1;

[0018] The AD-MADDPG algorithm is an improved algorithm based on the DDPG algorithm that combines the attention mechanism and the multiple explorers-single learner framework.

[0019] In the aforementioned power dispatching method involving electric vehicles, the transportation network-microgrid coupling network model of step S1 includes a transportation network model and a microgrid model, the transportation network model is used to collect traffic information, the microgrid model is used to collect distribution network information, and the transportation network-microgrid coupling network model is used to seek the optimal mobile route and charging / discharging scheduling of electric vehicles to achieve the power dispatching target of step S1.

[0020] In the aforementioned electric vehicle-participated power dispatching method, the partially observable Markov decision process POMDP in step S2 is Time state space Defined as: , consists of three channels:

[0021] The first channel in the state space Includes geographical location of charging / discharging stations h and the microgrid load reduction associated with h ,Right now ; where h represents the charging / discharging station number, H represents the total number of charging / discharging stations, represents a collection of charging / discharging stations, and Respectively represent the horizontal and vertical coordinates of the charging / discharging station h in the map;

[0022] The second channel of the state space Including the coordinates of the vertex of the transportation network road and roads Traffic flow ,Right now ; Where V represents the total number of road vertices, represents the set of road vertices, represents the road vertex, , , Represents the total number of road vertices, Represented by vertex Starting point The road to the end, and Represents vertices The horizontal and vertical coordinates in the map;

[0023] Indicates the road exist Real-time traffic flow at all times, Real-time traffic flow at all times The calculation formula is as follows:

[0024] ,

[0025] in, Indicates the road Other types of vehicles with specific daily travel patterns, Indicates the road The number of electric vehicles participating in power dispatch;

[0026] The third channel of state space Includes the current location of the electric vehicle and remaining power ,Right now ;in Indicates the electric vehicle number, represents the total number of electric vehicles, represents the set of electric vehicle numbers, and Respectively indicate the number The horizontal and vertical coordinates of electric vehicles on the map;

[0027] Action space of the partially observable Markov decision process POMDP in step S2 Defined as ;

[0028] in, Indicates the moving direction of the electric vehicle, where the moving direction of the electric vehicle is the clockwise rotation angle of the front of the vehicle. Indicates the range of values ​​of the moving direction of the electric vehicle, represents the moving distance of the electric vehicle in time step t, represents the maximum moving distance of the electric vehicle within the time step t, represents the range of distance that the electric vehicle moves within the time step t, Indicates the maximum capacity of an EV relative to its battery The charge-discharge ratio, Indicates the maximum capacity of an EV relative to its battery The range of charge and discharge ratio, It means charging electric vehicles. A negative value indicates that the electric vehicle is discharging;

[0029] Step S2 is the partially observable Markov decision process POMDP numbered of electric vehicles in The instant reward function calculation formula at the moment is as follows:

[0030] ,

[0031] in, Indicator with charging / discharging station Associated microgrids load reduction; represents a preset unit load reduction cost factor; Indicates when the number When an electric vehicle moves beyond the road boundary or does not participate in the charging and discharging scheduling of the microgrid group, the penalty imposed on it is a preset constant term; Indicates the number The discharge of electric vehicles, Indicates the number The charging status of electric vehicles, including ; Represents the energy consumption cost of electric vehicles participating in microgrid group scheduling.

[0032] The above-mentioned electric vehicle-participated power dispatching method, the energy consumption cost of the electric vehicle participating in the microgrid group dispatching The calculation formula is as follows:

[0033] ,

[0034] in, represents the preset electric vehicle battery depreciation cost factor; Indicates the moving distance of the electric vehicle; represents the road travel time of electric vehicles; and They respectively represent the preset electric vehicle unit distance movement energy consumption cost coefficient and unit time cost coefficient;

[0035] Road travel time According to real-time traffic flow Calculated, the road travel time The calculation formula is as follows:

[0036] ,

[0037] in, It indicates the travel time when the road is not congested. and A preset coefficient indicating road congestion, Indicates the preset maximum capacity of the road.

[0038] In the aforementioned electric vehicle-participated power dispatching method, the explorer of the AD-MADDPG algorithm in step S3 is a single electric vehicle;

[0039] The explorer and learner of the AD-MADDPG algorithm include the same deep neural network structure;

[0040] The deep neural network parameter update process of the AD-MADDPG algorithm learner includes:

[0041] S31: according to the state space in the partially observable Markov decision process in step S2, using a convolutional neural network to extract local features in the state space data and output private observations;

[0042] S32: Input the private observation outputted from step S31 into the learner’s Actor network, combine it with the attention mechanism, and output the action;

[0043] S33: Each electric vehicle executes the action of step S32, generates experience samples and stores them in the corresponding local experience replay pool. When the local experience replay pool is full, the experience samples are processed using the priority replay mechanism and stored in the global experience replay pool;

[0044] Each electric car corresponds to a local experience playback pool;

[0045] S34: loop through step S33 until the global experience replay pool is full;

[0046] When the global experience replay pool in step S34 is full, start sampling experience samples and generate new experience samples;

[0047] S35: Use the new experience sample in step S34 to train the deep neural network, update the learner's Critic network online by minimizing the loss function and calculate the predicted Q value of the current state and action of the Critic network;

[0048] S36: Based on the predicted Q value calculated in step S35, the learner's Actor network is updated online by the policy gradient method;

[0049] S37: Soft-update the learner's target Actor network and target Critic network using the Actor network updated in step S36 and the Critic network updated in step S35.

[0050] In the aforementioned electric vehicle-participated power dispatching method, the calculation formula of the attention mechanism in step S32 is:

[0051] ,

[0052] in, Represents a new private observation generated by the attention mechanism; Softmax() is an activation function; is the weight matrix in the Softmax function, the weight matrix The initial value of each element is generated by a random initialization strategy; Represents a private observation of the input.

[0053] In the aforementioned electric vehicle-participated power dispatching method, step S33 comprises:

[0054] S331: After each electric vehicle performs the action of step S32, an instant reward is calculated respectively;

[0055] S332: Obtain the next state according to the current state, the action taken, and the state transition function;

[0056] S333: Use the private observation function to obtain the next private observation of the local environment from the next state, generate an experience sample including the current private observation, action, immediate reward and the next private observation, and store it in the corresponding local experience replay pool;

[0057] S334: Calculate the priorities of the experience samples in step S333 respectively, add the calculated priorities of the experience samples to the experience samples in step S333, and store the experience samples with added priorities into the corresponding local experience playback pool.

[0058] Among them, the priority The calculation formula is:

[0059] ,

[0060] in, Represents experience sample The target Q value of for The state space at time, Respectively represent EV 1~EV ​​I The action of the moment, For the state space of the Critic network Next select action The predicted Q value of

[0061] S335: loop through steps S31-S334 until the local experience replay pool is full;

[0062] S336: Select the priority calculated in step S334 from the local experience replay pool filled in step S335 Extract experience samples and add the corresponding explorer ID to the extracted experience samples;

[0063] S337: Store the experience samples extracted in step S336 and adding the explorer ID into the global experience replay pool.

[0064] In the aforementioned electric vehicle-participated power dispatching method, step S34 comprises:

[0065] S341: Loop through step S33 until the global experience replay pool is full

[0066] S342: sampling a preset batch number of experience samples from the global experience replay pool according to the priority replay mechanism of step S33;

[0067] The preset batch number of empirical samples is a sequence consisting of multiple continuous time step empirical samples;

[0068] S343: Calculate the cumulative reward of the experience samples sampled in step S342 , cumulative rewards The calculation formula is as follows:

[0069] ,

[0070] in, is the discount factor, which is a preset value between 0 and 1; represents the N-1th power of the discount factor; express Instant rewards of the moment;

[0071] S344: Using a long short-term memory (LSTM) neural network to process the empirical samples sampled in step S342 to generate new private observations;

[0072] S345: The new private observation generated in step S344 and the cumulative reward calculated in step S343 replace the original private observation and the immediate reward respectively, and a new experience sample is obtained.

[0073] In the above-mentioned electric vehicle-participated power dispatching method, step S342 samples Time to Moment Empirical samples of time steps,

[0074] Step S344 includes:

[0075] S3441: Obtain two state observation sequences from the global experience replay pool according to the experience samples sampled in step S342,

[0076] The two state observation sequences are Private observation at all times to Private observation at all times Collection and Private observation at all times to Private observation at all times Collection

[0077] in, is a positive integer within the range of the preset batch number sampled in step S342, that is, ;

[0078] S3442: Input the two state observation sequences obtained in step S3441 into the LSTM network and output new private observations respectively. and .

[0079] In the aforementioned electric vehicle-participated power dispatching method, the AD-MADDPG algorithm process of step S3 further includes:

[0080] S38: After the update of each deep neural network parameter of the learner is completed in step S37, the priority of the experience samples in the global experience replay pool is updated according to the priority calculation method in the priority replay mechanism of step S33;

[0081] and / or,

[0082] The AD-MADDPG algorithm process of step S3 also includes:

[0083] S39: After the update of each deep neural network parameter of the learner is completed in step S37, the updated deep neural network parameters of the learner are copied to update the deep neural network parameters of the explorer.

[0084] Beneficial Effects

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] The present invention adopts a power dispatching method involving electric vehicles. By constructing an observable Markov decision process, it is more suitable for simulating the continuous process of real-world power dispatching. On the basis of the DDPG algorithm, an AD-MADDPG algorithm combining the attention mechanism and a multiple explorer-single learner framework is designed. Multiple explorers (Actors) are trained in a distributed manner, and a single learner (Leaner) is used for centralized learning, which reduces the complexity of training and ensures that electric vehicles can learn the optimal routing and dispatching strategies. The power dispatching method generated by the present invention can minimize the electric vehicle dispatching cost and achieve the power dispatching goal of maximizing load recovery of the microgrid group, thereby improving the ability of the distribution system to resume operation in an emergency.

[0087] In the transportation network-microgrid coupling network model constructed by the present invention, the transportation network model is used to collect traffic information, and the microgrid model is used to collect distribution network information. The transportation network-microgrid coupling can be used to seek the optimal mobile route and charging / discharging scheduling of electric vehicles, providing a reliable model basis and data source for the implementation of the power scheduling method of the present invention.

[0088] In existing reinforcement learning algorithms, the immediate reward only represents the agent's gain or loss at the current moment, and cannot capture the agent's cumulative reward over multiple time steps. Therefore, during the training process, it may not be accurate to modify the agent's strategy based only on the state of the previous moment and the predicted action value of the next moment.

[0089] The present invention uses a convolutional neural network (CNN) to capture the spatial state characteristics of the microgrid-transportation network coupling network to assist electric vehicle clusters in making optimal movement and charging and discharging decisions.

[0090] The present invention adopts the long short-term memory (LSTM) neural network to extract the movement and charging and discharging time series features of electric vehicles at multiple steps during the movement process, and integrates them into the input of agent training, so as to help the electric vehicle agent understand and evaluate the long-term value of its actions, better reflect the long-term impact of strategy selection, and further improve the scheduling performance of electric vehicle clusters in long-term decision-making.

[0091] The present invention proposes a distributed multi-agent deep deterministic policy gradient algorithm (AD-MADDPG algorithm) combined with an attention mechanism. The attention mechanism can adaptively adjust the attention weight and feature weight according to the changes in the environment and tasks, thereby optimizing the strategy selection and improving the reinforcement learning performance. The distributed multi-agent multiple explorer-single learner framework, multiple explorers (Actors) distributed training, and a single learner (Leaner) centralized learning, the learner learns the optimal network parameters and collaboratively updates the network parameters of the explorer, while reducing the complexity of training to ensure that multiple electric vehicles can cooperate with each other, while reducing the overall energy consumption while improving the load recovery capacity of the microgrid, solving the decision-making problems of mobile path optimization and charging and discharging scheduling of large-scale electric vehicles.

[0092] The present invention introduces priority replay during the training process and by frequently sampling experience samples with high priority, the model can correct the wrong estimation of unpredictable states or actions more quickly, accelerate convergence, and improve the training efficiency of the present invention.

[0093] The present invention adopts advanced machine learning technology (CNN, LSTM) and a distributed multi-agent deep deterministic policy gradient algorithm combined with an attention mechanism, combined with the coordinated scheduling optimization of electric vehicle clusters and microgrid groups, to improve the elastic load recovery capability and energy efficiency ratio of the microgrid group in an off-grid state. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 is a schematic diagram of a transportation network-microgrid coupling network model according to Embodiment 1 of the present invention;

[0095] Figure 2 Schematic diagram of the first channel of the state space of the transportation network-microgrid coupling network model of Example 1 of the present invention;

[0096] Figure 3 is a schematic diagram of a second channel of the state space of the transportation network-microgrid coupling network model of Embodiment 1 of the present invention;

[0097] Figure 4 is a schematic diagram of the third channel of the state space of the transportation network-microgrid coupling network model of Example 1 of the present invention;

[0098] Figure 5 is a schematic diagram of the structure of the AD-MADDPG algorithm of Example 1 of the present invention;

[0099] Figure 6 It is a schematic diagram of a process of generating new experience samples by the LSTM network of Example 1 of the present invention;

[0100] Figure 7 is a schematic diagram of the structure of an LSTM unit in Example 1 of the present invention;

[0101] Figure 8 Schematic diagram of off-grid operation scenarios of five independent microgrids based on the IEEE 33-bus test system according to Embodiment 1 of the present invention;

[0102] Fig. 9 It is a schematic diagram of the Actor network training loss of the AD-MADDPG algorithm of Example 1 of the present invention;

[0103] Fig.10 Schematic diagram of the Critic network training loss of the AD-MADDPG algorithm of Example 1 of the present invention;

[0104] Fig.11 is a schematic diagram comparing convergence curves of EV training of agents using different algorithms in Example 1 of the present invention;

[0105] Fig.12 1 is a schematic diagram comparing the impact of the number of EVs on load recovery rate indicators of different algorithms in Example 1 of the present invention;

[0106] Fig.13 1 is a schematic diagram comparing the impact of the number of EVs on energy consumption ratio indicators of different algorithms in Example 1 of the present invention;

[0107] Fig.14 1 is a schematic diagram comparing the influence of the maximum moving distance of Example 1 of the present invention on the load recovery rate index of different algorithms;

[0108] Fig.15 This is a schematic diagram comparing the impact of the maximum moving distance of Example 1 of the present invention on the energy consumption ratio indicators of different algorithms. DETAILED DESCRIPTION

[0109] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0110] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0111] Embodiment 1:

[0112] In view of the shortcomings of the prior art, the present invention aims to propose a power dispatching method involving electric vehicles, which is used to improve the ability of the distribution system to resume operation in an emergency. This embodiment introduces a power dispatching method involving electric vehicles, including:

[0113] S1: Construct a transportation network-microgrid coupling network model and determine the goal of power dispatching: minimize the dispatching cost of electric vehicles to achieve maximum load recovery of the microgrid group;

[0114] S2: construct a partially observable Markov decision process based on the model of step S1 and the power dispatch target;

[0115] S3: Design the AD-MADDPG algorithm based on the partially observable Markov decision process in step S2;

[0116] The AD-MADDPG algorithm is used to solve the power dispatching method that achieves the power dispatching target according to the model constructed in step S1;

[0117] The AD-MADDPG algorithm is an improved algorithm based on the DDPG algorithm that combines the attention mechanism and the multiple explorers-single learner framework.

[0118] The electric vehicles of the present invention mainly consider the types of electric vehicles that participate in and accept dispatches from travel service platforms (such as Didi, Uber, Gaode, etc.).

[0119] In addition to being applicable to common mobile power sources such as electric vehicles, the present invention is also applicable to other mobile power sources that can participate in transportation networks.

[0120] In step S1, a spatiotemporal network model of coupled transportation network-microgrid is constructed considering multiple uncertainties, and the coordinated dispatch between microgrid group and electric vehicles is modeled as a sequential decision optimization problem. When the microgrid group is off-grid, the goal of power dispatch is to minimize the dispatch cost of electric vehicles and maximize the load recovery of the microgrid group.

[0121] The transportation network-microgrid coupling network model of step S1 includes a transportation network model and a microgrid model. The transportation network model is used to collect traffic information, and the microgrid model is used to collect distribution network information. The transportation network-microgrid coupling network model is used to seek the optimal mobile route and charging / discharging scheduling of electric vehicles to achieve the power scheduling target of step S1.

[0122] The traffic network-microgrid coupling network model architecture constructed by the present invention is as follows: Figure 1When an extreme event (such as an earthquake, typhoon, or continuous snow and ice weather) causes a main grid area failure or power outage, the microgrid group can operate in an off-grid mode, using local distributed renewable energy (such as photovoltaic PV, wind power WT), diesel generation (DG), energy storage system (ESS), and EVs to maintain power supply for critical loads.

[0123] The microgrid group control center forms the supply and demand balance deviation requirements for the isolated operation of the microgrid group by monitoring and collecting the fault conditions of the busbars and branches in the network, as well as the local power generation and load data of each microgrid;

[0124] The electric vehicle control center uses real-time communication to understand the status of available EVs in the current platform, including the remaining power, location information, dispatchable time period, and discharge rate of each EV.

[0125] In step S2, the partially observable Markov decision process POMDP is modeled

[0126] In order to solve the elastic load recovery problem of isolated microgrids, the optimal routing and charging and discharging scheduling problem of electric vehicles (EVs) is modeled as a partially observable Markov decision process (POMDP). "Partially observable" in POMDP means that the agent cannot directly obtain the complete state information of the environment, but can only infer and make decisions based on limited observations.

[0127] Compared with fully observable Markov decision processes (MDPs), POMDPs are more suitable for simulating continuous processes in the real world, because in real environments, agents often cannot directly observe all the details and states of the environment. For example, in robot navigation problems, the robot may not be able to directly observe all obstacles and paths, but can only obtain limited observation information through sensors.

[0128] In the POMDP problem, the agent cannot directly observe the current state, but can only obtain the observation value of the current state. This observation value is determined by the state observation probability matrix, which represents the probability distribution of the agent's possible observations when the state transition is known. The goal of the POMDP problem is to find an optimal strategy so that the agent can make the best action choice through the obtained observation value and the previous historical sequence (including actions and observation values) when it cannot directly observe the state.

[0129] Define the POMDP element:

[0130] Represents the state space, which is the set of all environmental states of the system; is the action space of the agent, then the joint action space of multiple agents, that is, the joint action space of EV1, EV2, ..., EV i, can be expressed as ; represents the observation space, which is the set of private observations obtained by the agent based on the state of the environment; Represents the immediate reward obtained by the agent when performing an action in a given state; is the state transfer function, which represents the probability distribution of the environment state transferring from the current state to the next environment state after the agent takes a certain action.

[0131] According to an environment state in the state space, each agent obtains a private observation according to the observation function, then applies the strategy on the obtained private observation to select an action to perform, and transitions to the next environment state according to the state transition function.

[0132] exist At this moment, all agents follow their private observations Execute an action , return instant reward and a new private observation A private observation is an element in the observation space. It is generated by the environment based on the agent's actions and current state, rather than being directly calculated by the agent or POMDP model. The agent needs to infer the probability distribution of the state based on the private observation and make decisions accordingly. The goal of each agent is to maximize its cumulative expected reward by learning the optimal action strategy.

[0133] In order to conveniently capture the spatial characteristics of the transportation network and the microgrid, the present invention divides the transportation network model and the microgrid model into a two-dimensional grid map of multiple channels, and uses Convolutional Neural Networks (CNN) to extract spatial features and output them as private observations.

[0134] 1) State Space

[0135] system The state space at time Defined as: , consists of three channels:

[0136] The first channel in the state space like Figure 2 As shown, including the geographical location of the charging / discharging station h and the microgrid load reduction associated with the charging / discharging station h ,Right now In order to simplify the calculation, in this embodiment, there is only one charging / discharging station associated with a microgrid. Related charging / discharging station load reduction Used to represent microgrids The load reduction amount, is the microgrid number; when there are multiple When there are multiple charging / discharging stations associated with each other, the load reduction of these multiple charging / discharging stations needs to be summed up to represent the microgrid The load reduction amount; when is positive, indicating that there is load reduction in the microgrid associated with the charging / discharging station h; is negative, indicating that the microgrid associated with the charging / discharging station h has a surplus of electricity. Where h represents the charging / discharging station number, H represents the total number of charging / discharging stations, represents a collection of charging / discharging stations, and They respectively represent the horizontal and vertical coordinates of the charging / discharging station h in the map. represents the time step, represents a set of time steps, Indicates the maximum time step. The maximum time step of this embodiment is 12.

[0137] The present invention adopts a unified coordinate system, and the establishment of a specific coordinate system can be determined according to actual conditions.

[0138] The second channel of the state space like Figure 3 As shown, including the vertex coordinates of the traffic network road and traffic volume on the road ,Right now . Where V represents the total number of road vertices, represents the set of road vertices, Indicates the road vertex (the value range is , ), Indicates the total number of road vertices. Represented by vertex Starting point The road to the end, and Represents vertices The horizontal and vertical coordinates in the map. Indicates the road exist The real-time traffic flow at the moment is calculated as follows:

[0139] , (1)

[0140] in, Indicates the road The number of other types of vehicles with specific daily travel patterns on Indicates the road The number of electric vehicles participating in power dispatch.

[0141] The third channel of state space like Figure 4 Shown, including the current position of the electric vehicle EV and its remaining power ,Right now .in Indicates the EV number, represents the total number of EVs participating in power dispatch, Represents the set of EV numbers participating in power dispatch, and Respectively represent the horizontal and vertical coordinates of EV in the map.

[0142] 2) Action Space

[0143] The action space of EV is defined as ,in, Indicates the moving direction of the EV. The moving direction of the EV is the clockwise rotation angle of the front of the vehicle. Indicates the range of values ​​of the EV moving direction, represents the distance moved by EV in time step t, represents the maximum moving distance of EV in time step t, It represents the range of the distance that EV moves within the time step t. Figures 2 to 4 As shown in the figure, 0 / 180 / 270 / 90 degrees (up, down, left, and right) are the correct moving angles according to driving rules, but the EV agent does not know the driving rules before learning, and its moving angle can be any angle from 0 to 360. When the EV moves beyond the road boundary, it will be punished. In order to avoid being punished, the EV will learn the correct moving angle. Indicates the maximum capacity of an EV relative to its battery The charge-discharge ratio, Indicates the maximum capacity of an EV relative to its battery The range of charge and discharge ratio, To indicate EV charging, A negative value indicates that the EV is discharging. For example: When EV is The power is used to charge itself.

[0144] Different from the prior art, the moving direction of the EV in the present invention is a continuous variable, which is more in line with the actual moving characteristics of the EV. Existing works usually simplify the moving direction of the EV into a direct road selection or four discrete direction selections of up, down, left and right, which is not accurate enough in estimating the energy consumption cost and time cost generated by the EV in the actual moving process.

[0145] 3) Reward Function

[0146] For EV number i, when we observe Microgrid load reduction at time , Traffic flow of each road , and its remaining power In the case of Formulate the best mobile path and the best charging and discharging strategy, with the goal of maximizing load recovery of the microgrid group at the lowest EV dispatch energy consumption cost. The instant reward function calculation formula of the numbered electric vehicle at the time is as follows:

[0147] , (2)

[0148] in, Represents microgrid The load reduction amount, that is, the first channel of the state space In and charging / discharging station Associated microgrids Load reduction . Microgrid The load reduction can also be actively released by the microgrid operator at each moment, and its calculation method is as follows: The initial load Subtract the current load ; represents a preset unit load reduction cost factor; Indicates when the number When an EV moves beyond the road boundary or does not participate in the microgrid group charging and discharging scheduling, the penalty imposed on it is a preset constant term. Whether the EV moves beyond the road boundary can be judged based on the current position coordinates of the EV and the coordinates of the road vertex; Indicates the number EV discharge conditions, Indicates the number EV charging conditions, including: , To indicate EV charging, Negative means EV is discharging. When EV is The power is used to charge itself; It represents the energy consumption cost of EV participating in microgrid group dispatching, which is calculated as follows:

[0149] , (3)

[0150] in, It represents the preset EV battery depreciation cost coefficient. Each charge and discharge will affect the chemical composition of the battery and cause certain losses to the battery. As the number of charge and discharge increases, the performance of the battery gradually decreases. Therefore, existing studies usually use the "depreciation cost coefficient" to represent the wear of EV batteries. The depreciation cost of EV batteries is related to both charging and discharging. Indicates the travel distance of the EV; Indicates the travel time of the road at the time; and They represent the EV energy consumption cost per unit distance and the cost per unit time, respectively. With real-time traffic flow Related, the calculation method is as follows:

[0151] , (4)

[0152] in, It indicates the travel time when the road is not congested. and is the preset coefficient for road congestion, Indicates the preset maximum capacity of the road. At different times, the traffic flow of the road is different, which will lead to different road travel times. For example, the road travel time will be different during the morning peak period and non-peak period. Here, the road travel time is used as the time cost of EV, which can better reflect the actual situation.

[0153] The optimal routing decision and charging and discharging scheduling of EVs involve the spatial characteristics of the transportation network-microgrid coupling network. In addition, since the maximum distance that an EV can move in a single time step is limited, it may take multiple time steps to reach the microgrid charging / discharging station. Therefore, utilizing the time series characteristics and spatial characteristics of EV movement is crucial to the problem to be solved by the present invention.

[0154] In step S3, a distributed multi-agent deep deterministic policy gradient algorithm (AD-MADDPG algorithm) is designed with attention mechanism.

[0155] The Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm for continuous action space. The core feature of the algorithm is to use deep neural networks to approximate deterministic policies and optimize this policy through gradient descent. "Deep" refers to the use of deep neural networks, "deterministic policies" means that in each state, the algorithm will output a deterministic action instead of the probability distribution of the action, and "policy gradient" means updating the policy by calculating the gradient of the policy function to maximize the cumulative expected return. It is based on the DQN (Deep Q-Network) algorithm and uses deep neural networks to approximate the Q-value function and policy function, thereby realizing the prediction and optimization of continuous actions.

[0156] The DDPG algorithm consists of four neural networks (Actor, Actor target, Critic, Critic target), experience replay pool, target network update, loss function and optimization, and specific algorithm flow, which realizes reinforcement learning tasks in continuous action space.

[0157] Actor network: also known as the policy network, its input is the current state, and its output is the action that should be taken under the current state. The Actor network is responsible for deterministically outputting actions based on the current state;

[0158] Actor Target Network: It is the "target" version of the Actor Network. Its structure is the same as the Actor Network, but the parameter update method is different. The Actor Target Network is used to provide the action of the next state during training to calculate the target Q value.

[0159] Critic network: Also known as value network or Q network, its input is the current state and action, and its output is the Q value of taking the action in the state. The Critic network is responsible for evaluating the value of the current state-action pair.

[0160] Critic target network: It is the "target" version of the Critic network. Its structure is the same as the Critic network, but the parameters are updated in a soft update manner. The Critic target network is used to provide the Q value estimate of the next state-action pair during training to calculate the target Q value.

[0161] The experience replay pool is used to store the experience data of the agent's interaction with the environment, including the state, action, immediate reward, next state, and whether it is ended. These data are randomly sampled during the training process and used to train the Actor network and the Critic network. The use of the experience replay pool helps to break the correlation between data and improve training efficiency and stability.

[0162] In the DDPG algorithm, the parameters of the target network are updated through soft updates. Specifically, the parameters of the target network are a mixture of the current network parameters and the target network parameters in a certain proportion. This update method helps stabilize the training process and avoids training instability caused by frequent updates of the target network.

[0163] Loss function of the Critic network: The mean square error (MSE) is used as the loss function to measure the gap between the Q value output by the Critic network and the target Q value. By minimizing this loss function, the parameters of the Critic network can be updated.

[0164] The loss function of the Actor network: The strategy is optimized by maximizing the Q value. Specifically, the loss function of the Actor network can be defined as the negative of the Q value of the action taken in the current state. By maximizing this Q value, the parameters of the Actor network can be updated.

[0165] The specific process of the DDPG algorithm is as follows:

[0166] 1. Initialize the Actor network, Critic network and their corresponding target networks.

[0167] 2. Initialize the experience replay pool.

[0168] 3. For each training round:

[0169] 1) Initialize the random process for action exploration.

[0170] 2) Get the initial state of the environment.

[0171] 3) For each time step:

[0172] (1) Select an action based on the current policy and noise.

[0173] (2) Execute the action, obtain immediate reward, and the environment state transitions to the next state.

[0174] (3) The current state, action, immediate reward, next state, and whether the process ends are stored in the experience replay pool.

[0175] (4) Randomly sample a batch of experience samples from the experience replay pool.

[0176] (5) Use the target network to calculate the target Q value.

[0177] (6) Update the Critic network so that its evaluation value is close to the target value.

[0178] (7) Update the actor network so that its strategy maximizes the cumulative expected return.

[0179] (8) Soft update the parameters of the target network.

[0180] 4. Repeat the above process until the training end condition is reached.

[0181] In order to continuously solve the optimal routing and charging and discharging scheduling problems of EVs, this paper proposes a distributed multi-agent deep deterministic policy gradient algorithm (AD-MADDPG) combined with an attention mechanism based on the traditional DDPG algorithm.

[0182] Given the scale of the transportation grid-microgrid coupled network, EVs may require a lot of time to fully explore their spatial and temporal characteristics, resulting in a large number of records.

[0183] In order to reduce the complexity of training and ensure that EVs can learn the optimal routing and scheduling strategies, the algorithm adopts a framework of multiple explorers-single learners, multiple explorers (Actors) distributed training, and a single learner (Leaner) centralized learning. The AD-MADDPG algorithm framework is as follows Figure 5 shown.

[0184] Specifically, the deep neural network of the AD-MADDPG algorithm consists of four deep neural networks (DNNs). Each independent explorer periodically copies network parameters from the learner and updates the explorer's own policy network. Each explorer (Actor) maintains a local experience replay pool used by EV to sequentially store the transfer experience sample data generated within its local observation range. Maintained local experience replay pool Indicates the number The local experience replay pool used by the EV of the learner is the global experience replay pool used by each EV. .in, Represents the global experience replay pool Used for numbering Since multiple explorers and a single learner are executed in parallel, task exploration and learning can be performed in a distributed manner.

[0185] The explorer of the AD-MADDPG algorithm in step S3 is a single electric vehicle;

[0186] The explorer and learner of the AD-MADDPG algorithm include the same deep neural network structure;

[0187] The deep neural network parameter update process of the AD-MADDPG algorithm learner includes:

[0188] S31: according to the state space in the partially observable Markov decision process in step S2, using a convolutional neural network to extract local features in the state space data and output private observations;

[0189] S32: Input the private observation outputted from step S31 into the learner’s Actor network, combine it with the attention mechanism, and output the action;

[0190] S33: Each electric vehicle executes the action of step S32, generates experience samples and stores them in the corresponding local experience replay pool. When the local experience replay pool is full, the experience samples are processed using the priority replay mechanism and stored in the global experience replay pool;

[0191] Each electric car corresponds to a local experience playback pool;

[0192] S34: loop through step S33 until the global experience replay pool is full;

[0193] When the global experience replay pool in step S34 is full, start sampling experience samples and generate new experience samples;

[0194] S35: Use the new experience sample in step S34 to train the deep neural network, update the learner's Critic network online by minimizing the loss function and calculate the predicted Q value of the current state and action of the Critic network;

[0195] S36: Based on the predicted Q value calculated in step S35, the learner's Actor network is updated online by the policy gradient method;

[0196] S37: Soft-update the learner's target Actor network and target Critic network using the Actor network updated in step S36 and the Critic network updated in step S35.

[0197] The algorithm sub-steps are described as follows:

[0198] Sub-step S31:

[0199] In order to extract spatial state features from the transportation network-microgrid coupling network, the present invention divides the transportation network-microgrid coupling network model into the following Figures 2 to 4 The two-dimensional grid map shown in Figure 2 is used to extract the spatial state information using a convolutional neural network (CNN).

[0200] Specifically, according to the definition of the system state space, the present invention uses a three-layer image to represent the state observation of the EV, such as Figures 2 to 4 shown.

[0201] like Figure 2 As shown, the first layer contains the location of the charging / discharging stations in the map and the load reduction of each associated microgrid; Figure 3 As shown in , the second layer contains the road vertex coordinates and road traffic flow in the traffic network; Figure 4 As shown, the third layer contains the EV coordinates and its remaining charge.

[0202] like Figure 5 As shown in the upper left corner, the three-layer image is used as the three channels of the CNN input layer, and the convolutional neural network uses a learnable convolution kernel (filter) to perform sliding calculations on the input image data to generate a feature map; among them, in the first convolutional layer (conv1) of the convolutional neural network, there are 16 convolution kernels of size 3x3, and the step length of these convolution kernels moving on the input data is 2; in the second convolutional layer (conv2) of the convolutional neural network, there are 32 convolution kernels of size 3x3, and the step length of these convolution kernels moving on the input data is 2.

[0203] After extracting the spatial features of the input image (such as road edges, EV and charging / discharging station locations, road traffic flow, etc.) on the generated feature map, the extracted spatial feature output is flattened into a one-dimensional vector through the fully connected layer 1 (fc1) and the fully connected layer 2 (fc2), and the one-dimensional vector is used as the input Actor network Private observations The number of neurons in the fully connected layer 2 (Fc2) is the same as the number of neurons in the input layer of the Actor network.

[0204] By stacking the above neural network layers, CNN can extract local features in the data layer by layer.

[0205] Sub-step S32:

[0206] For each EV, the deep neural network of the distributed deep deterministic policy gradient AD-MADDPG algorithm combined with the attention mechanism includes four deep neural networks (DNNs), namely, the Actor network , Critic Network , target Actor network and target critic network .

[0207] like Figure 5 As shown, the private observation obtained in substep 1 Input to the Actor Network , EV is a private observation based on the current input , combined with the attention mechanism, output action The attention mechanism is to learn a weight distribution for the input private observation features, and then apply the weight distribution to the original features, so that the task learning focuses on important features and improves efficiency. The calculation formula of the current input state information combined with the attention mechanism is , where Softmax() is an activation function that can convert a vector containing any real number into a probability distribution, that is, the value range of each element is between (0,1), and the sum of all elements is 1; It is the weight matrix in the Softmax function, which is used to map the input features to the scores of each category and convert these scores into probability distributions through the Softmax function. The initial values ​​of each element are generated by a random initialization strategy, and each element in the weight matrix is ​​a learnable parameter that is updated by the back-propagation algorithm during the training of the neural network.

[0208] like Figure 5 As shown in the Attention network of the Actor network, the weight matrix The initial value of each element is generated by a random initialization strategy, through Perform a linear transformation on the input vector and then pass The linear transformation result is normalized so that the sum of the weight distribution is 1, ensuring that the weight distribution can be interpreted as the relative importance of each input feature. The Softmax function can be used to convert a vector into a probability distribution, that is, the sum of all elements is 1, and each element is between 0 and 1; the higher the weight of the feature, the greater its contribution to the model's prediction or decision-making process. Multiplying together can generate new private observations weighted by the attention mechanism The attention mechanism can adaptively adjust the attention weights and feature weights according to changes in the environment and tasks, thereby optimizing strategy selection and improving reinforcement learning performance.

[0209] Sub-step S33 includes:

[0210] S331: After each electric vehicle performs the action of step S32, an instant reward is calculated respectively;

[0211] S332: Obtain the next state according to the current state, the action taken, and the state transition function;

[0212] S333: using the private observation function to obtain the next private observation of the local environment from the next state, and generating an experience sample including the current private observation, action, immediate reward and the next private observation;

[0213] S334: Calculate the priorities of the experience samples in step S333 respectively , and the priority of the calculated empirical samples Add to the experience samples of step S333, and store the experience samples with added priority in the corresponding local experience playback pool;

[0214] S335: loop through steps S31-S334 until the local experience replay pool is full;

[0215] S336: Select the priority calculated in step S334 from the local experience replay pool filled in step S335 Extract experience samples and add the corresponding explorer ID to the extracted experience samples;

[0216] S337: Store the experience samples extracted in step S336 and adding the explorer ID into the global experience replay pool.

[0217] In this embodiment, sub-step S33 is specifically implemented as follows:

[0218] Each EV performs an action , calculate the instant reward of each electric vehicle EV according to formula (2) , and based on the current state, the action taken, and the state transition function Predict the next state, and then use the private observation function to get the next private observation of the local environment from the next state , generate including the current private observation ,action , Instant Rewards and the next private observation Experience samples And store it in the corresponding local experience replay pool According to the Markov characteristic, after executing the action, the current environment state will be transferred to the environment state at the next moment, that is, the state space at the next moment t+1 updates the state space at the current moment t: ,like Figure 5 Shown in the lower left corner.

[0219] when When full, all data is sent to the global experience replay pool corresponding to part i. Used for training. "Full" refers to the local experience replay pool The specific capacity upper limit value is usually customized according to the experiment or task requirements. For example, the present invention sets The capacity is 5000, The capacity is 10,000. Since multiple explorers and a single learner run in parallel, they can be deployed to perform exploration and learning tasks in a distributed manner. Specifically, there is no need to wait for all explorers' experience replay pools to be full before adding experience samples to the global experience replay pool. When an explorer's experience replay pool is full, experience samples will be extracted to the global experience replay pool for subsequent neural network training. When other explorers' experience replay pools are full, the extracted experience samples will be added to the global experience replay pool. When the global experience replay pool is full, the newly extracted experience samples will overwrite the earliest experience sample records.

[0220] for Experience samples in , the present invention uses temporal difference error (TD-error) to indicate the priority of each experience sample , which is used to represent the experience sample The importance of being sampled later for training.

[0221] The time series difference error can be used as an indicator to measure the value of experience learning. Specifically, the larger the absolute value of the time series difference error, the greater the gap between the current experience sample and the true value, and the more information can be learned from the current experience sample. Therefore, the role of this experience sample in correcting the estimated value and improving the model performance is greater. Determining the priority of data in the experience pool according to the absolute value of the time series difference error can make it more likely that the experience with higher priority will be extracted for learning.

[0222] Priority The calculation formula is as follows:

[0223] , (5)

[0224] in, Represents experience sample The target Q value (updated as in formula (10)), for The state space at time, Respectively represent EV 1~EV ​​I The action of the moment, For the state space of the Critic network Next select action The predicted Q value of .

[0225] It should be noted that the predicted Q value is calculated by the Critic network. ) According to the current system state space Execute an action , you can get instant rewards and the next state space , then the Critic network will be based on the current state space and the current action Calculate the predicted Q value.

[0226] After obtaining the priority of the experience sample, Add priority to and Explorer ID Generating Experience Samples And stored in the global experience replay pool In (such as Figure 5 lower right corner).

[0227] Sub-step S34 includes:

[0228] S341: loop through step S33 until the global experience replay pool is full;

[0229] S342: sampling a preset batch number of experience samples from the global experience replay pool according to the priority replay mechanism of step S33;

[0230] The preset batch number of empirical samples is a sequence consisting of multiple continuous time step empirical samples;

[0231] S343: Calculate the cumulative reward of the experience samples sampled in step S342;

[0232] S344: Using a long short-term memory (LSTM) neural network to process the empirical samples sampled in step S342 to generate new private observations;

[0233] S345: The new private observation generated in step S344 and the cumulative reward calculated in step S343 replace the original private observation and the immediate reward respectively, and a new experience sample is obtained.

[0234] When S342 samples Time to Moment Empirical samples of time steps,

[0235] Step S344 includes:

[0236] S3441: Acquire two state observation sequences from the global experience replay pool according to the experience samples sampled in step S342;

[0237] S3442: Input the two state observation sequences obtained in step S3441 into the LSTM network and output new private observations respectively. and .

[0238] In this embodiment, sub-step S34 is specifically implemented as follows:

[0239] Repeat sub-step S33 until When full, When the memory is full, start sampling and replaying experience and start training the neural network.

[0240] In the distributed environment of the present invention, the global experience replay pool of each EV may not be filled at the same time;

[0241] The learner does not have to wait until all EVs' experience pools are full before training. As long as one EV's global experience pool is full, sampling training can begin. At the same time, other EVs are also generating experience in parallel, and the learner can also combine the experience of other EVs during sampling training.

[0242] The present invention uses the global experience replay pool In the example, the priority experience replay mechanism samples a preset number of experience samples in batches and calculates the cumulative rewards of multiple time steps. The priority experience replay mechanism assigns a priority to each experience sample, allowing important experience samples to be selected more frequently, thereby improving training efficiency and performance. Combined with the sample priority in formula (5), the experience sample The probability of being sampled can be calculated as follows:

[0243] , (6)

[0244] Among them, the preset index It is used to control the influence of priority on the probability of sample acquisition. For example, It means that all empirical samples are sampled with equal probability, regardless of priority, that is, uniform sampling.

[0245] "Experience samples of preset batch size" is a set that contains multiple experience samples generated in sub-step 3. The specific number depends on the set batch size, such as 32, 64 or 128.

[0246] Considering that EV is The moving distance within a certain time frame is limited, and reaching a target charging / discharging station may require multiple time steps. The calculated instant reward only represents the moment The gains or losses of EV cannot represent the cumulative rewards over multiple time steps.

[0247] Therefore, in order to characterize each EV The time series feature information of the accumulated rewards at each time step and the Time has come The present invention uses a long short-term memory (LSTM) neural network to capture the time series feature information of EV at multiple time steps. In this embodiment, The cumulative reward calculation formula for time steps is as follows:

[0248] , (7)

[0249] in, is the discount factor, which indicates the relative importance of future rewards in the current decision. The discount factor is a preset value between 0 and 1 and can be set according to the specific situation; Represents the discount factor of Power; express Instant rewards of the moment.

[0250] The Long Short-Term Memory (LSTM) neural network is a recurrent neural network (RNN) variant specifically designed to solve time series data, and is particularly suitable for capturing long-term dependencies in time series data. Traditional RNNs have the problem of gradient vanishing or exploding when processing long sequences, while LSTM solves this problem by introducing gating mechanisms (including input gates, forget gates, and output gates) and memory cells. The core of the LSTM neural network lies in its unique "gate" mechanism and cell state. Together, these mechanisms enable LSTM to effectively remember important information while forgetting irrelevant content.

[0251] In the field of electric vehicles (EVs), LSTM neural networks can be used to capture the time series feature information of EVs at multiple time steps, which is of great significance for analyzing the performance of EVs, predicting their behaviors or trends, etc.

[0252] Specifically, from the global experience replay pool Two state observation sequences are obtained from Private observation at all times to Private observation at all times Collection and Private observation at all times to Private observation at all times Collection , and input it into the LSTM network to output new private observations respectively and .in, is the length of the sequence input to the LSTM network. is the preset batch number sampled in step S342 A positive integer in the range, that is .

[0253] In this embodiment, if Figure 6 As shown, The value is 3, from the global experience replay pool Two state observation sequences are obtained from Private observation 2 moments before the moment to Private observation at all times Collection and Private observation 2 moments before the moment to Private observation at all times Collection . Private observation at all times Input into the LSTM network and update The hidden state at time , and output a new private observation ;Will Private observation at all times and newer Always hide status Input into the LSTM network and update The hidden state at time , and output a new private observation ;Will Private observation at all times and newer Always hide status Input into the LSTM network and output new private observations Similarly, Private observation at all times Input into the LSTM network and update The hidden state at time , and output a new private observation ;Will Private observation at all times and newer Always hide status Input into the LSTM network and update The hidden state at time , and output a new private observation ;Will Private observation and update at all times Always hide status Input into the LSTM network and output new private observations . Now get a new private observation and .

[0254] Generate new private observations based on LSTM network modeling and Replace the original experience sample Private observations in and , calculate the cumulative reward Replace the instant reward in the original experience sample , and got a new experience sample , the new experience sample Replace the original experience sample In this way, when EV learns from a preset batch size of experience samples, it can retrospectively consider multiple time steps, gaining insight into the cumulative impact of a series of actions and decisions.

[0255] The LSTM neural network consists of multiple LSTM units, which are connected end to end to form a chain structure. Figure 7 As shown in the figure, each LSTM unit includes an input gate, a forget gate, an output gate, and a memory cell. Each LSTM unit can update its internal state and generate output based on the previous input and the current input. This structure enables the LSTM network to retain or discard information over a longer time span, thereby effectively processing long-term dependencies in sequence data.

[0256] The short-term memory of a unit in an LSTM (Long Short-Term Memory) network refers to the network's ability to remember recent data. This short-term memory capability enables LSTM networks to effectively handle local dependencies in time series data. The following is a detailed explanation of the short-term memory of a unit in an LSTM network and how it is generated:

[0257] 1. Definition of Short-term Memory

[0258] In the LSTM network, short-term memory is mainly reflected in the hidden state. The hidden state is updated at each time step and is used to store and transmit short-term information about the input sequence. This short-term memory capability enables the LSTM network to capture local features in the input sequence and make predictions or classifications based on them.

[0259] 2. How short-term memory is generated

[0260] Forget gate: The forget gate is used to control how much information in the hidden state of the previous time step needs to be forgotten. It generates an output value between 0 and 1 through the sigmoid activation function, which indicates the degree to which each element in the previous hidden state is retained. When the output value of the forget gate is close to 0, it means that most of the information is forgotten; when the output value is close to 1, it means that most of the information is retained.

[0261] Input gate: The input gate is used to control how much information in the input of the current time step needs to be added to the memory cell. Similarly, it generates an output value between 0 and 1 through the sigmoid activation function, which indicates the extent to which each element in the current input is added to the memory cell. When the output value of the input gate is close to 0, it means that most of the input information is ignored; when the output value is close to 1, it means that most of the input information is added to the memory cell.

[0262] Memory cell: Memory cell is the core part of LSTM network, which is used to store long-term memory information. It updates its state by combining the output values ​​of forget gate and input gate.

[0263] Output gate: The output gate is used to control how much information in the memory cell needs to be output to the hidden state. It generates an output value between 0 and 1 through the Sigmoid activation function, which indicates the degree to which each element in the memory cell is output to the hidden state. Then, the state of the memory cell is scaled using the tanh activation function and multiplied by the output value of the output gate to obtain the final hidden state.

[0264] like Figure 7 As shown, the new private observation in the LSTM network The calculation is based on LSTM units. Each LSTM unit contains a memory cell and three gate controllers. The calculation process includes:

[0265] Based on the private observation of input , the hidden state of the previous unit at time t-1 and the state of the memory cell of the previous unit , respectively calculate the input gate , Forget Gate , output gate and state scaling of memory cells ;

[0266] According to the input gate , Forget Gate , output gate and state scaling of memory cells The calculation results of memory cells are calculated respectively. , the hidden state of the current unit at time t and the new private state of the output .

[0267] Computed input gate , forget gate, output gate , Scaling the state of memory cells , the state of memory cells , the hidden state of the current unit at time t and the new private state of the output The calculation formula is as follows:

[0268] (8)

[0269] in, , , , , , , , , , , , represents the weight matrix of the control gate or memory unit, , , , , Represents the bias vector of the control gate or memory unit. The weight matrix and bias vector of the control gate or memory unit are parameters in the LSTM (Long Short-Term Memory) network. They are automatically generated by the LSTM network during initialization and updated and optimized through the back-propagation algorithm during the training process. After the training is completed, these parameters will be fixed and used for subsequent prediction or analysis tasks. represents the Sigmoid function, and tanh() represents the hyperbolic tangent activation function.

[0270] Sub-step S35:

[0271] Use the new experience samples to train the neural network, input the new experience samples into the Critic network to calculate the predicted Q value and the target Q value, and minimize the loss function Update the learner's critic network online (such as Figure 5 As shown in the upper right part), the loss function is calculated as follows:

[0272] , (9)

[0273] in, Indicates EV Critic network parameters; represents the state space at time t; Indicates EV 1~EV ​​I at time Actions; For the state space of the Critic network Next select action The predicted Q value indicates that in state EV takes action The expected value of the cumulative future rewards that can be obtained after

[0274] Represents the target-based Critic network The target Q value is determined by the immediate reward and the next time step The weighted sum of the values ​​is obtained, and the calculation formula is as follows:

[0275] , (10)

[0276] Among them, instant rewards The immediate impact of the current action, This means that from the next time step The possible return in the future. This part is calculated by the discount factor Weighted to reflect the importance of future rewards.

[0277] In formula (9) The () operation represents the expected value operation, that is, the expectation of the average error for all possible situations.

[0278] Specifically, the loss function of the Critic network is It is updated based on the average error of multiple training experience samples. Due to the randomness of the environment and strategy, the rewards and state-action pairs of each training experience sample may be different, so the expected value To represent the average of the errors of all these empirical samples.

[0279] The loss function updates the Critic network by minimizing the expected error.

[0280] , (11)

[0281] in, is the learning rate, which is used to control the step size of each parameter update; Represents the loss function Parameters gradient.

[0282] Online update: refers to the current strategy (Actor network) or value estimation (Critic network) directly updating parameters through current experience. Whenever the system obtains new state, action, reward and other data through interaction with the environment, the parameters of the Critic and Actor networks will be immediately updated based on this data.

[0283] In this update method, the parameters of the network will change frequently, especially after each state transition or batch data replay, the network will be adjusted. This frequent update based directly on experience is called online update.

[0284] Sub-step S36:

[0285] The learner's Actor network is updated online using the policy gradient method. The updated gradient is as follows:

[0286] , (12)

[0287] in Represents the Critic network for the state space Next select action The predicted Q value of Represents the action output strategy of the Actor network, that is, in the private observation The action generated by ; Is the Critic network about the action The gradient of , which indicates the sensitivity of the action to the Q value, is usually calculated by the output of the Critic network with respect to the action To perform the derivation, ; Represents the gradient of the Actor network with respect to its parameters, that is, describes how to adjust the parameters of the Actor network To change the output action , which is calculated as the output of the Actor network About parameters To perform the derivation, Final Actor Network Parameters Updated as follows:

[0288] , (13)

[0289] in, is the learning rate, which controls the amplitude of parameter updates. In addition, the Actor network Attention weight matrix As part of the Actor network parameters, they are also updated.

[0290] Sub-step S37:

[0291] like Figure 5 As shown in the upper right corner, after the learner's Actor network and Critic network are updated, the target Actor network is soft-updated using the learner's Actor network and Critic network. and target critic network . The update method is as follows:

[0292] , (14)

[0293] in is a positive number much smaller than 1, such as 0.001. Soft update means that each time the target network (Targetactor and Target critic ) does not immediately copy the Actor network and Critic Network Parameters, but through a smoothing coefficient To control the update speed of the target network to slowly approach the main network. This method can make the update of the target network more stable and reduce the shock during training.

[0294] Target Actor Network It is an Actor network A slowly updated copy of , whose main role is to stabilize the policy update. At each time step In the current Actor network Continuous updates will cause the strategy to change quickly, which will cause the target value of the Critic network to fluctuate dramatically.

[0295] To prevent this instability, the target Actor network Provide a relatively stable strategy , used to determine the next action when calculating the target Q value of the Critic network , instead of directly using the actions output by the current Actor network, to avoid excessive fluctuations during strategy updates.

[0296] Target Critic Network It is the Critic network The slowly updated copy of is mainly used to calculate the target Q value (as shown in formula (10)), where is the action generated by the target Actor network , It is the Q value output by the target critic. The Q value generated by the target critic network is relatively stable, which reduces the target Q value fluctuation problem caused by frequent updates of the critic network, thereby improving the training stability of the critic network.

[0297] In order to verify and illustrate the improvement of the proposed method AD-MADDPG algorithm compared with the prior art, the present invention compares the proposed algorithm with three other benchmark algorithms on the IEEE 33-bus test system. The specific comparison settings and results are described as follows.

[0298] 1. Contrast settings

[0299] In order to verify the performance of the proposed AD-MADDPG algorithm in elastic load recovery of isolated microgrids coordinated by electric vehicles, the transportation network is modeled as a 3232 two-dimensional grid. For the microgrid group, an off-grid operation scenario containing 5 microgrids is constructed based on the IEEE 33-bus test system. Its topology is as follows: Figure 8 As shown. There are 5 charging and discharging stations in the transportation network-microgrid coupling network to support the charging / discharging of EVs. For the convenience of calculation and analysis, it is assumed that multiple faults occur in the main grid at 10:00 am, causing the microgrid group to be disconnected from the main grid. The estimated power outage time is 6 hours, and the scheduling interval of EVs is 0.5 hours (ie, T = 12). When the method of the present invention is applied to solve the power dispatching method, T represents the duration of the off-grid operation of the microgrid group after the main grid fails, that is, the power outage time. The interval between t and t+1 is 30 minutes, that is, the time step is 0.5 hours.

[0300] In addition, in order to verify the performance of EVs in the load recovery of the microgrid group, it is assumed that the lines B1-B2, B4-B5, B5-B6 and B11-B12 are disconnected due to faults, and the microgrid group is decomposed into 5 autonomously operated microgrids. Each microgrid uses internal distributed renewable energy (such as photovoltaic and wind power), distributed generation (DG), energy storage system (ESS), etc. to supply power to the loads in their respective areas. The technical parameters of each microgrid and EV are shown in Table 1.

[0301] Table 1 Technical parameters of microgrids and EVs

[0302]

[0303] In order to verify the performance of the proposed AD-MADDPG algorithm, we compare it with the following three benchmark algorithms:

[0304] 1) MADDPG: This method is regarded as one of the most advanced methods for multi-agent deep reinforcement learning, and shows superior performance in cooperative and competitive multi-agent environments compared to other deep reinforcement learning methods. This method uses MADDPG's centralized training and decentralized execution strategy to learn the optimal EVs routing and scheduling strategy.

[0305] 2) MADDQN: This method studies the routing and scheduling optimization of mobile energy storage systems to achieve load recovery. To achieve MADDQN-based simulation, the routing and scheduling of the mobile energy storage system MESS are discretized to verify its convergence.

[0306] 3) AD-MADDPG w / o LSTM: In this method, the policy network only depends on the moment in the decision-making process. The current observation value of , the proposed LSTM-based feature extraction method for multiple time-step sequences is not utilized.

[0307] 2. Comparative results analysis

[0308] 2.1 Convergence Performance Comparison

[0309] Fig. 9 The Actor network training loss of the AD-MADDPG algorithm of this invention is shown. Fig.10 The training loss of the Critic network of the AD-MADDPG algorithm of the present invention is shown. It can be seen from the figure that due to the inaccurate approximate estimation in the early stage of training, the loss values ​​of the Actor network and the Critic network are high and drop rapidly. By using the Multi-Actor and Single-Learner architecture and the LSTM-based multi-time step sequence feature extraction, the two losses gradually converge to a smaller value range after about 4000 rounds, verifying the convergence of the proposed AD-MADDPG algorithm.

[0310] Comparison of the convergence curves of agent EV training using different algorithms Fig.11 As shown in , the average reward and its standard deviation (shaded area) over 10,000 rounds are shown. Fig.11 As shown in Figure 2, the AD-MADDPG algorithm outperforms the other three baseline algorithms in terms of average reward after convergence and convergence speed. Specifically, the average reward of AD-MADDPG after convergence is 21.6% higher than that of AD-MADDPGw / o LSTM.

[0311] The above results show that the electric vehicle-participating power dispatching method of the present invention has learned more effective strategies in the microgrid group in off-grid mode, and can achieve the goal of maximizing load recovery of the microgrid group with the lowest EV dispatching energy consumption cost.

[0312] 2.2 Computational Performance Comparison

[0313] The computational performance of the proposed AD-MADDPG method and different methods are shown in Table 2. Obviously, MADDQN has the longest total training time, mainly because the number of deep neural networks (DNNs) increases approximately linearly with the number of EVs, and the large-scale discrete action space brings a large computational burden. In contrast, MADDPG has the shortest training time, which is attributed to its deterministic policy update and simplified action selection process. In addition, the AD-MADDPG algorithm requires the least number of convergence training rounds, thanks to the training framework of multiple explorers Actors and a single learner Learner adopted by the present invention and the multi-time step sequence feature modeling based on LSTM.

[0314] The above results show that by calculating the expected rewards accumulated over multiple time steps, the AD-MADDPG algorithm achieves higher action exploration efficiency during training, which in turn brings higher average rewards to EV in load recovery and shows better computational performance within an acceptable training time.

[0315] Table 2 Comparison of computational performance of different methods

[0316]

[0317] 2.3 Comparison of EVs dispatch performance

[0318] In order to evaluate the performance of EVs participating in the flexible dispatch of microgrid groups, the proposed AD-MADDPG algorithm is compared with three other benchmark algorithms (MADDQN algorithm, MADDPG algorithm and AD-MADDPG algorithm).

[0319] w / o LSTM algorithm) were compared in terms of load recovery rate and energy consumption ratio:

[0320] 1) Load recovery rate :exist The total amount of load restored within a time step and the initial load reduction ratio.

[0321] exist The total amount of load restored within a time step is The load reduction at the time minus The load reduction at the time is expressed as: ;For example: The load reduction at the moment is 100. If the load reduction at the moment is 40, it means that 60% of the total load has been restored during this period. The more load is restored, the greater the load recovery rate.

[0322] 2) Energy consumption ratio :exist The ratio of the total power consumed by all EVs moving within a time step to the initial power reserve.

[0323] The energy cost of electric vehicle movement is shown in formula (3). A more reasonable scheduling strategy can reduce the energy consumption of EV movement, thereby achieving a lower energy consumption ratio.

[0324] The AD-MADDPG algorithm proposed in this invention is different from the other three algorithms in terms of load recovery rate. , energy consumption ratio The comparison results are as follows Figures 12 to 15 shown.

[0325] like Fig.12 As shown in the figure, the number of EVs increases from 2 to 6, and the AD-MADDPG algorithm always outperforms the other three algorithms in terms of load recovery. For example, when the number of EVs I = 3, the load recovery rate of the algorithm AD-MADDPGw / o LSTM with the best load recovery rate among the other three algorithms is only 0.814, while the load recovery rate of the AD-MADDPG algorithm reaches 0.924, an increase of 13.5%. On average, the load recovery rate of the AD-MADDPG algorithm is improved by 9.5%, 22.6% and 41.5% compared with the AD-MADDPG w / o LSTM algorithm, the MADDPG algorithm and the MADDQN algorithm, respectively.

[0326] In addition, if Fig.13 As shown in the figure, the total energy consumption of all methods increases with the increase in the number of EVs. However, thanks to the LSTM-based multi-time step sequence feature modeling and the multi-actors single learner architecture design, the AD-MADDPG algorithm promotes effective collaboration between multiple EVs during the learning process and obtains a better scheduling strategy, thereby improving the load recovery rate while the energy consumption ratio increases relatively slowly. Specifically, in terms of energy consumption ratio, the AD-MADDPG algorithm is reduced by an average of 8.9%, 17.8%, and 21.4% compared to the AD-MADDPG w / o LSTM algorithm, the MADDPG algorithm, and the MADDQN algorithm, respectively.

[0327] Fig.14 and Fig.15 Evaluated and compared the maximum travel distances of EVs Impact on performance indicators.

[0328] like Fig.14 and Fig.15 As shown, the maximum moving distance In the process of increasing from 0.6 to 1.4 with a step size of 0.2, compared with the other three algorithms (MADDQN algorithm, MADDPG algorithm and AD-MADDPG w / o LSTM algorithm), the AD-MADDPG algorithm of the present invention has always shown significant performance advantages in terms of load recovery rate and energy consumption ratio. Specifically, Fig.14 As shown in Figure 2, when using the AD-MADDPG algorithm, the load recovery rate of the microgrid group is increased by an average of 9.6% compared with the AD-MADDPG w / o LSTM algorithm, an average of 19.8% compared with the MADDPG algorithm, and an average of 25.5% compared with the MADDQN algorithm; at the same time, Fig.15 As shown in the figure, when using the AD-MADDPG algorithm, the energy consumption of EVs is reduced by an average of 8.2% compared with the AD-MADDPG w / o LSTM algorithm, by an average of 15.7% compared with the MADDPG algorithm, and by an average of 23.6% compared with the MADDQN algorithm.

[0329] In addition, if Fig.14 As shown, as the maximum moving distance increases, when the maximum moving distance reaches 1.0 to 1.4, the AD-MADDPG algorithm of the present invention improves the load recovery rate of the microgrid group MMGs to a plateau, while the energy consumption ratio shows slight fluctuations. This is because the increase in the maximum moving distance of EVs within a single time step means that in the entire training round, EVs can cover a larger distance within a fixed time step. This shows that in the elastic load recovery scheduling process of the isolated microgrid group MMGs, EVs can reach farther microgrids MGs, and the AD-MADDPG algorithm of the present invention can still adaptively provide flexible routing and charging and discharging scheduling, thereby continuously improving the collaborative performance of power scheduling of the entire power system.

[0330] Embodiment 2:

[0331] Based on the same inventive concept as the first embodiment, a method for electric vehicle-involved power dispatching is introduced. The AD-MADDPG algorithm process of step S3 further includes:

[0332] S38: After the update of each deep neural network parameter of the learner is completed in step S37, the priority of the experience samples in the global experience replay pool is updated according to the priority calculation method in the priority replay mechanism of step S33;

[0333] and / or,

[0334] The AD-MADDPG algorithm process of step S3 also includes:

[0335] S39: After the update of each deep neural network parameter of the learner is completed in step S37, the updated deep neural network parameters of the learner are copied to update the deep neural network parameters of the explorer.

[0336] In this embodiment, sub-step S38 is specifically implemented as follows:

[0337] After all neural network parameters of the learner are updated, the new TD-error is calculated according to formula (5) and the priority of the experience samples in the global experience replay pool is updated. The larger the TD-error, the higher the priority of the experience sample.

[0338] In the traditional experience replay mechanism, experience samples are randomly sampled, which may waste a lot of computing resources on unimportant experience samples. Priority experience replay frequently samples experience samples with large TD-error, so that the model can correct the misestimation of unpredictable states or actions more quickly and accelerate convergence.

[0339] In this embodiment, sub-step S39 is specifically implemented as follows:

[0340] DNNs parameters updated by copying the learner (Learner) , , , ) to update the network parameters of the local explorer (Actor).

[0341] The multiple explorers of the present invention have the same neural network structure as a single learner.

[0342] The only difference is that the explorer conducts local exploration based on its own private observations, while the learner learns based on the experience samples of all explorers, and then sends the learned parameters to all explorers, that is, distributed exploration and centralized training. When the algorithm training reaches the preset number of Episode iterations, the training ends.

[0343] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0344] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0345] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0346] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0347] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for dispatching electric vehicles, characterized in that: include: S1: Construct a transportation network-microgrid coupling network model and determine the goal of power dispatching: minimize the dispatching cost of electric vehicles to achieve maximum load recovery of the microgrid group; S2: construct a partially observable Markov decision process based on the model of step S1 and the power dispatch target; S3: Design the AD-MADDPG algorithm based on the partially observable Markov decision process in step S2; The AD-MADDPG algorithm is used to solve the power dispatching method that achieves the power dispatching target according to the model constructed in step S1; The AD-MADDPG algorithm is an improved algorithm based on the DDPG algorithm that combines the attention mechanism and the multiple explorers-single learner framework.

2. The electric power dispatching method involving electric vehicles according to claim 1, characterized in that: The transportation network-microgrid coupling network model of step S1 includes a transportation network model and a microgrid model. The transportation network model is used to collect traffic information, and the microgrid model is used to collect distribution network information. The transportation network-microgrid coupling network model is used to seek the optimal mobile route and charging / discharging scheduling of electric vehicles to achieve the power scheduling target of step S1.

3. The electric power dispatching method involving electric vehicles according to claim 1, characterized in that: The partially observable Markov decision process POMDP in step S2 Time state space Defined as: , consists of three channels: The first channel in the state space Includes geographical location of charging / discharging stations h and the microgrid associated with h Load reduction ,Right now ; where h represents the charging / discharging station number, H represents the total number of charging / discharging stations, represents a collection of charging / discharging stations, and Respectively represent the horizontal and vertical coordinates of the charging / discharging station h in the map; The second channel of the state space Including the coordinates of the vertex of the traffic network road and roads Traffic flow ,Right now ; Where V represents the total number of road vertices, represents the set of road vertices, represents the road vertex, , , Represents the total number of road vertices, Represented by vertex Starting point The road to the end, and Represents vertices The horizontal and vertical coordinates in the map; Indicates the road exist Real-time traffic flow at all times, Real-time traffic flow at all times The calculation formula is as follows: ， in, Indicates the road Other types of vehicles with specific daily travel patterns, Indicates the road The number of electric vehicles participating in power dispatch; The third channel of state space Includes the current location of the electric vehicle and remaining power ,Right now ; where i represents the electric vehicle number, represents the total number of electric vehicles, represents the set of electric vehicle numbers, and Respectively indicate the number The horizontal and vertical coordinates of electric vehicles on the map; Action space of the partially observable Markov decision process POMDP in step S2 Defined as ; in, Indicates the moving direction of the electric vehicle, where the moving direction of the electric vehicle is the clockwise rotation angle of the front of the vehicle. Indicates the range of values ​​of the moving direction of the electric vehicle, Indicates that the electric vehicle is at a time step The moving distance within Indicates that the electric vehicle is The maximum moving distance within Indicates that the electric vehicle is at a time step The range of the moving distance. Indicates the maximum capacity of an EV relative to its battery The charge-discharge ratio, Indicates the maximum capacity of an EV relative to its battery The range of charge and discharge ratio, It means charging electric vehicles. A negative value indicates that the electric vehicle is discharging; The partially observable Markov decision process POMDP of step S2 is numbered of electric vehicles in Instant rewards at the moment The function calculation formula is as follows: ， in, Indicator with charging / discharging station Associated microgrids load reduction; represents a preset unit load reduction cost factor; Indicates when the number When an electric vehicle moves beyond the road boundary or does not participate in the charging and discharging scheduling of the microgrid group, the penalty imposed on it is a preset constant term; Indicates the number The discharge of electric vehicles, Indicates the number The charging status of electric vehicles, including ; Represents the energy consumption cost of electric vehicles participating in power dispatching.

4. The electric power dispatching method involving electric vehicles according to claim 3, characterized in that: The energy consumption cost of electric vehicles participating in power dispatching The calculation formula is as follows: ， in, represents the preset electric vehicle battery depreciation cost factor; Indicates the moving distance of the electric vehicle; represents the road travel time of electric vehicles; and They respectively represent the preset electric vehicle unit distance movement energy consumption cost coefficient and unit time cost coefficient; Road travel time According to real-time traffic flow Calculated, the road travel time The calculation formula is as follows: ， in, It indicates the travel time when the road is not congested. and A preset coefficient indicating road congestion, Indicates the preset maximum capacity of the road.

5. The electric power dispatching method involving electric vehicles according to claim 1 or 3, characterized in that: The explorer of the AD-MADDPG algorithm in step S3 is a single electric vehicle; The explorer and learner of the AD-MADDPG algorithm include the same deep neural network structure; The deep neural network parameter update process of the AD-MADDPG algorithm learner includes: S31: according to the state space in the partially observable Markov decision process in step S2, using a convolutional neural network to extract local features in the state space data and output private observations; S32: Input the private observation outputted from step S31 into the learner’s Actor network, combine it with the attention mechanism, and output the action; S33: Each electric vehicle executes the action of step S32, generates experience samples and stores them in the corresponding local experience replay pool. When the local experience replay pool is full, the experience samples are processed using the priority replay mechanism and stored in the global experience replay pool; Each electric car corresponds to a local experience playback pool; S34: loop through step S33 until the global experience replay pool is full; When the global experience replay pool in step S34 is full, start sampling experience samples and generate new experience samples; S35: Use the new experience sample in step S34 to train the deep neural network, update the learner's Critic network online by minimizing the loss function and calculate the predicted Q value of the current state and action of the Critic network; S36: Based on the predicted Q value calculated in step S35, the learner's Actor network is updated online by the policy gradient method; S37: Soft-update the learner's target Actor network and target Critic network using the Actor network updated in step S36 and the Critic network updated in step S35.

6. The electric power dispatching method involving electric vehicles according to claim 5, characterized in that: The calculation formula of the attention mechanism in step S32 is: ， in, Represents a new private observation generated by the attention mechanism; Softmax() is an activation function; is the weight matrix in the Softmax function, the weight matrix The initial value of each element is generated by a random initialization strategy; Represents a private observation of the input.

7. The electric power dispatching method involving electric vehicles according to claim 5, characterized in that: Step S33 includes: S331: After each electric vehicle performs the action of step S32, an instant reward is calculated respectively; S332: Obtain the next state according to the current state, the action taken, and the state transition function; S333: using the private observation function to obtain the next private observation of the local environment from the next state, and generating an experience sample including the current private observation, action, immediate reward and the next private observation; S334: Calculate the priorities of the experience samples in step S333 respectively, add the calculated priorities of the experience samples to the experience samples in step S333, and store the experience samples with added priorities into the corresponding local experience playback pool. Among them, the priority The calculation formula is: ， in, Represents experience sample The target Q value of for The state space at time, Respectively represent EV 1~EV ​​I The action of the moment, For the state space of the Critic network Next select action The predicted Q value of S335: loop through steps S31-S334 until the local experience replay pool is full; S336: Select the priority calculated in step S334 from the local experience replay pool filled in step S335 Extract experience samples and add the corresponding explorer ID to the extracted experience samples; S337: Store the experience samples extracted in step S336 and adding the explorer ID into the global experience replay pool.

8. The electric power dispatching method involving electric vehicles according to claim 5, characterized in that: Step S34 includes: S341: loop through step S33 until the global experience replay pool is full; S342: sampling a preset batch number of experience samples from the global experience replay pool according to the priority replay mechanism of step S33; The preset batch number of empirical samples is a sequence consisting of multiple continuous time step empirical samples; S343: Calculate the cumulative reward of the experience samples sampled in step S342 , cumulative rewards The calculation formula is as follows: ， in, is the discount factor, which is a preset value between 0 and 1; Represents the discount factor N-1 power; express Instant rewards of the moment; S344: Using a long short-term memory (LSTM) neural network to process the empirical samples sampled in step S342 to generate new private observations; S345: The new private observation generated in step S344 and the cumulative reward calculated in step S343 replace the original private observation and the immediate reward respectively, and a new experience sample is obtained.

9. The electric power dispatching method involving electric vehicles according to claim 8, characterized in that: Step S342 samples Time to Moment Empirical samples of time steps, Step S344 includes: S3441: Obtain two state observation sequences from the global experience replay pool according to the experience samples sampled in step S342, The two state observation sequences are Private observation at all times to Private observation at all times Collection and Private observation at all times to Private observation at all times Collection ; in, is the preset batch number sampled in step S342 A positive integer in the range, that is ; S3442: Input the two state observation sequences obtained in step S3441 into the LSTM network and output new private observations respectively. and .

10. The electric power dispatching method involving electric vehicles according to claim 5 or 7, characterized in that: The AD-MADDPG algorithm process of step S3 also includes: S38: After the update of each deep neural network parameter of the learner is completed in step S37, the priority of the experience samples in the global experience replay pool is updated according to the priority calculation method in the priority replay mechanism of step S33; and / or, The AD-MADDPG algorithm process of step S3 also includes: S39: After the update of each deep neural network parameter of the learner is completed in step S37, the updated deep neural network parameters of the learner are copied to update the deep neural network parameters of the explorer.

Images