Optimal scheduling method for electric vehicles to participate in energy storage market based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based optimal scheduling method for an electric vehicle to participate in an energy storage market, and relates to the technical field of energy management, and the method comprises the steps: constructing a vehicle intelligent agent and an optimization target of the vehicle intelligent agent; dynamically executing the action of each vehicle agent and pre-introducing a reward function for optimizing a target weight coefficient so as to coordinate the charging sequence among the vehicle agents; constructing a schedulable amount solving model of the vehicle intelligent agents, and solving the schedulable amounts of the vehicle intelligent agents in different areas; and taking the schedulable amount of the vehicle intelligent agent as the input of the neural network model, and outputting an optimal charging planning strategy to realize the optimal scheduling of the vehicle intelligent agent cluster. By establishing private car, commercial car and taxi intelligent agents and utilizing the reinforcement learning algorithm to optimize the operation strategy of the electric vehicle, the problem that the electric vehicle is connected to the power grid in a large scale to cause impact on the power system is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of energy management, and in particular to an optimal scheduling method for electric vehicles participating in an energy storage market based on reinforcement learning. Background Art

[0002] The development and popularization of flexible resources have great prospects, and distributed photovoltaics are developing rapidly. However, photovoltaic power generation is greatly affected by the outside world, power supply reliability is low, stability is poor, and direct scheduling is difficult. As a mobile energy storage carrier, electric vehicles have seen rapid market development in recent years. With the increase in the number of electric vehicles and the development of electric vehicle-to-grid technology, scheduling through aggregator electric vehicle clusters can reduce the impact of large-scale access of new energy and electric vehicles to the grid on its stability, while also creating certain economic benefits for electric vehicle users.

[0003] However, due to the existence of many uncertain factors in the system, such as load demand, time when electric vehicles are connected to the system, power status, time when electric vehicles leave the system, and power required when leaving, these factors are highly random and dynamically changeable, and traditional methods are difficult to accurately model and optimize the charging and discharging of electric vehicles. Deep reinforcement learning can effectively handle dynamic and complex environments. Through deep reinforcement learning, the charging and discharging scheduling of electric vehicles can better adapt to the real-time changes of the power system and improve the accuracy and efficiency of scheduling. However, at this stage, the application of deep reinforcement learning in electric vehicle scheduling is still relatively limited.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] In response to the problems in the related technology, the present invention proposes an optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning to overcome the above-mentioned technical problems existing in the existing related technology.

[0006] To this end, the specific technical solution adopted by the present invention is as follows:

[0007] An optimal dispatching method for electric vehicles to participate in an energy storage market based on reinforcement learning, the optimal dispatching method for electric vehicles to participate in an energy storage market includes:

[0008] S1. Based on the expected charging power characteristics of electric vehicles, construct a vehicle agent and the optimization goal of the vehicle agent;

[0009] S2. Establish a corresponding Markov environment based on the optimization goal of the vehicle agent. In the Markov environment, dynamically execute the actions of each vehicle agent and pre-introduce a reward function with an optimization goal weight coefficient to coordinate the charging sequence between the vehicle agents.

[0010] S3. Based on the adaptive conjugate gradient method, a model for solving the dispatchable amount of vehicle intelligent bodies is constructed, and the dispatchable amount of vehicle intelligent bodies in different areas is solved by combining the multiplier alternating direction algorithm and the sequential quadratic programming algorithm;

[0011] S4. Use the dispatchable amount of the vehicle agent as the input of the neural network model, iteratively train the neural network model, and output the optimal charging planning strategy to achieve the optimal scheduling of the vehicle agent cluster.

[0012] Preferably, based on the expected charging power characteristics of the electric vehicle, the vehicle agent and the optimization objectives of the vehicle agent are constructed including:

[0013] S11, obtaining expected charging power characteristics of different types of electric vehicles, where the types of electric vehicles include private cars, commercial vehicles and taxis;

[0014] S12, establishing the optimization objective function and vehicle intelligent body of private cars and commercial vehicles respectively with minimizing charging cost and reducing the overall peak power consumption of microgrid as the optimization goal; establishing the optimization objective function and vehicle intelligent body of taxi with minimizing charging time as the optimization goal;

[0015] S13. Based on the expected power characteristics of charging, the power curve of the vehicle intelligent body during the charging process is maintained within the upper and lower bounds of the power curve, so that the vehicle intelligent body reaches the expected power before the charging is completed.

[0016] Preferably, based on the expected power characteristics of charging, keeping the power curve of the vehicle intelligent body during the charging process within the upper and lower bounds of the power curve so that the vehicle intelligent body reaches the expected power before the charging ends includes:

[0017] S131, obtaining the aggregator power of the private car before charging begins, and generating upper and lower bound curves of power during the charging process of the private car based on the charging and discharging actions of the private car;

[0018] S132, generating upper and lower bound curves of electric energy of the commercial vehicle and the taxi based on the charging actions of the commercial vehicle and the taxi;

[0019] S133. Keep the power curves of the private car, commercial vehicle and taxi during the charging process within the corresponding upper and lower bounds of the power curve to ensure that the vehicle intelligent body reaches the expected power before the charging is completed.

[0020] Preferably, a corresponding Markov environment is established based on the optimization goal of the vehicle agent. In the Markov environment, the actions of each vehicle agent are dynamically executed and a reward function with an optimization goal weight coefficient is pre-introduced to coordinate the charging sequence between the vehicle agents, including:

[0021] S21. Based on the optimization goal of the vehicle agent, a corresponding Markov game environment is established, and in the Markov game environment, the vehicle agent action is dynamically executed in combination with the environment state;

[0022] S22, based on the optimization goal of the vehicle intelligent body, establishing the state space and action space of the vehicle intelligent body, and reflecting the decision factors of the charging sequence of the vehicle intelligent body based on the state space and action space;

[0023] S23. Introduce an optimization target weight coefficient to establish a reward function for the vehicle agent, and coordinate the charging sequence between the vehicle agents based on the reward function.

[0024] Preferably, based on the optimization goal of the vehicle agent, establishing the state space and action space of the vehicle agent includes:

[0025] S221. Based on the optimization objectives of private cars and commercial vehicles, the state space of private cars and commercial vehicles is established, and the expression of the state space is:

[0026]

[0027] S222. Based on the optimization goal of the taxi, a state space of the taxi is established, and the expression of the state space is:

[0028]

[0029] S223. The charging and discharging power of private cars, commercial vehicles and taxi aggregators is used as the action space, and the expression of the action space is:

[0030] a t = {P t};

[0031] Where, t a t1 and t2 represent the arrival time of the first vehicle agent to arrive at the charging pile and the departure time of the last vehicle agent to leave the charging pile under the current aggregator; N t represents the number of vehicle agents under the aggregator at time t; e t 、e n They represent the power of the aggregator and the expected power of the aggregator at time t respectively; They represent the upper and lower bounds of the power of the aggregator at time t respectively; P t up represents the net load of the upper microgrid at time t; price t represents the real-time electricity price of the main power grid at time t; a t represents the action space at time t; S t represents the state space at time t; P t is the charge and discharge power at time t.

[0032] Preferably, based on the adaptive conjugate gradient method, a schedulable quantity solution model for vehicle agents is constructed, and combined with the multiplier alternating direction algorithm and the sequential quadratic programming algorithm, the schedulable quantities of vehicle agents in different regions are solved, including:

[0033] S31, constructing a schedulable quantity solution model for the vehicle intelligent body, and introducing slack variables to convert inequality constraints based on the schedulable quantity solution model into equality constraints;

[0034] S32. Solve the dispatchable quantity solution model based on sequential quadratic programming theory to obtain a complete Lagrangian function;

[0035] S33, decomposing the complete Lagrangian function into a number of optimization variable subproblems and relaxation variable subproblems;

[0036] S34. Introduce an adaptive conjugate gradient algorithm into the sub-problem, and update the residual to accelerate the conjugate gradient algorithm to generate the dispatchable quantities of vehicle agents in different areas after convergence.

[0037] Preferably, an adaptive conjugate gradient algorithm is introduced into the sub-problem, and the schedulable quantities of vehicle agents in different regions generated after the residual acceleration conjugate gradient algorithm converges are updated, including:

[0038] S341, obtaining vehicle intelligent body data and initial values ​​of vehicle intelligent body optimization variables at a preset time;

[0039] S342, solving several optimization variable sub-problems and slack variable sub-problems to obtain residuals, and comparing the residuals with a preset threshold; if the residuals are less than or equal to the preset threshold, the iteration is stopped to obtain the dispatchable amount of vehicle agents in different areas; if the residuals are greater than the preset threshold, step S343 is executed;

[0040] S343, calculating the residual radius, updating the residual based on the residual radius to accelerate the convergence of the conjugate gradient algorithm, and generating the dispatchable amount of vehicle intelligent bodies in different areas.

[0041] Preferably, the dispatchable amount of the vehicle agent is used as the input of the neural network model, the neural network model is iteratively trained, and the optimal charging planning strategy is output to achieve the optimal dispatch of the vehicle agent cluster, including:

[0042] S41. Establish a neural network model based on the policy network and value network in the reinforcement learning algorithm, and use the dispatchable amount of the vehicle intelligent body as the input of the neural network model;

[0043] S42, iteratively training the neural network model to obtain an initial state and an initial action, superimposing noise on the initial action to obtain a random action, and interacting the random action with the environment to obtain a cumulative reward;

[0044] S43, storing the sample in the experience memory bank, determining whether the capacity of the experience memory bank meets the sampling requirements, and updating the neural network model parameters, state and accumulated rewards based on the determination result;

[0045] S44, determine whether the iterative training is finished. If the iterative training is finished, output the cumulative reward and action quality of the current round and then execute step S45. Otherwise, repeat steps S42-S44;

[0046] S45. Evaluate the cumulative rewards and action quality of the current round, generate an optimal charging planning strategy based on the evaluation results, and execute optimal scheduling of the vehicle intelligent body cluster based on the optimal charging planning strategy.

[0047] Preferably, storing the samples in the experience memory, determining whether the capacity of the experience memory meets the sampling requirements, and updating the neural network model parameters, status and accumulated rewards based on the determination results include:

[0048] S431, configuring an experience memory for the vehicle agent and storing samples in the experience memory, and each vehicle agent compares the instant reward value of the experience obtained with the average reward value of all experiences in the current experience memory;

[0049] S432, if the experience instant reward value is greater than the experience average reward value, the corresponding sample is classified into a high-importance experience memory bank, otherwise, the corresponding sample is classified into a low-importance experience memory bank;

[0050] S433. Determine whether the capacity of the experience memory bank after sample scheduling meets the sampling requirements. If the experience memory bank is full, update the neural network model parameters, status and accumulated rewards.

[0051] Preferably, the expression of the optimization variable subproblem is:

[0052]

[0053] The expression of the slack variable subproblem is:

[0054]

[0055] In the formula, f s , g s 、h s denote the objective function, equality constraint and inequality constraint of region s respectively; x s represents the optimization variable in region s; They represent the objective function of region s in x s The gradient at and the inequality constraints of region s are in The gradient at represents the optimization variable in the corresponding region s of the kth iteration; represents the k-th iteration descent direction matrix of region s; Both represent the Lagrange multipliers of the kth iteration; represents the Hessian matrix of the k-th iteration objective function of region s for variables; β1 and β2 both represent penalty factors; β k represents the penalty factor for the kth iteration; d k represents the k-th iteration descent direction matrix; r k+1 represents the residual of the k+1th iteration, z k represents the slack variable for the kth iteration; g z represents the objective function of region s after adding slack variables.

[0056] The beneficial effects of the present invention are:

[0057] 1. The present invention can limit the scope of strategy changes through improved reinforcement learning algorithms and proximal strategy optimization algorithms, avoid excessive strategy updates, and prevent the algorithm from falling into an unstable state, thereby improving robustness. In the training process of the environment, an experience replay mechanism is adopted to obtain transfer samples through online processing, and the transfer samples obtained by the interaction between the intelligent agent and the environment are stored in the memory unit, which reduces the correlation between samples and thereby improves the stability of the algorithm.

[0058] 2. The optimal scheduling method for electric vehicles participating in the energy storage market provided by the present invention can transform urban electric vehicles into mobile energy storage modules, which can not only reduce the impact of large-scale access of electric vehicles to the power grid on the stability of the power grid, but also create certain economic benefits for electric vehicle users. In addition, due to the strong randomness of charging and discharging of electric vehicles, traditional methods are difficult to accurately model and optimize. The deep reinforcement learning provided by the present invention can effectively handle dynamic and complex environments, adapt to real-time changes, and thus improve the system scheduling performance.

[0059] 3. The present invention solves the problem that large-scale access of electric vehicles to the power grid will have an impact on the power system by establishing private cars, commercial vehicles, and taxi intelligent entities and using reinforcement learning algorithms to optimize the operation strategies of electric vehicles, so that aggregators can determine the current charging and discharging range of electric vehicles based on key information in the system, thereby achieving optimal scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 is a flow chart of an optimal scheduling method for electric vehicles participating in an energy storage market based on reinforcement learning according to an embodiment of the present invention;

[0062] Figure 2 It is a flow chart of an algorithm for solving the dispatchable quantity of electric vehicles by combining the multiplier alternating direction method and the sequential quadratic programming method in an optimal dispatch method for electric vehicles participating in the energy storage market based on reinforcement learning according to an embodiment of the present invention;

[0063] Figure 3 This is a flow chart of a reinforcement learning algorithm that considers gradient random perturbation in an optimal scheduling method for electric vehicles participating in an energy storage market based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments. They can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementation methods and advantages of the present invention.

[0065] According to an embodiment of the present invention, an optimal scheduling method for electric vehicles participating in an energy storage market based on reinforcement learning is provided.

[0066] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an optimal scheduling method for electric vehicles to participate in the energy storage market based on reinforcement learning according to an embodiment of the present invention, the optimal scheduling method for electric vehicles to participate in the energy storage market includes:

[0067] S1. Based on the expected charging power characteristics of electric vehicles, construct a vehicle agent and the optimization objectives of the vehicle agent.

[0068] Among them, based on the expected charging power characteristics of electric vehicles, the vehicle intelligent body and the optimization goals of the vehicle intelligent body include:

[0069] S11, obtaining expected charging power characteristics of different types of electric vehicles, where the types of electric vehicles include private cars, commercial vehicles and taxis;

[0070] S12, establishing the optimization objective function and vehicle intelligent body of private cars and commercial vehicles respectively with minimizing charging cost and reducing the overall peak power consumption of microgrid as the optimization goal; establishing the optimization objective function and vehicle intelligent body of taxi with minimizing charging time as the optimization goal;

[0071] S13. Based on the expected power characteristics of charging, the power curve of the vehicle intelligent body during the charging process is maintained within the upper and lower bounds of the power curve, so that the vehicle intelligent body reaches the expected power before the charging is completed.

[0072] Wherein, based on the expected power characteristics of charging, the power curve of the vehicle intelligent body during the charging process is kept within the upper and lower bounds of the power curve so that the vehicle intelligent body reaches the expected power before the charging is completed, including:

[0073] S131, obtaining the aggregator power of the private car before charging begins, and generating upper and lower bound curves of power during the charging process of the private car based on the charging and discharging actions of the private car;

[0074] S132, generating upper and lower bound curves of electric energy of the commercial vehicle and the taxi based on the charging actions of the commercial vehicle and the taxi;

[0075] S133. Keep the power curves of the private car, commercial vehicle and taxi during the charging process within the corresponding upper and lower bounds of the power curve to ensure that the vehicle intelligent body reaches the expected power before the charging is completed.

[0076] It should be noted that according to the different charging requirements of electric vehicles, the establishment of three types of intelligent agents, namely private cars, taxis, and buses (electric vehicles and intelligent agents are both vehicle intelligent agents) includes the following steps:

[0077] (1) Combining the charging characteristics of the three different electric vehicles, private cars and commercial vehicles are arranged to be charged during the time when taxis have completed charging, which can reduce the overall peak power consumption of the power grid. Therefore, the optimization goal of the charging process of private cars and commercial vehicles is to minimize the charging cost and reduce the overall peak power consumption of the microgrid. Among them, the optimization objective function of the charging process of private cars can be expressed as:

[0078]

[0079] Where f1 is the charging cost of a private car in the microgrid at time Δt; f2 is the output function of the electric vehicle in the microgrid; P t C , P t SW , P t SJ are the charging powers of taxis, commercial vehicles, and private electric vehicles at time t respectively; are the electricity demand of taxis, private cars and commercial vehicles respectively; P ave is the net load power value of the upper layer at time t; T is the total duration of electric vehicle charging scheduling; Price t is the real-time electricity price at time t.

[0080] The optimization target of electric commercial vehicles can be simply replaced by the following formula (2):

[0081]

[0082] Taxis need to complete charging as quickly as possible, so the optimization goal of the electric vehicle charging process in this mode is to minimize the charging time. The optimization objective function can be expressed as:

[0083]

[0084] Where f3 is the charging time function; are the sum of the electricity of all taxis at time t and time t-1 respectively.

[0085] (2) In order to ensure that various types of electric vehicles reach the expected power before the end, the charging and discharging power of each electric vehicle (EV) is considered constant in each time period, and the charging and discharging model of a single electric vehicle is shown in formula (7):

[0086]

[0087] Where, t a t1 and t2 are the arrival time of the first electric vehicle to arrive at the charging pile and the departure time of the last electric vehicle to leave the charging pile under the current aggregator; e t is the battery state of charge (SOC) at time t; e a 、e1、e u are the SOC of the electric vehicle at the time of arrival and departure and the SOC required by the user; η is the charge and discharge efficiency; c is the battery capacity; Δt is the time interval; P t is the charge and discharge power at time t; P t dis,max and P t ch,max The charging power P is t ch and discharge power P t dis The maximum value of .

[0088] For electric vehicle users, it is necessary that the SOC reaches the required value when leaving. During the entire charging process, as long as the electric energy curve of the electric vehicle is kept within the upper and lower bounds of the electric energy curve, it can be guaranteed that the expected power will be reached before the end. For private cars, the aggregator power before the start is e a If you charge to the upper limit of the energy at the maximum power immediately after starting max and remain unchanged. The electric energy curve obtained in this process is used as the upper limit curve d max .

[0089] If the maximum power is immediately discharged to the lower limit of electric energy e min, and charge at maximum power before the aggregator finishes charging, ensuring that the aggregator reaches the expected power e when the aggregator finishes charging n The electric energy curve obtained in this process is used as the electric energy lower limit curve d min .

[0090] For commercial vehicles and taxis, if you start charging at maximum power immediately to the energy limit e max The electric energy curve obtained in this process is used as the upper limit curve of electric energy d max If the battery is not charged at the beginning and is charged at the maximum power before the end of charging, it is ensured that the expected power of the aggregator is reached when the aggregator ends charging. The power curve obtained in this process is used as the power lower limit curve d min .

[0091] S2. Establish a corresponding Markov environment based on the optimization goal of the vehicle agent. In the Markov environment, dynamically execute the actions of each vehicle agent and pre-introduce a reward function with an optimization goal weight coefficient to coordinate the charging sequence between the vehicle agents.

[0092] Among them, a corresponding Markov environment is established based on the optimization goal of the vehicle agent. In the Markov environment, the actions of each vehicle agent are dynamically executed and the reward function of the optimization target weight coefficient is pre-introduced to coordinate the charging sequence between the vehicle agents, including:

[0093] S21. Based on the optimization goal of the vehicle agent, a corresponding Markov game environment is established, and in the Markov game environment, the vehicle agent action is dynamically executed in combination with the environment state;

[0094] S22. Based on the optimization goal of the vehicle intelligent body, a state space and an action space of the vehicle intelligent body are established, and decision factors of the charging sequence of the vehicle intelligent body are reflected based on the state space and the action space.

[0095] Among them, based on the optimization goal of the vehicle intelligent body, the state space and action space of the vehicle intelligent body are established, and the decision factors of the charging sequence of the vehicle intelligent body reflected based on the state space and action space include:

[0096] S221. Based on the optimization goals of private cars and commercial vehicles, establish the state space of private cars and commercial vehicles;

[0097] S222. Based on the optimization goal of the taxi, establish a state space of the taxi;

[0098] S223. Use the charging and discharging power of private cars, commercial vehicles and taxi aggregators as action space.

[0099] S23. Introduce an optimization target weight coefficient to establish a reward function for the vehicle agent, and coordinate the charging sequence between the vehicle agents based on the reward function.

[0100] It should be noted that the corresponding Markov environment is established according to the different optimization objectives of different electric vehicle charging requirements, including:

[0101] (1) When the aggregator makes charging and discharging plans, it is necessary not only to achieve the charging targets of the three different types of electric vehicles, but also to coordinate the charging sequence of the three types of electric vehicles to prevent the three types of vehicles from charging in large numbers at the same time, causing a peak in electricity consumption and affecting the stable operation of the microgrid.

[0102] However, the single-agent algorithm cannot handle the interests of multiple agents well, nor can it achieve coordination and cooperation among agents. In the Markov game, the three agents, private cars, taxis, and buses, simultaneously select and execute their own actions based on the current environmental state. The action set of all agents jointly determines the transfer and update of the environmental state and the reward function of each agent. The Markov game process can be expressed as {S1,...S n ,A1...A n ,P,R1...R n ,γ}, where S n , A n Respectively represent the state information set and action information set of agent n, R n represents the reward information of agent n, P and γ represent the probability of environmental state transition and the discount factor respectively. At this time, the expected cumulative reward obtained by agent i at time t can be expressed as:

[0103]

[0104] Where E[X] represents the expected calculation of the accumulated future rewards under strategy X; represents the expected cumulative reward obtained by agent i at time t; S t Represents the state of the agent at time t, and selects action a according to the current strategy π t After executing the action, the current reward value γ is obtained from the environment feedback t and the next state S t+1 , S0 represents the initial state; π′,π - ′ represents the Nash equilibrium strategy of agent i; R i (s t ,a t ,s t+1 ) represents the reward information of agent i at time t. The discount factor γ determines the importance of the current reward value to the future reward value, and is usually taken as γ∈[0,1].

[0105] In Markov games, reaching Nash equilibrium is generally used as the criterion for algorithm convergence. Nash equilibrium refers to a fixed point reached in multiple agents. For any agent, it is impossible to obtain a higher cumulative return by adopting other strategies. It can be expressed as:

[0106]

[0107] In the formula, π′, π -′ A represents the Nash equilibrium strategy of agent i; gents Represents the set of all agents.

[0108] (2) Design of state-action space:

[0109] The optimization goals of the private car aggregator and the commercial vehicle aggregator are to minimize the charging cost and reduce the overall peak power consumption of the microgrid, so the same state space can be designed. The state space designs of the two are as follows:

[0110]

[0111] Where, t a ,t l are the arrival time of the first electric vehicle to arrive at the charging pile under the aggregator and the departure time of the last electric vehicle to leave the charging pile. These two variables determine the duration for which the aggregator participates in charging planning; N t is the number of electric vehicles under the aggregator at time t, which determines the maximum charging and discharging power of the aggregator at time t; e t 、e n are the power of the aggregator at time t and the expected power of the aggregator, are the upper and lower bounds of the power of the aggregator at time t respectively; P t up is the net load of the upper microgrid at time t; price t is the real-time electricity price of the main power grid at time t; a t Represents the action taken by the aggregator at time t.

[0112] Since the only optimization goal of electric taxis is to minimize the charging time, there is no need to refer to the real-time electricity price of the microgrid and the upper-level dispatch net load information. The state space design of the taxi aggregator can be expressed as:

[0113]

[0114] The action spaces of the three aggregators are the same, which are the charge and discharge powers corresponding to the aggregators, namely:

[0115] a t = {P t} (12)

[0116] Where P t is the charging (discharging) power of the aggregator at time t, P t Greater than 0 means charging, less than 0 means discharging.

[0117] (3) Design of reward function:

[0118] Different electric vehicle aggregators have different optimization goals, which need to be distinguished when designing the reward function. The optimization goals of private cars and commercial vehicles during the charging process are mainly to minimize the charging cost and reduce the overall peak power consumption of the microgrid. Combined with the proposed power boundary technology, the constraint of electric vehicles reaching the expected power is converted into the form of a reward function, which can be expressed as:

[0119]

[0120] In the formula, e t is the aggregator power at time t; are the upper and lower bounds of the aggregator’s energy at time t. As can be seen from the above formula, when the aggregator’s energy exceeds the upper and lower bounds of energy, the reward value of this part is negative, and the more it exceeds, the smaller the reward value. Secondly, the reward function corresponding to the optimization goal of reducing the charging cost of electric vehicles can be expressed as:

[0121]

[0122] In the formula, a t is the action taken by the aggregator at time t, price t is the real-time electricity price of the main power grid at time t.

[0123] Finally, the reward function corresponding to the optimization goal of reducing the overall peak power consumption of the microgrid can be expressed as:

[0124]

[0125] In the formula, They represent the sum of the charging (discharging) power of the fast taxi, commercial vehicle, and private electric vehicle at time t respectively; Represents the sum of the electricity demand of taxis, commercial vehicles, and private electric vehicles; P t up It represents the net load power value of the upper layer dispatching at time t; T represents the total dispatching time of electric vehicles.

[0126] From the above formula, we can see that when the net load of the entire microgrid system exceeds the average power of the microgrid system at each moment, the reward value of this part is negative, and the more it exceeds, the smaller the reward value. Therefore, the reward functions of private cars and commercial vehicles can be expressed as:

[0127]

[0128] In the formula, r t SJ 、r t SW They are the reward functions for private cars and commercial vehicles respectively; λ1, μ1, σ1 are the weight coefficients of different optimization objectives for private cars respectively; λ2, μ2, σ2 are the weight coefficients of different optimization objectives for commercial vehicles respectively.

[0129] The optimization goal of electric taxis during the charging process is mainly to minimize the charging time, and its reward function can be expressed as:

[0130]

[0131] Where λ3 and μ3 are weight coefficients of different optimization objectives of taxis; t energy represents the reward function for the constraint that the electric vehicle reaches the expected power; r t fast Reward function for minimizing the charging time constraint.

[0132] S3. Based on the adaptive conjugate gradient method, a model for solving the dispatchable quantity of vehicle intelligent bodies is constructed, and the dispatchable quantity of vehicle intelligent bodies in different areas is solved by combining the multiplier alternating direction algorithm and the sequential quadratic programming algorithm.

[0133] Among them, based on the adaptive conjugate gradient method, a model for solving the schedulable amount of vehicle intelligent bodies is constructed, and combined with the multiplier alternating direction algorithm and the sequential quadratic programming algorithm, the schedulable amount of vehicle intelligent bodies in different areas is solved, including:

[0134] S31, constructing a schedulable quantity solution model for the vehicle intelligent body, and introducing slack variables to convert inequality constraints based on the schedulable quantity solution model into equality constraints;

[0135] S32. Solve the dispatchable quantity solution model based on sequential quadratic programming theory to obtain a complete Lagrangian function;

[0136] S33, decomposing the complete Lagrangian function into a number of optimization variable subproblems and relaxation variable subproblems;

[0137] S34. Introduce an adaptive conjugate gradient algorithm into the sub-problem, and update the residual to accelerate the conjugate gradient algorithm to generate the dispatchable quantities of vehicle agents in different areas after convergence.

[0138] Among them, the adaptive conjugate gradient algorithm is introduced into the sub-problem, and the residual is updated to accelerate the conjugate gradient algorithm to converge. The dispatchable quantities of vehicle agents in different regions are generated, including:

[0139] S341, obtaining vehicle intelligent body data and initial values ​​of vehicle intelligent body optimization variables at a preset time;

[0140] S342, solving several optimization variable sub-problems and slack variable sub-problems to obtain residuals, and comparing the residuals with a preset threshold; if the residuals are less than or equal to the preset threshold, the iteration is stopped to obtain the dispatchable amount of vehicle agents in different areas; if the residuals are greater than the preset threshold, step S343 is executed;

[0141] S343, calculating the residual radius, updating the residual based on the residual radius to accelerate the convergence of the conjugate gradient algorithm, and generating the dispatchable amount of vehicle intelligent bodies in different areas.

[0142] It should be noted that if Figure 2 As shown in the figure, an improved conjugate gradient method is introduced into the electric vehicle dispatchable quantity solution model, and the multiplier alternating direction method and sequential quadratic programming method are combined to solve different types of electric vehicle dispatchable quantities, including:

[0143] A dispatchable solution model for electric vehicles is constructed, and a slack variable z is introduced to convert inequality constraints into equality constraints. According to the theory of sequential quadratic programming, the original problem is transformed into a series of simple sub-problems that converge to the original problem quadratically. Then, the Lagrangian function is decomposed into several optimization variable QP sub-problems and a slack variable QP sub-problem using the alternating direction multiplier method. At the same time, in order to ensure that the sub-problem group always has a solution, the adaptive conjugate gradient method is introduced in the QP sub-problem, and the convergence of the algorithm is accelerated by calculating the step size and updating the residual.

[0144] The multiplier alternating direction method and sequential quadratic programming method design solution algorithm includes:

[0145] Introducing the slack variable z converts the inequality constraint into an equality constraint:

[0146]

[0147] In the formula, f s , g s 、h s are the objective function, equality constraint and inequality constraint of region s respectively; x s is the optimization variable in region s.

[0148] According to the theory of sequential quadratic programming, the original problem is transformed into solving a series of simple sub-problems that converge quadratically to the original problem. The complete Lagrangian function expression is:

[0149]

[0150] In the formula, Respectively represent the objective function, constraints and The gradient at is the optimization variable in the corresponding region s for the kth iteration; d k H is the k-th iteration descent direction matrix; k is the Hessian matrix of the k-th iteration objective function for the variable, which is in the following form:

[0151]

[0152] The idea of ​​alternating direction multiplier method is used to decompose equation (5) into several optimization variable QP subproblems and one slack variable QP subproblem. At the same time, in order to ensure that the subproblem group always has a solution, the adaptive conjugate gradient method is introduced in the QP subproblem. The convergence of the algorithm is accelerated by calculating the step size and updating the residual.

[0153] Optimize the variable QP subproblem:

[0154]

[0155] Slack variable QP subproblem:

[0156]

[0157] In the formula, f s , g s 、h s denote the objective function, equality constraint and inequality constraint of region s respectively; x s represents the optimization variable in region s; They represent the objective function of region s in x s The gradient at and the inequality constraints of region s are in The gradient at represents the optimization variable in the corresponding region s of the kth iteration; represents the k-th iteration descent direction matrix of region s; Both represent the Lagrange multipliers of the kth iteration; represents the Hessian matrix of the k-th iteration objective function of region s for variables; β1 and β2 both represent penalty factors; β k represents the penalty factor for the kth iteration; d k represents the k-th iteration descent direction matrix; r k+1 represents the residual of the k+1th iteration, z k represents the slack variable for the kth iteration; g z represents the objective function of region s after adding slack variables; if r k+1 If it is small enough, the iteration stops; otherwise, the direction continues to be updated. The specific calculation steps include:

[0158] Step 1: Read the electric vehicle data at time t.

[0159] Step 2: Set the initial values ​​of the parameters and optimization variables of each type of electric vehicle.

[0160] Step 3: Solve the subproblems of each region and the slack variable subproblem in parallel to obtain the residual r k+1 , if r k+1 ≤ε, stop the iteration, ε is the precision, and get the dispatchable quantity of different types of electric vehicles, otherwise go to step 4.

[0161] Step 4: Calculate r k+1 And update the direction d k+1 ,renew z k+1 Return to step 4.

[0162] S4. Use the dispatchable amount of the vehicle agent as the input of the neural network model, iteratively train the neural network model, and output the optimal charging planning strategy to achieve the optimal scheduling of the vehicle agent cluster.

[0163] Among them, the dispatchable amount of the vehicle agent is used as the input of the neural network model, the neural network model is iteratively trained, and the optimal charging planning strategy is output to achieve the optimal scheduling of the vehicle agent cluster, including:

[0164] S41. Establish a neural network model based on the policy network and value network in the reinforcement learning algorithm, and use the dispatchable amount of the vehicle intelligent body as the input of the neural network model;

[0165] S42, iteratively training the neural network model to obtain an initial state and an initial action, superimposing noise on the initial action to obtain a random action, and interacting the random action with the environment to obtain a cumulative reward;

[0166] S43. Store the samples in the experience memory bank, determine whether the capacity of the experience memory bank meets the sampling requirements, and update the neural network model parameters, status and accumulated rewards based on the determination results.

[0167] Among them, the samples are stored in the experience memory bank, whether the capacity of the experience memory bank meets the sampling requirements is judged, and the neural network model parameters, status and accumulated rewards are updated based on the judgment results, including:

[0168] S431, configuring an experience memory for the vehicle agent and storing samples in the experience memory, and each vehicle agent compares the instant reward value of the experience obtained with the average reward value of all experiences in the current experience memory;

[0169] S432, if the experience instant reward value is greater than the experience average reward value, the corresponding sample is classified into a high-importance experience memory bank, otherwise, the corresponding sample is classified into a low-importance experience memory bank;

[0170] S433. Determine whether the capacity of the experience memory bank after sample scheduling meets the sampling requirements. If the experience memory bank is full, update the neural network model parameters, status and accumulated rewards.

[0171] S44, determine whether the iterative training is finished. If the iterative training is finished, output the cumulative reward and action quality of the current round and then execute step S45. Otherwise, repeat steps S42-S44;

[0172] S45. Evaluate the cumulative rewards and action quality of the current round, generate an optimal charging planning strategy based on the evaluation results, and execute optimal scheduling of the vehicle intelligent body cluster based on the optimal charging planning strategy.

[0173] It should be noted that based on the reinforcement learning method, combined with electricity price data, electric vehicle demand and other parameters for training, an improved reinforcement learning algorithm is proposed to solve the total benefit and achieve the optimal scheduling of electric vehicle clusters. Among them, the improved reinforcement learning algorithm solves the total benefit including:

[0174] (1) Establishing the Actor network (strategy network) and Critic network (value network) system:

[0175] The Critic network technology aims to minimize the difference between the target value and the estimated value of the temporal difference (TD) and establish a low-variance value function for the Actor to better evaluate and improve the dynamic strategy it generates. The Actor network is a normal distribution function based on the state output, and the mean and variance values ​​of the dynamic dimension are obtained by random sampling according to probability.

[0176] (2) Improved proximal strategy optimization algorithm with gradient random perturbation:

[0177] During the training process of the environment, the experience replay mechanism is used to process the transfer samples online. At each time step t, the transfer samples obtained by the interaction between the agent and the environment are stored in the memory unit. During training, small batches of transfer samples are randomly extracted from the memory unit. The correlation between samples is reduced through random sampling, which improves the stability of the algorithm.

[0178] In order to improve the performance of the algorithm and avoid the algorithm falling into the local optimal solution, the action of the intelligent agent By adding action noise γ to get random actions, we can get the global optimal solution:

[0179]

[0180] Where υ is the motion noise attenuation factor, and its value range is [0, 1]. When the training just starts, a t Its value is relatively large, which plays a role in fully exploring the environment. As the training progresses, its value gradually decays, reducing the error caused by noise in the action.

[0181] The algorithm uses the Adam optimizer, which is a gradient descent algorithm with adaptive learning rate. The Adam optimizer adaptively adjusts the learning rate according to the direction and size of the gradient. The update calculation of Adam can be expressed as:

[0182]

[0183] Where: m t , are respectively the exponentially weighted moving average and the moving average of the gradient 1st moment, used to estimate the mean of the gradient; v t , They are the exponentially weighted moving average and moving average of the second-order moment of the gradient, respectively; is the gradient of the current step (the derivative of the loss function with respect to the weight); are the hyperparameters of Adam, which control the attenuation degree of the first-order moment and the second-order moment respectively; α is the learning rate, which controls the step size of weight update; ε is a small constant used for numerical stability to prevent zero division errors.

[0184] In order to improve the performance of the Adam optimizer, the present invention proposes to adopt an optimization algorithm that takes into account random perturbations of gradients, that is, adding a random noise vector after calculating the gradient during forward propagation and backward propagation of the model.

[0185]

[0186] In the formula, is the current step gradient after adding noise; n t is a random noise vector with a Gaussian distribution of mean 0 and variance 0.05, and the dimension is the same as the current step gradient.

[0187] The dispatching process of electric vehicle dispatching model based on reinforcement learning is as follows Figure 3 As shown, specifically including:

[0188] Step 1: Initialize the parameters;

[0189] Step 2: Read the distributed photovoltaic power generation P on a certain day in the training set t pv 、Load demand power P t load , real-time electricity prices Number of electric vehicles that can participate in dispatchingt And total power

[0190] Step 3: Interact with the environment to obtain state S t , action a is obtained from the policy network t , superimpose noise on the action to get random action By Action Interacting with the environment gets reward g t and the next moment state S t ;

[0191] Step 4: The experience replay mechanism stores samples in the experience memory bank; two experience replay pools are configured for each agent in the multi-agent environment: one for storing experiences with high importance, and the other for storing experiences with low importance. During the experience storage process, the agent will compare the immediate reward value of the experience it has gained with the average reward value of all experiences in the current experience replay pool. If the reward value of the experience exceeds the average, it will be included in the high-importance experience pool; otherwise, it will be included in the low-importance experience pool. During the model training phase, a weighted sampling strategy is adopted to extract samples from the high-importance experience pool at a higher ratio, and from the low-importance experience pool at a lower ratio;

[0192] Step 5: Determine whether the memory bank is full. If it is full, extract batch data from it to update the network parameters, and then update the status and accumulated rewards;

[0193] Step 6: Update the time and determine whether the current training round has ended. If it has ended, output the accumulated reward of the current round and repeat steps 2 to 6. Otherwise, repeat steps 3 to 6.

[0194] Step 7: The aggregator evaluates the pros and cons of the output action. If the simulation experiment results show that the scheduling strategy model can effectively meet the charging needs of different users and achieve coordinated and orderly charging among electric vehicles, and reach above the preset profit threshold, a scheduling decision is made.

[0195] It should be noted that the present invention combines the charging characteristics of three different electric vehicles in establishing three intelligent entities, namely private cars, commercial vehicles and taxis, and the optimization objective function, and tries to arrange private cars and commercial vehicles in the time period when taxis have completed charging, which can reduce the overall peak power consumption of the power grid. In order to ensure that various types of electric vehicles reach the expected power before the end, the electric energy boundary method is introduced. During the entire charging process, as long as the electric energy curve of the electric vehicle is kept within the upper and lower bounds of the electric energy curve, it can be guaranteed that it can reach the expected power before the end. At the same time, in order to make different types of electric vehicle users actively respond to the scheduling strategy, the state-action space and the reward function are further designed when setting the Markov game environment, and the weight coefficients of different optimization objectives are considered.

[0196] It should be noted that the regional electric vehicle demand algorithm in the present invention combines the multiplier alternating direction method and the sequential quadratic programming method, and by establishing a multi-region electric vehicle dispatchable quantity solution model and introducing slack variables, it is possible to convert inequality constraints into equality constraints; according to the sequential quadratic programming theory, the original problem is converted into solving a series of simple quadratic sub-problems that converge to the original problem, and the idea of ​​the alternating direction multiplier method is used to decompose the equation into an optimization variable QP sub-problem and a slack variable QP sub-problem. At the same time, in order to ensure that the sub-problem group always has a solution, the trust region method is introduced in the QP sub-problem, and the convergence of the algorithm is accelerated by expanding or reducing the trust region size, thereby obtaining the number of electric vehicles required in different regions.

[0197] It should be noted that the present invention can limit the scope of strategy changes through an improved reinforcement learning algorithm and a proximal strategy optimization method, avoid excessive strategy updates, and prevent the algorithm from falling into an unstable state, thereby improving robustness. In addition, an experience replay mechanism is used in the training process of the environment, and transfer samples are obtained through online processing. The transfer samples obtained by the interaction between the intelligent agent and the environment are stored in the memory unit, which reduces the correlation between samples and improves the stability of the algorithm. In order to improve the performance of the Adam optimizer, the present invention proposes an improved proximal strategy optimization algorithm using gradient random perturbation, and a random noise vector is added after the gradient is calculated during the forward propagation and back propagation of the model.

[0198] In summary, with the help of the above technical solutions of the present invention, the present invention can limit the scope of strategy changes through an improved reinforcement learning algorithm and a proximal strategy optimization algorithm, avoid excessive strategy updates, and avoid the algorithm from falling into an unstable state, thereby improving robustness. In the training process of the environment, an experience replay mechanism is adopted to obtain transfer samples through online processing, and the transfer samples obtained by the interaction between the intelligent agent and the environment are stored in the memory unit, which reduces the correlation between the samples and thereby improves the stability of the algorithm. The optimal scheduling method for electric vehicles to participate in the energy storage market provided by the present invention can turn urban electric vehicles into mobile energy storage modules, which can not only reduce the impact of large-scale access of electric vehicles to the power grid on the stability of the power grid, but also create certain economic benefits for electric vehicle users. Due to the strong randomness of the charging and discharging of electric vehicles, traditional methods are difficult to accurately model and optimize, and the deep reinforcement learning provided by the present invention can effectively handle dynamic and complex environments, adapt to real-time changes, and thereby improve the system scheduling performance. The present invention solves the problem that large-scale access of electric vehicles to the power grid will cause impact on the power system by establishing private cars, commercial vehicles, and taxi intelligent agents and using reinforcement learning algorithms to optimize the operation strategies of electric vehicles, so that aggregators can determine the current charging and discharging amplitude of electric vehicles according to key information in the system, thereby achieving optimal scheduling.

[0199] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning, characterized in that: The optimal dispatching method for electric vehicles to participate in the energy storage market includes: S1. Based on the expected charging power characteristics of electric vehicles, construct a vehicle agent and the optimization goal of the vehicle agent; S2. Establish a corresponding Markov environment based on the optimization goal of the vehicle agent. In the Markov environment, dynamically execute the actions of each vehicle agent and pre-introduce a reward function with an optimization goal weight coefficient to coordinate the charging sequence between the vehicle agents. S3. Based on the adaptive conjugate gradient method, a model for solving the dispatchable amount of vehicle intelligent bodies is constructed, and the dispatchable amount of vehicle intelligent bodies in different areas is solved by combining the multiplier alternating direction algorithm and the sequential quadratic programming algorithm; S4. Use the dispatchable amount of the vehicle agent as the input of the neural network model, iteratively train the neural network model, and output the optimal charging planning strategy to achieve the optimal scheduling of the vehicle agent cluster.

2. According to claim 1, an optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning is characterized in that: The vehicle intelligent body and the optimization objectives of the vehicle intelligent body based on the expected charging power characteristics of the electric vehicle include: S11, obtaining expected charging power characteristics of different types of electric vehicles, where the types of electric vehicles include private cars, commercial vehicles and taxis; S12, establishing the optimization objective function and vehicle intelligent body of private cars and commercial vehicles respectively with minimizing charging cost and reducing the overall peak power consumption of microgrid as the optimization goal; establishing the optimization objective function and vehicle intelligent body of taxi with minimizing charging time as the optimization goal; S13. Based on the expected power characteristics of charging, the power curve of the vehicle intelligent body during the charging process is maintained within the upper and lower bounds of the power curve, so that the vehicle intelligent body reaches the expected power before the charging is completed.

3. The optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 2 is characterized in that: The method of maintaining the electric energy curve of the vehicle intelligent body during the charging process within the electric energy upper and lower bound curves based on the expected electric energy characteristics of charging, so that the vehicle intelligent body reaches the expected electric energy before the charging is completed, includes: S131, obtaining the aggregator power of the private car before charging begins, and generating upper and lower bound curves of power during the charging process of the private car based on the charging and discharging actions of the private car; S132, generating upper and lower bound curves of electric energy of the commercial vehicle and the taxi based on the charging actions of the commercial vehicle and the taxi; S133. Keep the power curves of the private car, commercial vehicle and taxi during the charging process within the corresponding upper and lower bounds of the power curve to ensure that the vehicle intelligent body reaches the expected power before the charging is completed.

4. The optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 3 is characterized in that: The method of establishing a corresponding Markov environment based on the optimization target of the vehicle agent, dynamically executing the actions of each vehicle agent and pre-introducing a reward function with an optimization target weight coefficient in the Markov environment to coordinate the charging sequence between the vehicle agents includes: S21. Based on the optimization goal of the vehicle agent, a corresponding Markov game environment is established, and in the Markov game environment, the vehicle agent action is dynamically executed in combination with the environment state; S22, based on the optimization goal of the vehicle intelligent body, establishing the state space and action space of the vehicle intelligent body, and reflecting the decision factors of the charging sequence of the vehicle intelligent body based on the state space and action space; S23. Introduce an optimization target weight coefficient to establish a reward function for the vehicle agent, and coordinate the charging sequence between the vehicle agents based on the reward function.

5. The optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 4 is characterized in that: The optimization goal based on the vehicle intelligent body, establishing the state space and action space of the vehicle intelligent body includes: S221. Based on the optimization objectives of private cars and commercial vehicles, a state space of private cars and commercial vehicles is established, and the expression of the state space is: S222. Based on the optimization target of the taxi, a state space of the taxi is established, and the expression of the state space is: S223, the charging and discharging power of private cars, commercial vehicles and taxi aggregators is used as the action space, and the expression of the action space is: a t ={P t }; In the formula, t a t1 and t2 represent the arrival time of the first vehicle agent to arrive at the charging pile and the departure time of the last vehicle agent to leave the charging pile under the current aggregator; N t represents the number of vehicle agents under the aggregator at time t; e t 、e n They represent the power of the aggregator and the expected power of the aggregator at time t respectively; They represent the upper and lower bounds of the electrical energy of the aggregator at time t respectively; represents the net load of the upper microgrid at time t; price t represents the real-time electricity price of the main power grid at time t; a t represents the action space at time t; S t represents the state space at time t; P t is the charge and discharge power at time t.

6. The optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 5 is characterized in that: The adaptive conjugate gradient method is used to construct a schedulable quantity solution model for vehicle agents, and combined with the multiplier alternating direction algorithm and the sequential quadratic programming algorithm, the schedulable quantities of vehicle agents in different regions are solved, including: S31, constructing a schedulable quantity solution model for the vehicle intelligent body, and introducing slack variables to convert inequality constraints based on the schedulable quantity solution model into equality constraints; S32. Solve the dispatchable quantity solution model based on sequential quadratic programming theory to obtain a complete Lagrangian function; S33, decomposing the complete Lagrangian function into a number of optimization variable subproblems and relaxation variable subproblems; S34. Introduce an adaptive conjugate gradient algorithm into the sub-problem, and update the residual to accelerate the conjugate gradient algorithm to generate the dispatchable quantities of vehicle agents in different areas after convergence.

7. The optimal dispatching method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 6 is characterized in that: The adaptive conjugate gradient algorithm is introduced into the sub-problem, and the schedulable quantities of vehicle intelligent bodies in different regions are generated after the residual acceleration conjugate gradient algorithm converges, including: S341, obtaining vehicle intelligent body data and initial values ​​of vehicle intelligent body optimization variables at a preset time; S342, solving several optimization variable sub-problems and slack variable sub-problems to obtain residuals, and comparing the residuals with a preset threshold; if the residuals are less than or equal to the preset threshold, the iteration is stopped to obtain the dispatchable amount of vehicle agents in different areas; if the residuals are greater than the preset threshold, step S343 is executed; S343, calculating the residual radius, updating the residual based on the residual radius to accelerate the convergence of the conjugate gradient algorithm, and generating the dispatchable amount of vehicle intelligent bodies in different areas.

8. The optimal dispatching method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 7 is characterized in that: The method of using the dispatchable amount of the vehicle agent as the input of the neural network model, iteratively training the neural network model, and outputting the optimal charging planning strategy to achieve the optimal dispatch of the vehicle agent cluster includes: S41. Establish a neural network model based on the policy network and value network in the reinforcement learning algorithm, and use the dispatchable amount of the vehicle intelligent body as the input of the neural network model; S42, iteratively training the neural network model to obtain an initial state and an initial action, superimposing noise on the initial action to obtain a random action, and interacting the random action with the environment to obtain a cumulative reward; S43, storing the sample in the experience memory bank, determining whether the capacity of the experience memory bank meets the sampling requirements, and updating the neural network model parameters, state and accumulated rewards based on the determination result; S44, determine whether the iterative training is finished. If the iterative training is finished, output the cumulative reward and action quality of the current round and then execute step S45. Otherwise, repeat steps S42-S44; S45. Evaluate the cumulative rewards and action quality of the current round, generate an optimal charging planning strategy based on the evaluation results, and execute optimal scheduling of the vehicle intelligent body cluster based on the optimal charging planning strategy.

9. The optimal scheduling method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 8 is characterized in that: The storing of samples into the experience memory bank, judging whether the capacity of the experience memory bank meets the sampling requirements, and updating the neural network model parameters, status and accumulated rewards based on the judgment results include: S431, configuring an experience memory for the vehicle agent and storing samples in the experience memory, and each vehicle agent compares the instant reward value of the experience obtained with the average reward value of all experiences in the current experience memory; S432, if the experience instant reward value is greater than the experience average reward value, the corresponding sample is classified into a high-importance experience memory bank, otherwise, the corresponding sample is classified into a low-importance experience memory bank; S433. Determine whether the capacity of the experience memory bank after sample scheduling meets the sampling requirements. If the experience memory bank is full, update the neural network model parameters, status and accumulated rewards.

10. The optimal dispatching method for electric vehicles participating in the energy storage market based on reinforcement learning according to claim 9 is characterized in that: The expression of the optimization variable subproblem is: The expression of the slack variable subproblem is: In the formula, f s , g s 、h s denote the objective function, equality constraint and inequality constraint of region s respectively; x s represents the optimization variable in region s; They represent the objective function of region s in x s The gradient at and the inequality constraints of region s are in The gradient at represents the optimization variable in the corresponding region s of the kth iteration; represents the k-th iteration descent direction matrix of region s; Both represent the Lagrange multipliers of the kth iteration; represents the Hessian matrix of the k-th iteration objective function of region s for variables; β1 and β2 both represent penalty factors; β k represents the penalty factor for the kth iteration; d k represents the k-th iteration descent direction matrix; r k+1 represents the residual of the k+1th iteration, z k represents the slack variable for the kth iteration; g z represents the objective function of region s after adding slack variables.

Images