Electric vehicle cluster scheduling strategy and system based on deep reinforcement learning

Abstract

The invention relates to the technical field of electric vehicle cluster scheduling, and discloses an electric vehicle cluster scheduling strategy and system based on deep reinforcement learning, and the method comprises the steps: determining an electric vehicle cluster scheduling range based on the charging willingness of a user; converting a cluster scheduling problem into a Markov process to obtain a conversion result; and designing a reward function for each charging station, and solving through a deep enhancement algorithm to obtain a final electric vehicle cluster scheduling strategy. According to the method, the load peak-valley difference and peak regulation cost of the urban power grid system are converted into an optimization problem by using deep reinforcement learning, so that the load pressure of the urban power grid system can be reduced, and normal and safe operation of a power distribution line is ensured; and the scheduling efficiency can be improved by considering the number of the electric vehicles needing to be scheduled instead of charging scheduling of all the electric vehicles in a city range.

Description

Technical Field

[0001] The present invention relates to the technical field of electric vehicle cluster scheduling, and in particular to an electric vehicle cluster scheduling strategy and system based on deep reinforcement learning. Background Art

[0002] As a means of transportation with less pollution and high energy efficiency, electric vehicles are gradually replacing traditional fuel vehicles. With the increase in the number of electric vehicles and their increasing share in the transportation network, the charging load of large-scale electric vehicles will cause uneven temporal and spatial distribution of urban power grids, and will further aggravate the peak load, leading to a series of problems such as voltage drop, affecting the normal operation of the distribution network, thus restricting the development of electric vehicles.

[0003] Electric vehicles have the characteristics of flexible electricity consumption and mobile electricity consumption locations. Therefore, reasonable scheduling of electric vehicle charging behavior can alleviate the pressure of electricity consumption during peak hours, and scheduling in a cluster manner can effectively reduce the possibility of dimensional disaster when scheduling electric vehicles. Therefore, the research on optimization of electric vehicle charging and discharging scheduling has become a hot topic in the development of future power grid systems. However, it is difficult to model the scheduling problem of electric vehicle clusters in actual scenarios, so there is no need to establish a model. The advantages of data-driven deep reinforcement methods in solving the scheduling problem of electric vehicle clusters are gradually emerging. For the peak-to-valley difference and peak-shaving cost of the urban power grid system, deep reinforcement learning is used to transform it into an optimization problem, which can reduce the load pressure of the urban power grid system and ensure the normal and safe operation of the distribution lines. In addition, considering the number of electric vehicles to be scheduled rather than all electric vehicle charging scheduling in the city can improve scheduling efficiency. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides an electric vehicle cluster scheduling strategy based on deep reinforcement learning to solve the problem of difficulty in modeling the electric vehicle cluster scheduling problem.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an electric vehicle cluster scheduling strategy based on deep reinforcement learning, including: determining an electric vehicle cluster scheduling range based on user charging willingness;

[0008] Based on the electric vehicle cluster scheduling scope, the cluster scheduling problem is converted into a Markov process to obtain a conversion result;

[0009] Based on the conversion results, a reward function is designed for each charging station, which is solved by a deep reinforcement algorithm to obtain the final electric vehicle cluster scheduling strategy.

[0010] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, the determination of the electric vehicle cluster scheduling range includes:

[0011] According to the user's charging willingness and the charging power of the electric vehicle cluster at the initial moment, the dispatchable range corresponding to the electric vehicle cluster participating in the incentive demand is obtained, which is expressed as:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] in, represents the original charging power of the electric vehicle cluster at time t, λ EV represents the user's willingness to charge at time t, and They represent the upper and lower limits of the dispatchable range of the electric vehicle group at time t, and They represent the minimum and maximum values ​​of the aggregated charging power of electric vehicles at time t, respectively, and p ch Indicates the rated charging power of each electric vehicle.

[0018] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, wherein: the transformation of the cluster scheduling problem into a Markov process includes:

[0019] Defined as a tuple, represented as:

[0020] <J,S,A 1 ,...,A m ,O 1 ,...,O m ,r 1 ,...,r m ,p,γ>

[0021] Where J represents a group of m charging stations participating in the scheduling, and S represents a group of system states s t ∈S,A m Indicates the joint action of the charging station, O m represents the joint observation of charging stations, r m represents the expected reward of state-action, function p represents the state transition probability, and γ represents the discount factor.

[0022] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, the final electric vehicle cluster scheduling strategy includes:

[0023] Establish an IPPO agent model, including the IPPO agent state space, action space and reward function, and establish the IPPO agent strategy objective function;

[0024] By comparing the difference in objective functions between the new and old strategies, approximate processing is used to process the action distribution through important sampling. The new strategy is estimated and optimized based on the data sampled by the old strategy, and trained through the IPPO algorithm until the function converges to achieve the electric vehicle cluster scheduling goal.

[0025] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, establishing the IPPO agent model includes:

[0026] With the goal of grid voltage stability and even power distribution, the state space of IPPO agent is established. The state space variable is selected based on the actual voltage effective value U of the i-th charging station. i , actual active power P i , actual reactive power Q i and the dispatch distance of electric vehicles As the observed quantity of charging station;

[0027] Establish the IPPO agent action space. The action a taken by charging station i includes guiding the car to charge and prohibiting the car from charging, indicating whether the electric car within the dispatching range is guided to its own position;

[0028] The IPPO agent reward function is established, that is, the reward function obtained by the charging station is the sum of the voltage reward function and the distance reward function of calling the electric vehicle.

[0029] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, wherein: the establishment of the IPPO agent strategy objective function includes:

[0030] Assume that the current strategy is π θ , parameter is θ, define parameter θ', so that J(θ')>J(θ), the strategy πθ The optimization goal J(θ) is in the new strategy π θ' The expected form is:

[0031]

[0032] Among them, a represents action, Indicates the charging station status s t+1 Strategy π θ The expected reward under Represents the agent state s t Strategy π θ The expected reward under t ,a) means in state s t The immediate reward obtained by taking action a, In the strategy π θ' The expected reward under t represents the discount rate after time t.

[0033] As a preferred solution of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in the present invention, the electric vehicle cluster scheduling goal is achieved by:

[0034] The difference between the new and old strategy objective functions is expressed as:

[0035]

[0036] in, represents the advantage function, Represents the expected reward difference of the agent under different strategies in the same state;

[0037] Based on the old strategy π θ Sampled data, estimate and optimize the new strategy π θ' , solved by PPO-Clip, expressed as:

[0038]

[0039] Among them, clip(x,l,r):=max(min(x,r),l), which means limiting x to [l,r]. ε is a hyperparameter, indicating the range of truncation, and π θ represents the strategy under parameter θ, Represents the parameter θ after k steps of optimization k strategy.

[0040] In a second aspect, the present invention provides an electric vehicle cluster dispatching system based on deep reinforcement learning, comprising:

[0041] The scheduling analysis module is used to determine the scheduling scope of electric vehicle clusters based on user charging willingness;

[0042] A conversion module, used for converting the cluster scheduling problem into a Markov process based on the electric vehicle cluster scheduling range to obtain a conversion result;

[0043] The scheduling output module is used to design a reward function for each charging station based on the conversion result, and solve it through a deep reinforcement algorithm to obtain the final electric vehicle cluster scheduling strategy.

[0044] In a third aspect, the present invention provides an electronic device, comprising:

[0045] Memory and processor;

[0046] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of implementing the electric vehicle cluster scheduling strategy based on deep reinforcement learning are implemented.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the electric vehicle cluster scheduling strategy based on deep reinforcement learning.

[0048] Compared with the prior art, the present invention has the following beneficial effects: the present invention determines the range of dispatchable electric vehicles by evaluating the user's charging willingness, takes into account the electricity safety of charging stations and the dispatching cost, and finally uses deep reinforcement learning to obtain the optimal strategy for dispatching electric vehicle clusters. It uses deep reinforcement learning to transform the peak-to-valley difference and peak-shaving cost of the urban power grid system into an optimization problem, which can reduce the load pressure of the urban power grid system and ensure the normal and safe operation of the distribution lines. In addition, the dispatching efficiency can be improved by considering the number of electric vehicles to be dispatched rather than dispatching all the electric vehicles within the city. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0050] Figure 1 A schematic diagram of the overall process of an electric vehicle cluster scheduling strategy based on deep reinforcement learning according to an embodiment of the present invention;

[0051] Figure 2An overall operation block diagram of an electric vehicle cluster scheduling strategy based on deep reinforcement learning according to an embodiment of the present invention;

[0052] Figure 3 A graph showing changes in the number of dispatchable electric vehicles according to an electric vehicle cluster dispatch strategy based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Figure 4 This is the IPPO algorithm training result of the electric vehicle cluster scheduling strategy based on deep reinforcement learning described in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0057] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0058] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0059] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] Example 1

[0061] Reference Figure 1-Figure 2 , is an embodiment of the present invention, which provides an electric vehicle cluster scheduling strategy based on deep reinforcement learning, such as Figure 1 , 2 As shown, including:

[0062] S100, determining a dispatch range of an electric vehicle cluster based on a user's charging intention.

[0063] For example, the dispatchable range of electric vehicle clusters is crucial for effectively formulating demand response strategies. Assume that there are N EV electric vehicles, let the average SOC of the vehicle group at time t be S(t), then the SOC of the electric vehicle group at time t+1 is expressed as:

[0064]

[0065] Among them, η ch is the charging efficiency of electric vehicles, P EV (t) is the charging power of the electric vehicle cluster at time t, K(t) is the stopping probability of the vehicle at time t, M unit is the mechanical energy consumed by the car per kilometer, η em is the efficiency of converting electrical energy into mechanical energy, v(t) is the speed of the electric vehicle at time t, B av is the average battery capacity of the electric vehicle group, Δt is the time interval between time t and time t+1, S max and S min The upper and lower limits of the electric vehicle battery SOC.

[0066] If the user's willingness to charge is not considered, and the electric vehicle cluster is arranged to charge the vehicle based on the power purchase price of the power grid, the charging cost is minimized, which can be expressed as:

[0067]

[0068] Among them, F 0 is the charging cost of the electric vehicle cluster, T is the total number of scheduling times, which is 24 hours, and C tou (t) is the price of electricity purchased from the power grid at time t, is the charging power of the electric vehicle cluster at the initial time t.

[0069] S101, determining the user's charging willingness based on the electric vehicle SOC and the grid electricity price;

[0070] Considering the user's willingness to charge from the battery SOC and grid price, it can be expressed as the following function:

[0071]

[0072] Among them, λ EV represents the charging willingness value of electric vehicle cluster users, x 1 represents the average incentive price of the electric vehicle cluster, x 2 represents the battery charge state of the electric vehicle cluster, and a and b are fixed parameters.

[0073] S102, determining a dispatchable electric vehicle range based on the user's charging willingness;

[0074] According to the user's charging willingness and the charging power of the electric vehicle cluster at the initial moment, the dispatchable range corresponding to the electric vehicle cluster participating in the incentive demand is obtained, which is expressed as:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] in, represents the original charging power of the electric vehicle cluster at time t, λ EV represents the user's willingness to charge at time t, and They represent the upper and lower limits of the dispatchable range of the electric vehicle group at time t, and They represent the minimum and maximum values ​​of the aggregated charging power of electric vehicles at time t, respectively, and p ch Indicates the rated charging power of each electric vehicle.

[0081] It should be noted that through the above steps, it is possible to obtain EV When there are electric vehicles, how many of them can participate in the current cluster scheduling behavior.

[0082] S200, based on the electric vehicle cluster scheduling scope, transforms the cluster scheduling problem into a Markov process and obtains the transformation result.

[0083] Preferably, converting the cluster scheduling problem into a Markov process includes:

[0084] Defined as a tuple, represented as:

[0085] <J,S,A 1 ,...,A m ,O 1 ,...,O m ,r 1 ,...,r m ,p,γ>

[0086] Where J represents a group of m charging stations participating in the scheduling, and S represents a group of system states s t ∈S,A m represents the joint action of charging stations, each charging station i∈J chooses action a i ∈A i , the action profile is the joint action of all charging stations: a: = (a 1 ,a 2 ,…,a m ),a∈A=A 1 ×A 2 ×…×A m , O m represents the joint observation of charging stations, o i ∈O i is the local observation of the system state s of charging station i, and the observation profile is the joint observation of all charging stations: o: = (o 1 ,o 2 ,…,o m ),o∈O=O 1 *O 2 *…*O m , r m Represents the expected reward of state-action, function r i :S×A→R, defines the expected reward of a state-action pair, where r i (s t ,a) indicates that charging station i is in state s k The instant reward obtained by taking action a, function p represents the state transition probability, function p: S×A×S→[0,1] is the state transition probability, where p(s t+1 |s t , a) represents the state s after taking joint action a t Transfer to t+1 The transition probability, γ represents the discount coefficient, γ∈[0,1).

[0087] S300, based on the conversion results, designs a reward function for each charging station, solves it through a deep reinforcement algorithm, and obtains the final electric vehicle cluster scheduling strategy.

[0088] Specifically, it is divided into the following steps:

[0089] S301, designing a reward function based on the voltage stability of the charging station and the distance of the electric vehicle;

[0090] S302, setting parameters and training times, solving using the IPPO algorithm, and determining whether convergence occurs;

[0091] S303: If converged, output the strategy reward and scheduling strategy.

[0092] Preferably, the final electric vehicle cluster dispatching strategy includes:

[0093] Establish an IPPO agent model, including the IPPO agent state space, action space and reward function, and establish the IPPO agent strategy objective function;

[0094] By comparing the difference in objective functions between the new and old strategies, approximate processing is used to process the action distribution through important sampling. The new strategy is estimated and optimized based on the data sampled by the old strategy, and trained through the IPPO algorithm until the function converges to achieve the electric vehicle cluster scheduling goal.

[0095] Preferably, establishing the IPPO agent model includes:

[0096] With the goal of grid voltage stability and even power distribution, the state space of IPPO agent is established. The state space variable is selected based on the actual voltage effective value U of the i-th charging station. i , actual active power P i , actual reactive power Q i and the dispatch distance of electric vehicles As the observed quantity of charging station;

[0097] Exemplarily, the state space representation is established as:

[0098]

[0099] in, represents the distance between the kth electric vehicle and the i-th charging station.

[0100] Establish the IPPO agent action space. The action a taken by charging station i includes guiding the car to charge and prohibiting the car from charging, indicating whether the electric car within the dispatch range should be guided to its own position;

[0101] Exemplarily, the action space is established as:

[0102] A i =(a 1 ,a 2 )

[0103] Among them, a 1 Indicates guiding the car to charge, a 2 Indicates that it is prohibited to guide the car to charge.

[0104] Establish the IPPO agent reward function, that is, the reward function obtained by the charging station is the sum of the voltage reward function and the distance reward function of calling the electric vehicle;

[0105] Exemplarily, an IPPO agent reward function is established, and the voltage reward function of each charging station is expressed as:

[0106]

[0107] Where ΔU=U N -U i ,U N is the rated voltage of the charging station, U i is the actual output voltage of the charging station, α 1 , α 2 , α 3 Represents the reward function parameters of the charging station at different output voltages.

[0108] The distance reward function for each charging station to call an electric vehicle is expressed as:

[0109]

[0110] Among them, D max is the maximum dispatch distance allowed by the charging station, D min is the maximum dispatch distance that the charging station can allow with the lowest cost, β 1 , β 2 Represents the reward function parameters for charging stations to call electric vehicles at different distances;

[0111] The reward function obtained by each intelligent agent, i.e., the charging station, is the sum of the voltage reward function and the distance reward function, expressed as:

[0112]

[0113] Preferably, establishing the IPPO agent strategy objective function includes:

[0114] Assume that the current strategy is π θ , parameter is θ, define parameter θ', so that J(θ')>J(θ), strategy π θ The optimization goal J(θ) is in the new strategy πθ' The expected form is:

[0115]

[0116] Among them, a represents action, Indicates the charging station status s t+1 Strategy π θ The expected reward under Represents the agent state s t Strategy π θ The expected reward under t ,a) means in state s t The immediate reward obtained by taking action a, In the strategy π θ' The expected reward under t represents the discount rate considered after time t.

[0117] Preferably, the difference between the new and old strategy objective functions is as follows:

[0118]

[0119]

[0120] in, represents the advantage function, represents the expected reward difference of the agent under different strategies in the same state, so This ensures that the strategy performance increases monotonically.

[0121] By using approximate processing and important sampling to process the action distribution, we can achieve the old strategy π θ The sampled data is used to estimate and optimize the new strategy π θ' , using PPO-Clip to solve, expressed as:

[0122]

[0123] Among them, clip(x,l,r):=max(min(x,r),l), which means limiting x to [l,r]. ε is a hyperparameter, indicating the range of truncation (clip), π θ represents the strategy under parameter θ, Represents the parameter θ after k steps of optimization k strategy.

[0124] if This means that the value of this action is higher than the average, and the maximum solution formula will increase But it will not exceed 1+ε; on the contrary, if Maximizing the solution formula will reduce But it will not exceed 1-ε.

[0125] Use IPPO algorithm for training to obtain the convergence parameter θ k , and then get the scheduling strategy If the function does not converge after the training, increase the number of training steps or adjust the hyperparameter ε until the function converges and the electric vehicle cluster scheduling goal is achieved.

[0126] It should be noted that the present invention first evaluates the user's willingness to charge from two aspects: the SOC of the electric vehicle and the power purchase price of the power grid, and determines the dispatch range of the electric vehicles participating in the cluster dispatch; converts the electric vehicle dispatch problem into a Markov process, designs a reward function for each charging station, and uses the IPPO algorithm for learning, thereby obtaining the electric vehicle cluster dispatch strategy. This method has the following advantages and effects:

[0127] (1) Evaluate user willingness based on the proposed aspects and determine the range of dispatchable electric vehicles, which is closer to practical application.

[0128] (2) The use of deep reinforcement learning algorithms to realize electric vehicle cluster scheduling has better real-time performance than using traditional control algorithms, avoids the tedious modeling of traditional methods, and has better adaptability.

[0129] (3) The IPPO algorithm is a completely decentralized algorithm that enables each agent to learn independently in its own environment. It has good scalability with the increase in the number of charging stations and will not be unable to adapt due to the curse of dimensionality.

[0130] The present invention determines the range of dispatchable electric vehicles by evaluating users' charging willingness, takes into account the electricity safety of charging stations and the dispatching cost, and finally uses deep reinforcement learning to obtain the optimal strategy for electric vehicle cluster dispatch. It uses deep reinforcement learning to transform the peak-to-valley difference and peak-shaving cost of the urban power grid system into an optimization problem, which can reduce the load pressure of the urban power grid system and ensure the normal and safe operation of the distribution lines. In addition, considering the number of electric vehicles to be dispatched rather than scheduling the charging of all electric vehicles within the city can improve the dispatching efficiency.

[0131] The above is a schematic scheme of an electric vehicle cluster scheduling strategy based on deep reinforcement learning in this embodiment. It should be noted that the technical scheme of the electric vehicle cluster scheduling system based on deep reinforcement learning and the technical scheme of the electric vehicle cluster scheduling strategy based on deep reinforcement learning belong to the same concept. The details of the technical scheme of the electric vehicle cluster scheduling system based on deep reinforcement learning in this embodiment that are not described in detail can all be referred to the description of the technical scheme of the electric vehicle cluster scheduling strategy based on deep reinforcement learning.

[0132] The electric vehicle cluster dispatching system based on deep reinforcement learning in this embodiment includes:

[0133] The scheduling analysis module is used to determine the scheduling scope of electric vehicle clusters based on user charging willingness;

[0134] A conversion module, used to convert the cluster scheduling problem into a Markov process based on the electric vehicle cluster scheduling range, and obtain a conversion result;

[0135] The scheduling output module is used to design a reward function for each charging station based on the conversion results, and solve it through a deep reinforcement algorithm to obtain the final electric vehicle cluster scheduling strategy.

[0136] This embodiment also provides an electronic device, which is applicable to the case of electric vehicle cluster scheduling based on deep reinforcement learning, including:

[0137] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the electric vehicle cluster scheduling strategy based on deep reinforcement learning as proposed in the above embodiment.

[0138] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the electric vehicle cluster scheduling strategy based on deep reinforcement learning as proposed in the above embodiment is implemented.

[0139] The storage medium proposed in this embodiment and the electric vehicle cluster scheduling strategy based on deep reinforcement learning proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0140] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0141] Example 2

[0142] Refer to Table 1 and Figure 3-Figure 4 , which is an embodiment of the present invention, provides an electric vehicle cluster scheduling strategy based on deep reinforcement learning. In order to verify its beneficial effects, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0143] In the simulation, the number of vehicles N involved in the dispatch is max =60, the number of charging stations is m=3, the initial SOC of the electric vehicle battery is sampled and set by the Monte Carlo method, and the electricity price is set as shown in Table 1.

[0144] Table 1: Electricity price setting table

[0145] Electricity Prices Start time End time 0.3 yuan / kWh 00：00 08：00 0.5 yuan / kWh 08：00 17：00 0.8 yuan / kWh 17：00 22：00 0.5 yuan / kWh 22：00 24：00

[0146] Determine the dispatch range of electric vehicle clusters and obtain Figure 3 The change graph of the number of dispatchable electric vehicles shown in Figure 1 is solved by designing a reward function for each charging station and using a deep reinforcement algorithm to obtain the following: Figure 4 From the training results of the IPPO algorithm, it can be seen that it can reduce the load pressure of the urban power grid system, ensure the normal and safe operation of the distribution lines, and improve the dispatching efficiency.

[0147] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An electric vehicle cluster scheduling strategy based on deep reinforcement learning, characterized in that: include: Determine the dispatch scope of electric vehicle clusters based on user charging willingness; Based on the electric vehicle cluster scheduling scope, the cluster scheduling problem is converted into a Markov process to obtain a conversion result; Based on the conversion results, a reward function is designed for each charging station, which is solved by a deep reinforcement algorithm to obtain the final electric vehicle cluster scheduling strategy.

2. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as claimed in claim 1, characterized in that: Determining the electric vehicle cluster dispatching scope includes: According to the user's charging willingness and the charging power of the electric vehicle cluster at the initial moment, the dispatchable range corresponding to the electric vehicle cluster participating in the incentive demand is obtained, which is expressed as: in, represents the original charging power of the electric vehicle cluster at time t, λ EV represents the user's willingness to charge at time t, and They represent the upper and lower limits of the dispatchable range of the electric vehicle group at time t, and They represent the minimum and maximum values ​​of the aggregated charging power of electric vehicles at time t, respectively, and p ch Indicates the rated charging power of each electric vehicle.

3. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as described in claim 1 or 2, characterized in that: The transformation of the cluster scheduling problem into a Markov process includes: Defined as a tuple, represented as: <J,S,A1,...,A m ,O1,...,O m ,r1,...,r m ,p,γ> Where J represents a group of m charging stations participating in the scheduling, and S represents a group of system states s t ∈S,A m Indicates the joint action of the charging station, O m represents the joint observation of charging stations, r m represents the expected reward of state-action, function p represents the state transition probability, and γ represents the discount factor.

4. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as claimed in claim 3, characterized in that: The final electric vehicle cluster dispatching strategy includes: Establish an IPPO agent model, including the IPPO agent state space, action space and reward function, and establish the IPPO agent strategy objective function; By comparing the difference in objective functions between the new and old strategies, approximate processing is used to process the action distribution through important sampling. The new strategy is estimated and optimized based on the data sampled by the old strategy, and trained through the IPPO algorithm until the function converges to achieve the electric vehicle cluster scheduling goal.

5. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as claimed in claim 4, characterized in that: Building the IPPO agent model includes: With the goal of grid voltage stability and even power distribution, the state space of IPPO agent is established. The state space variable is selected based on the actual voltage effective value U of the i-th charging station. i , actual active power P i , actual reactive power Q i and the dispatch distance of electric vehicles As the observed quantity of charging station; Establish the IPPO agent action space. The action a taken by charging station i includes guiding the car to charge and prohibiting the car from charging, indicating whether the electric car within the dispatching range is guided to its own position; The IPPO agent reward function is established, that is, the reward function obtained by the charging station is the sum of the voltage reward function and the distance reward function of calling the electric vehicle.

6. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as claimed in claim 5, characterized in that: The establishment of the IPPO agent strategy objective function includes: Assume that the current strategy is π θ , parameter is θ, define parameter θ', so that J(θ')>J(θ), the strategy π θ The optimization goal J(θ) is in the new strategy π θ' The expected form is: Among them, a represents action, Indicates the charging station status s t+1 Strategy π θ The expected reward under Represents the agent state s t Strategy π θ The expected reward under t ,a) means in state s t The immediate reward obtained by taking action a, In the strategy π θ' The expected reward under t Represents the discount rate considering rewards after time t.

7. The electric vehicle cluster scheduling strategy based on deep reinforcement learning as claimed in claim 6, characterized in that: The electric vehicle cluster dispatching objectives include: The difference between the new and old strategy objective functions is expressed as: in, represents the advantage function, Represents the expected reward difference of the agent under different strategies in the same state; Based on the old strategy π θ Sampled data, estimate and optimize the new strategy π θ' , solved by PPO-Clip, expressed as: Among them, clip(x,l,r):=max(min(x,r),l), which means limiting x to [l,r]. ε is a hyperparameter, indicating the range of truncation, and π θ represents the strategy under parameter θ, Represents the parameter θ after k steps of optimization k strategy.

8. An electric vehicle cluster dispatching system based on deep reinforcement learning, characterized in that: include, The scheduling analysis module is used to determine the scheduling scope of electric vehicle clusters based on user charging willingness; A conversion module, used for converting the cluster scheduling problem into a Markov process based on the electric vehicle cluster scheduling range to obtain a conversion result; The scheduling output module is used to design a reward function for each charging station based on the conversion result, and solve it through a deep reinforcement algorithm to obtain the final electric vehicle cluster scheduling strategy.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the electric vehicle cluster scheduling strategy based on deep reinforcement learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the electric vehicle cluster scheduling strategy based on deep reinforcement learning as described in any one of claims 1 to 7.

Images