Electric vehicle charging station charging power allocation method based on deep reinforcement learning

Abstract

The application discloses a kind of electric vehicle charging station charging power distribution methods based on deep reinforcement learning, it is characterized in that, including the following steps: step 1, according to the location of charging station, the initial electric quantity and position of electric vehicle, for user screening out suitable charging station, to obtain the charging station candidate solution set of electric vehicle;Step 2, according to space load balancing selection will be electric vehicle to the lowest load charging station, and give optimal charging station in combination with electric vehicle to reach charging station time;Step 3, according to the information of electric vehicle in the charging station, electric vehicle charging resource allocation problem is converted into Markov process problem;Step 4, Markov process problem is solved iteratively using deep reinforcement learning algorithm, so that the final action value is as the final power distribution mode.The method can effectively reduce the charging cost of user and the load gap of distribution network, be conducive to the load balancing between multiple distribution networks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric vehicle charging network, and particularly refers to a charging power allocation method for electric vehicle charging station based on deep reinforcement learning. BACKGROUND

[0002] In the global fight against climate change and the urgent task of reducing greenhouse gas emissions, electric vehicles are playing an increasingly important role and offering great potential for improving the sustainability and ecological friendliness of urban areas.

[0003] However, the popularity of electric vehicles has also had a significant impact on the load distribution in the distribution network, affecting the operation of the distribution network and the charging cost of users. On the one hand, the charging behavior of electric vehicle users has high uncertainty; disordered charging can exacerbate the burden on the distribution network during peak periods, threatening the safe operation of the grid and increasing the charging cost of users. On the other hand, as an important adjustable resource in the distribution network, the flexibility of electric vehicles lies in the controllability of charging location and charging process. If the charging behavior of electric vehicles can be effectively guided and their flexibility fully utilized, not only can the negative impact of electric vehicles on the operation of the distribution network be reduced, but also the load balancing among distribution networks and the reduction of user charging costs can be promoted.

[0004] In recent years, the academic community has made significant progress in the study of electric vehicle charging behavior, which can be mainly divided into two categories. The first category focuses on how to optimize the selection of electric vehicle charging stations to achieve effective coordination of charging behavior. In this regard, researchers will guide users to specific charging stations according to different optimization goals. However, this type of research mainly focuses on the selection of charging stations, relatively ignoring the charging flexibility of electric vehicle users within the charging station. The other category focuses on how to implement orderly charging control within the charging station for large-scale electric vehicles. Although these studies have made significant progress, most of the research scenarios are limited to a single charging station, and the mutual influence between the distribution networks where the charging stations are located is not fully considered.

[0005] In order to make up for the shortcomings of existing research and fully utilize the charging flexibility of electric vehicles, this application proposes a new electric vehicle charging process optimization method to minimize the charging cost of electric vehicles and balance the load among distribution networks. This method considers the flexibility of electric vehicles in both charging station selection and charging process within the station, and divides the charging process optimization into two optimization stages: charging station selection and charging within the station. In the first stage, the appropriate charging station is selected for the user, and in the second stage, the appropriate charging power is selected for the user. Compared with the load balancing matching strategy, this method not only achieves load balancing, but also minimizes user cost. SUMMARY

[0006] The present application aims at the deficiencies of the prior art, and proposes a charging power allocation method for electric vehicle charging stations based on deep reinforcement learning, which fully utilizes the flexibility of electric vehicles, reduces the charging cost of electric vehicle users and the load fluctuation between distribution networks, and determines the candidate charging station solution set of the electric vehicle through the electric vehicle and the charging station position. The candidate solution set is recommended to the electric vehicle. And after the electric vehicle arrives at the charging station, the electric vehicle is allocated with appropriate charging power according to the different optimization objectives of the charging station. Compared with the load balancing matching strategy, the power allocation method considering the minimum user cost minimization and the power allocation method considering the minimum load balancing minimization, the charging cost of the user and the load difference of the distribution network can be effectively reduced, which is beneficial to the load balancing among multiple distribution networks.

[0007] In order to solve the above technical problems, the technical scheme of the present application is:

[0008] A charging power allocation method for electric vehicle charging stations based on deep reinforcement learning, comprising the following steps:

[0009] S1, selecting a suitable charging station for the user according to the position of the charging station, the initial electric quantity and position of the electric vehicle;

[0010] S1-1, calculating the distance matrix R of the user to each charging station according to the position and initial electric quantity of the electric vehicle.

[0011] S1-2, calculating the farthest distance R that the electric vehicle can travel under the remaining electric quantity according to the initial battery state of charge S i ini of the i-th electric vehicle and the maximum distance that the electric vehicle can travel. i Combined with the matrix R, the charging station candidate solution set Cand i of the electric vehicle is obtained.

[0012] S2, recommending the optimal charging station for the electric vehicle in the candidate charging station solution set.

[0013] S2-1, in order to realize spatial load balancing, the method selects to distribute the electric vehicle to the charging station with the lowest load. Assuming that the electric vehicle travels to the charging station j at a constant travel speed v in the road, the travel time of the i-th electric vehicle to the j-th charging station is

[0014] S2-2, based on the travel time and the time when the electric vehicle requests charging , the time when the electric vehicle i arrives at the charging station j can be obtained.

[0015] S3, converting the electric vehicle charging resource allocation problem into a Markov process problem according to the information of the electric vehicle in each charging station.

[0016] S3-1, according to the user's charging demand d i and charging parking time p i Calculate the number of charging time slots c of the electric vehicle in the full charging state i,t and the number of expected parking time slots e i,t , and calculate the user's emergency degree according to the two data.

[0017] S3-2, count the number of electric vehicles with the same number of charging time slots and the number of expected parking time slots in each charging station, and the current time t as the state S of the Markov process problem t .

[0018] S3-3, select the charging action a of the current charging station at the current time j,t , and execute the charging action according to the user's emergency degree.

[0019] S3-4, calculate the reward function respectively The reward function of minimizing the load difference DRL_Space between charging stations and the reward function r considering the user cost and space load balancing DRL_Both j,t .

[0020] S3-5, according to the state S t , action a j,t and different reward functions, calculate the Q value function Q j (s,a)

[0021] S4, according to the information of electric vehicles arriving at the charging station, select the number of charging vehicles by using deep reinforcement learning algorithm.

[0022] S4-1, randomly initialize the parameters of the training network, and initialize the parameters of the target network so that the parameters in the two networks are the same.

[0023] S4-2, generate the initial state s of the charging station at the current time according to the information of electric vehicles in the charging station t

[0024] S4-3, select the charging action a according to the current network parameters and the greedy strategy t , and execute the charging action according to the user's emergency degree.

[0025] S4-4, calculate the immediate reward r t , and get the Markov state s of the next time according to the information of the newly arrived electric vehicles in the charging station and the electric vehicles which have not completed the charging demand t+1 .

[0026] S4-5, put the sample (s t, ,a t ,r t ,s t+1 ) into the playback cache pool, and select a certain number of samples from the cache pool for training, and update the network parameters.

[0027] S4-6, repeat the operations of S4-1 to S4-6 until the episode reaches the preset maximum value, and the action a episode as the final power allocation mode.

[0028] The application has the following characteristics and beneficial effects:

[0029] By using the above technical solution, the flexibility of the charging behavior of the electric vehicle user can be fully utilized, the combination of the load balancing matching strategy and the deep reinforcement learning is used, and the charging station selection and the charging process in the station are effectively optimized. Compared with the greedy algorithm, the DRL_Cost and the DRL_Space strategy, the DRL_Both can more effectively reduce the charging cost of the user, significantly improve the load balancing problem between the multiple power distribution networks, and improve the operation stability of the system. Moreover, the DRL_Both strategy can still achieve the best balance between the charging cost and the load balancing under different user participation, which is helpful for the safe and efficient operation of the power grid. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0031] Figure 1 It is a method flowchart of the embodiments of the present application.

[0032] Figure 2 It is the algorithm convergence when the optimization target is to minimize the load difference between the charging stations in the embodiments of the present application.

[0033] Figure 3 It is the algorithm convergence when the optimization target is to minimize the user charging cost in the embodiments of the present application.

[0034] Figure 4 It is the algorithm convergence when the charging station load difference and the user cost are comprehensively considered in the embodiments of the present application.

[0035] Figure 5Comparison of space load valley-to-peak ratio under different vehicle numbers in the embodiments of the present application.

[0036] Figure 6 Comparison of space load standard deviation under different vehicle numbers in the embodiments of the present application.

[0037] Figure 7 Comparison of total user charging cost under different vehicle numbers in the embodiments of the present application.

[0038] Figure 8 Comparison of comprehensive performance considering charging station load gap and user cost under different vehicle numbers in the embodiments of the present application.

[0039] Figure 9 Comparison of space load valley-to-peak ratio under different user participation in the embodiments of the present application.

[0040] Figure 10 Comparison of space load standard deviation under different user participation in the embodiments of the present application.

[0041] Figure 11 Comparison of total user charging cost under different user participation in the embodiments of the present application.

[0042] Figure 12 Comparison of comprehensive performance considering charging station load gap and user cost under different user participation in the embodiments of the present application. DETAILED DESCRIPTION

[0043] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0044] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0045] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0046] Embodiments

[0047] The present application provides a kind of based on deep reinforcement learning electric vehicle charging station recommendation and charging power allocation method, as shown in Figure 1 Including the following steps:

[0048] S1, according to the location of charging station, the initial electric quantity and position of electric vehicle, select suitable charging station for user;

[0049] S1-1, according to the position and initial electric quantity of electric vehicle, the distance matrix R of user to reach each charging station is calculated.In the present application, the definition of matrix R is as follows:

[0050]

[0051] Wherein R i,j Represent the distance of the i-th vehicle to reach the j-th charging station, N represents the total number of electric vehicles requesting charging.

[0052] S1-2, according to the initial battery charge state And the maximum distance R max Electric vehicle can travel, the farthest distance R i It can travel under the remaining electric quantity, R i Satisfies

[0053]

[0054] The charging station candidate solution set Cand i Of this electric vehicle is obtained in combination with matrix R.Cand i Should satisfy

[0055]

[0056] Wherein Is all charging station set that meets the above market.

[0057] S2, recommend the optimal charging station for electric vehicle in candidate charging station solution set.

[0058] S2-1, in order to achieve spatial load balancing, the method selects to distribute electric vehicles to the charging station with the lowest load, that is

[0059]

[0060] where P j,t represents the load of the jth charging station at time t.

[0061] Assuming that the electric vehicle travels to the charging station j at a constant speed v in the road, the travel time of the ith electric vehicle to the jth charging station is satisfying the formula

[0062]

[0063] S2-2, based on the travel time and the time when the electric vehicle requests charging The time when the electric vehicle i arrives at the charging station j can be obtained, and the formula can be expressed as

[0064]

[0065] S3, according to the information of the electric vehicles in each charging station, the electric vehicle charging resource allocation problem is converted into a Markov process problem.

[0066] S3-1, according to the user's charging demand d i and the charging parking time p i Calculate the number of charging time slots c i,t and the expected parking time slot e i,t of the electric vehicle in the full charging state, and the formula can be expressed as

[0067]

[0068]

[0069] where ΔT is the time difference between adjacent two time slots, and p is the charging power of the charging station. And according to the two data, the urgency of the user is calculated The calculation formula can be expressed as

[0070]

[0071] S3-2, count the number of electric vehicles with the same charging time slot and expected parking time slot in each charging station and the current time t as the state S of the Markov process problem t , then S t can be expressed as

[0072]

[0073] wherein represents the total number of electric vehicles in the charging station, the number of charging time slots is c, and the number of expected parking time slots is e. max represents the maximum charging time required by the electric vehicle, E max represents the maximum parking time of the electric vehicle. Thus, S t has a dimension of

[0074]

[0075] In this embodiment, the Markov state dimension is 64. The action is divided into 11 equidistant discrete values between 0 and 1.

[0076] S3-3, selecting the charging action a of the current charging station at the current time j,t , i.e. the action value of the Markov process problem, in the present application, a j,t can be expressed as

[0077]

[0078] wherein represents the number of electric vehicles being charged at the current time t, the total number of electric vehicles at the current time. And according to the urgency of the user, the charging action is executed.

[0079] It can be understood that the greater the value, the more urgent the user's charging demand, which needs to be charged in priority; the same user means the same degree of urgency, in order to minimize the risk brought by unknown future vehicles, the charging with less remaining parking time should be selected in priority.

[0080] S3-4, respectively calculating the reward function of the optimization target of minimizing the user cost DRL_Cost can be expressed as

[0081]

[0082] wherein p j,t represents the charging price of the jth charging station at time t.

[0083] minimizing the reward function of the load difference DRL_Space between charging stations can be expressed as

[0084]

[0085] wherein represents the basic load of the jth power distribution network at time t when there is no electric vehicle charging, Total load of all power distribution networks at time t when there is no electric vehicle charging.

[0086] Because and have different quantization levels, in order to enable the user's charging cost and the load gap between charging stations to be jointly optimized, both of them need to be normalized, and the formula for normalization is as follows

[0087]

[0088] Wherein, x t represents the value of the element at time t, x max represents the maximum value of the same type of element, and x min represents the minimum value of the same type of element. Then consider the reward function r j,t of user cost and space load balancing DRL_Both

[0089]

[0090] Wherein, α and β represent the weight factors when the reward function is minimized, respectively, user charging cost and charging station load balancing, represents the normalized value of the price, represents the normalized value of the load gap.

[0091] S3-5, according to the state S t , action a j,t and different reward functions, calculate the Q value function Q j (s,a) under the current state-action, and the calculation formula is

[0092]

[0093] Wherein, E represents the expectation function, γ represents the discount factor, r j,t+k represents the reward of the agent at future time, and Q j (s,a) represents the cumulative reward expectation of taking action a in state s.

[0094] S4, according to the information of electric vehicles arriving at the charging station, select the number of charging vehicles by using deep reinforcement learning algorithm.

[0095] S4-1, construct two deep neural networks, randomly initialize the parameters θ of the training network, and initialize the parameters of the target network so that the parameters in the two networks are the same.

[0096] Specifically, in this embodiment, both the training network and the target network are set with two hidden layers, each layer having 128 neurons, and the activation function being a Relu function. The discount factor γ, the learning rate μ, the batch size, the update frequency, and the number of iteration rounds are set to 0.95, 0.0005, 64, 10000, and 80000, respectively.

[0097] S4-2, initialize the number of training iterations episode = 0. Initialize t = 0, and generate the initial state s0 of the current charging station according to the information of the electric vehicles in the charging station;

[0098] S4-3, select the charging action a according to the current training network parameters and the greedy strategy argmaxQ(s, a | θ) t The strategy for selecting the action is as follows

[0099]

[0100] where ε is the exploration rate, λ is a random number gradually decreasing from 1 to 0, and A is the set of actions that can be selected. The charging action is executed according to the urgency of the user.

[0101] S4-4, calculate the immediate reward r t , and obtain the next state s t+1 according to the information of the newly arrived electric vehicles in the charging station and the electric vehicles that have not completed the charging demand.

[0102] S4-5, put the sample (s t, , a t , r t , s t+1 ) into the replay buffer pool, select a certain number of samples from the buffer pool for training, and update the network parameters. The update formula is

[0103]

[0104] L(θ) = E[(q target - Q(s, a | θ)) 2 ] (20)

[0105]

[0106] where q target is the target Q value, s' represents the next state, a' represents all possible actions in the next state, μ is the learning rate, is the gradient operator

[0107] S4-6, repeat the operations of S4-1 to S4-6 until the episode reaches the preset maximum value, and the action a episodeAs the final power allocation manner.

[0108] Comparative example

[0109] In this comparative example, the greedy, DRL_Cost, DRL_Space and DRL_Both strategies are compared.

[0110] First, the simulation parameters are set:

[0111] The basic load data of the distribution network comes from the actual power demand data of California. After scaling down by a certain proportion, the data is randomly allocated to the distribution network. Then, according to the basic load of the distribution network, the time-of-use price is formulated according to reference [1]. In this paper, the direct current charging method is adopted, and according to the current charging rate estimation, the maximum charging time required by the electric vehicle is set to 7 time slots, and the maximum parking time of the electric vehicle in the charging station is set to 12 time slots. The sampling interval of the data is 15 minutes, i.e. ΔT = 15. In the simulation, the total number of time samples T d is 96, and the sampling time window is 0:00-23:55. All charging requests of electric vehicles are sent in this time window. The number of distribution networks M is set to 4, which are evenly distributed in a range of 30x30km, and the charging power of the charging station in the distribution network is p = 20kW. The average speed of electric vehicle users in the city is v = 30km / h, and the maximum driving distance of the electric vehicle is R max = 300km.

[0112] Then, through simulation result analysis, the convergence of DRL_Cost, DRL_Space and DRL_Both strategies.

[0113] Using the above parameter configuration, the neural network is configured, and the convergence performance of the DDQN algorithm under different optimization objectives is compared in the comparative example. Figure 2 、 Figure 3 and Figure 4 show the algorithm convergence under DRL_Space, DRL_Cost and DRL_Both strategies when 750 electric vehicles request charging. From the figure, it can be observed that the exploration rate is high at the beginning of the algorithm, which encourages the agent to explore, so that the reward value increases continuously. With the passage of time, the exploration rate gradually decreases, and the reward value tends to be stable, and finally the convergence is realized. The increase of the reward value and the final stability show that the MDP model used in this paper performs well when applying the DDQN algorithm.

[0114] (1) Through simulation result analysis, the space load balancing performance of the four algorithms

[0115] To compare the performance of different algorithms in terms of spatial load optimization, we introduce the valley-to-peak ratio η(t) to measure the degree of spatial load balancing of the system, and comprehensively evaluate the impact of electric vehicle charging on spatial load by calculating the average spatial load valley-to-peak ratio (SLVTPR) and the average spatial load standard deviation. Specifically, the formulas for calculating the spatial load valley-to-peak ratio and the SLVTPR are

[0116]

[0117] However, Only reflects the extreme difference between distribution networks, it is difficult to capture the overall load difference between all distribution networks. Therefore, it is necessary to introduce the average spatial load standard deviation to better demonstrate the load difference between all distribution networks. Specifically, the average spatial load standard deviation The formula for calculating the average spatial load standard deviation is

[0118]

[0119] Where D(t) is the load standard deviation between distribution networks at time t.

[0120] Figure 5 And Figure 6 Show the performance of the average spatial load valley-to-peak ratio and the average spatial load standard deviation in the distribution network under different numbers of electric vehicle charging requests. From these figures, it can be seen that under different optimization functions, the comparison between the greedy strategy and the DDQN-based strategy shows that although the greedy strategy can achieve spatial load balancing to some extent, its effect is not as good as that of the DRL_Space strategy and the DRL_Both strategy. This is because the greedy strategy only considers the load of the distribution network when the user makes a charging request when selecting a charging station for the user. However, when the user arrives at the charging station, the load of the station may no longer be the lowest among all distribution networks. In addition, the greedy strategy prioritizes meeting the immediate charging needs of users, which, although to some extent, achieves spatial load balancing, but fails to achieve the best results.

[0121] In contrast, the DRL_Space strategy and the DRL_Both strategy perform better in terms of spatial load balancing. Although the DDQN algorithm uses the same charging station allocation strategy as the greedy strategy when allocating charging stations for users, it optimizes the charging power selection based on the base load of the distribution network when the user arrives at the charging station. Under this charging power allocation strategy, the actions taken by the agent enable the load distribution of the distribution network to be more balanced.

[0122] On the other hand, the DRL_Cost strategy does not consider spatial load balancing in its charging strategy, so its load balancing effect is weaker compared to the other three strategies.

[0123] (2) Through the analysis of simulation results, the charging price performance of the four algorithms

[0124] From Figure 7 It can be seen that, compared with the strategy based on the DDQN algorithm, the greedy strategy leads to the highest user charging cost. This is because the greedy strategy satisfies the user's charging demand as quickly as possible, ignoring the user's flexibility during parking, resulting in an increase in charging cost.

[0125] In contrast, the DDQN-based strategy takes advantage of the user's flexibility and can effectively adjust the charging power according to different load conditions and electricity prices. Therefore, the user charging cost under the three DDQN algorithms is lower than that of the greedy strategy. However, different reward functions have different effects on user charging cost. In the DRL_Space strategy, the agent does not consider the electricity price information, so it cannot adjust the charging power according to the electricity price of the charging station, resulting in its performance in user charging cost being inferior to the other two DDQN algorithms that include cost considerations.

[0126] For the DRL_Both strategy, its reward function design includes the consideration of user charging cost, so the user charging cost is lower than that of the DRL_Space strategy. However, this strategy needs to balance between minimizing charging cost and spatial load standard deviation, so its cost is higher than that of the DRL_Cost strategy.

[0127] In the case of the DRL_Cost strategy, the information input to the agent only includes the charging price of each time period. After multiple iterations, the agent will choose a charging strategy that can minimize the user's charging cost. Therefore, this reward function achieves the lowest user charging cost.

[0128] (3) Through the analysis of simulation results, the comprehensive performance of the DRL_Cost, DRL_Space, and DRL_Both strategies

[0129] As Figure 8 shown, the normalized reward values of the DRL_Space strategy and the DRL_Cost strategy are both 1. In the DRL_Cost strategy, this method selects the charging power that can minimize the electric vehicle charging price, ignoring the load distribution between power grids. On the contrary, the DRL_Space strategy prioritizes charging power that can balance the load between power grids, but does not consider the user's charging cost. These two strategies focus on one aspect respectively, limiting the overall performance of the algorithm.

[0130] However, the highest normalized reward value is achieved by the DRL_Both policy. This is because the policy considers both the user charging cost and the load balancing when selecting the charging power at each time period. After balancing these two factors, the actions selected by the policy can both reduce the user charging cost and minimize the load difference among distribution networks. Therefore, the policy achieves the highest reward value and the most balanced overall performance of the system.

[0131] From Figure 9 and Figure 10 It can be seen that as the number of users participating in scheduling increases, the SLVTPR gradually increases, while the spatial load standard deviation gradually decreases. This indicates that the more users participate in scheduling, the more balanced the spatial load is.

[0132] In the case of low user participation, only a small number of users choose the charging station with the lowest load, while most users randomly choose one of the four charging stations. In this case, the greedy policy performs the worst in terms of spatial load balancing performance. This is because the greedy policy starts charging immediately after the user arrives, and this method highly depends on user participation. If users do not accept the allocation of charging stations, the policy can exacerbate load imbalance. In contrast, the DDQN-based charging policy is generally superior to the greedy policy. This is because the DDQN policy adjusts the charging power according to different objective functions, and the agent can still find the most suitable charging rate for the charging station even in the case of low user participation.

[0133] As user participation increases, the greedy policy outperforms the DRL_Cost policy in spatial load balancing performance. This is because, under high user participation, most users will choose a charging station with lower load when requesting, while the DRL_Cost policy selects actions to reduce charging costs, but it does not consider the load of other charging stations, so it is weaker than the greedy policy, the DRL_Space policy, and the DRL_Both policy in terms of spatial load balancing performance.

[0134] From Figure 11 It can be seen that as the number of users participating in scheduling increases, the overall charging cost of users gradually decreases. This is because higher user participation allows users to choose charging stations with lower loads, and these charging stations also have lower prices, thereby reducing the total charging cost. Under different user participation, the simulation results are consistent with the previous simulation results: the DRL_Cost policy has the lowest charging cost, followed by the DRL_Both policy, then the DRL_Space policy, and the greedy policy has the highest cost.

[0135] (4) Analysis of the performance of the four algorithms under different user participation levels based on simulation results

[0136] Finally, from Figure 12It can be seen that under different user participation, the algorithm based on DDQN still maintains the same rule as the previous simulation results under different optimization functions. The DRL_Both strategy achieves the best system performance, indicating that this weighting method can enable the system to select the optimal charging strategy.

[0137] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components are made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A charging power allocation method for electric vehicle charging stations based on deep reinforcement learning, characterized in that: The steps include: Step 1: Based on the location of the charging station, the initial power level of the electric vehicle, and the location, a suitable charging station is selected for the user, thereby obtaining a set of candidate charging station solutions for the electric vehicle; Step 2: Based on spatial load balancing, electric vehicles are assigned to the charging station with the lowest load, and the optimal charging station is determined based on the time it takes for the electric vehicles to arrive at the charging station. Step 3: Based on the information of electric vehicles arriving at the charging station, the electric vehicle charging resource allocation problem is converted into a Markov process problem. The electric vehicle information includes the charging demand and charging parking time of the electric vehicle. The Markov process problem includes solving the action value, state value and reward value; select the charging action a of the current charging station at the current moment. j,t As an action of the Markov process problem, a j,t Expressed as: in, represents the number of electric vehicles that the charging station chooses to charge at the current time t, The total number of electric vehicles at the current moment; and perform charging actions based on the user's urgency; Calculate the reward function whose optimization goal is to minimize the user cost DRL_Cost The expression is as follows: where p j,t represents the charging price of the jth charging station at time t; Calculate the reward function that minimizes the load gap DRL_Space between charging stations The expression is as follows: in represents the base load of the j-th distribution network when there is no electric vehicle charging at time t, P t tot The total load of all distribution networks at time t when there are no electric vehicles charging; The final Markov process problem is constructed as follows: because and The quantization levels of are different. Therefore, in order to optimize the user's charging cost and the load gap between charging stations, the two need to be normalized. The normalization formula is as follows: Among them, x t Represents the value of the element at time t, x max Indicates the maximum value of the same type of elements, x min Represents the minimum value of the same type of elements, then the reward function r of DRL_Both considering user cost and spatial load balancing j,t Calculated using the following formula: Among them, α and β represent the weight factors when the reward function is to minimize the user charging cost and the charging station load balance, respectively. Represents the normalized value of the price. It represents the value after normalizing the load gap; According to the state S t , action a j,t And different reward functions, calculate the Q value function Q under the current state-action j (s,a), the calculation formula is: Among them, E represents the expectation function, γ represents the discount factor, and r j,t+k Represents the agent’s reward at the future moment, Q j (s,a) represents the expected cumulative reward obtained by taking action a in state s; Step 4: Use the deep reinforcement learning algorithm to iteratively solve the Markov process problem, so that the final action value is used as the final power allocation method.

2. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 1, characterized in that: In step 1, the method for constructing a set of charging station candidate solutions is: First, based on the location and initial charge of the electric vehicle, the distance matrix R of the user to each charging station is calculated. The matrix R is defined as follows: where R i,j represents the distance from the i-th vehicle to the j-th charging station, and N represents the total number of electric vehicles requesting charging; According to the initial battery charge state of the i-th electric vehicle The maximum distance R that electric vehicles can travel max , calculate the maximum distance R that can be traveled with the remaining power i , R i satisfy: Combined with the matrix R, we can get the candidate solution set Cand for the electric vehicle charging station. i , Cand i Should meet the following requirements: in It is a collection of all charging stations that meet the listing requirements.

3. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the number of electric vehicles with the same number of charging time slots and expected parking time slots in each charging station and the current time t are counted as the state S of the Markov process problem. t , then S t Expressed as: in It represents the total number of electric vehicles in the charging station with a charging time slot number of c and an expected parking time slot number of e. max Represents the maximum charging time required for electric vehicles, E max It represents the maximum parking time of an electric vehicle.

4. The method for allocating charging power at an electric vehicle charging station based on deep reinforcement learning according to claim 3, characterized in that: The state S of the Markov process problem t The dimension calculation method is:

5. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 4 is characterized in that: The calculation method of the user urgency is: According to the user's charging needs i and charging parking time p i Calculate the number of charging time slots c of an electric vehicle in a fully charged state i,t and the expected number of parking slots e i,t , the formula is: Where ΔT is the time difference between two adjacent time slots, and p is the charging power of the charging station. The user's urgency is calculated based on these two data The calculation formula is:

6. The method for allocating charging power at an electric vehicle charging station based on deep reinforcement learning according to claim 5, characterized in that: In step 4, a deep reinforcement learning algorithm is implemented by constructing two deep neural networks with the same structure, where one deep neural network is a training network and the other deep neural network is a target network.

7. The method for allocating charging power at an electric vehicle charging station based on deep reinforcement learning according to claim 6, characterized in that: The step 4 includes the following sub-steps: Step 4-1: Randomly initialize the parameters θ of the training network and initialize the parameters of the target network Make the parameters in the two networks the same; Step 4-2: Initialize the number of training iterations to episode = 0, initialize t = 0, and generate the initial state s0 of the current charging station based on the information of the electric vehicles in the charging station; Step 4-3: Select charging action a based on the current training network parameters and greedy strategy argmaxQ(s,a|θ) t , the strategy for selecting actions is as follows: Step 4-4. Calculate the instant reward r t , and obtain the Markov state s at the next moment based on the information of electric vehicles that have newly arrived at the charging station and electric vehicles that have not yet completed charging needs t+1 ; Step 4-5, the sample (s t, ,a t ,r t ,s t+1 ) into the playback buffer pool, and select a certain number of samples from the buffer pool for training and update the network parameters; Step 4-6, repeat steps 4-1 to 4-6 until the episode reaches the preset maximum value, and set the action a of the episode to episode As the final power distribution method.

8. The method for allocating charging power at an electric vehicle charging station based on deep reinforcement learning according to claim 7, characterized in that: In steps 4-5, the method for updating the network parameters is: L(θ)=E[(q target -Q(s,a|θ)) 2 ] where q target The target Q value, s' represents the next state, a' represents all possible actions in the next state, μ is the learning rate, is the gradient operator.

Images