Electric vehicle charging station charging power distribution method based on deep reinforcement learning

Abstract

The invention discloses an electric vehicle charging station charging power distribution method based on deep reinforcement learning, and the method is characterized in that the method comprises the following steps: 1, screening a proper charging station for a user according to the position of the charging station, and the initial electric quantity and position of an electric vehicle, and obtaining a charging station candidate solution set of the electric vehicle; step 2, selecting to distribute the electric vehicle to the charging station with the lowest load according to the space load balance, and giving an optimal charging station by combining the time when the electric vehicle arrives at the charging station; step 3, according to the information of the electric vehicle arriving at the charging station, converting an electric vehicle charging resource allocation problem into a Markov process problem; and step 4, iteratively solving a Markov process problem by using a deep reinforcement learning algorithm, and taking a final action value as a final power distribution mode. The method can effectively reduce the charging cost of the user and the load gap of the power distribution network, and is beneficial to load balancing among multiple power distribution networks.

Description

Technical Field

[0001] The present invention relates to the technical field of electric vehicle charging networks, and in particular to a charging power distribution method for electric vehicle charging stations based on deep reinforcement learning. Background Art

[0002] In the urgent global task of combating climate change and reducing greenhouse gas emissions, electric vehicles are playing an increasingly important role and offer great potential for making urban areas more sustainable and eco-friendly.

[0003] However, the popularity of electric vehicles has also had a significant impact on the load distribution in the distribution network, thereby affecting the operation of the distribution network and the charging costs of users. On the one hand, the charging behavior of electric vehicle users has a high degree of uncertainty; disorderly charging may increase the burden on the distribution network during peak hours, threaten the safe operation of the power grid, and increase the charging costs of users. On the other hand, as an important adjustable resource in the distribution network, the flexibility of electric vehicles lies in the controllability of the charging location and charging process. If the charging behavior of electric vehicles can be effectively guided and their flexibility can be fully utilized, it can not only reduce their negative impact on the operation of the distribution network, but also promote the balancing of loads between distribution networks and the reduction of user charging costs.

[0004] In recent years, the academic community has made significant progress in the study of electric vehicle charging behavior, which can be divided into two categories. The first type of research focuses on how to optimize the selection of charging stations for electric vehicles to achieve effective coordination of charging behavior. In this work, researchers will guide users to specific charging stations based on different optimization objectives. However, this type of research mainly focuses on the selection of charging stations and relatively ignores the charging flexibility of electric vehicle users in charging stations. Another type of research focuses on how to implement orderly charging control in charging stations for large-scale electric vehicles. Although these studies have made significant progress, most of the research scenarios are limited to a single charging station and fail to fully consider the mutual influence between the distribution networks where the charging stations are located.

[0005] In order to make up for the shortcomings of existing research, make full use of the charging flexibility of electric vehicles, and achieve the minimization of electric vehicle charging costs and load balancing between distribution networks in a multi-distribution network environment, the present invention proposes a new electric vehicle charging process optimization method. This method comprehensively considers the flexibility of electric vehicles in the selection of charging stations and the charging process in the station, and divides the charging process optimization into two optimization stages: charging station selection and charging in the station. In the first stage, a suitable charging station is selected for the user, and in the second stage, a suitable charging power is selected for the user. Compared with the load balancing matching strategy, this method can not only achieve load balancing, but also minimize user costs. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention proposes a charging power allocation method for electric vehicle charging stations based on deep reinforcement learning, which makes full use of the flexibility of electric vehicles, reduces the charging cost of electric vehicle users and the load fluctuation between distribution networks, and determines the candidate charging station solution set for electric vehicles through the locations of electric vehicles and charging stations. Recommend suitable charging stations for electric vehicles in the candidate solution set. After the electric vehicle arrives at the charging station, appropriate charging power is allocated to the electric vehicle according to the different optimization objectives of the charging station. Compared with the load balancing matching strategy, the power allocation method considering user cost minimization and the power allocation method considering load balancing minimization, it can effectively reduce the user's charging cost and the load gap of the distribution network, which is conducive to load balancing between multiple distribution networks.

[0007] In order to solve the above technical problems, the technical solution of the present invention is:

[0008] A method for recommending electric vehicle charging stations and allocating charging power based on deep reinforcement learning comprises the following steps:

[0009] S1. Select a suitable charging station for the user based on the location of the charging station, the initial power level and location of the electric vehicle;

[0010] S1-1. According to the location and initial power of the electric vehicle, the distance matrix R of the user to each charging station is calculated.

[0011] S1-2, according to the initial battery charge state S of the i-th electric vehicle i ini and the maximum distance that an electric car can travel, calculate the maximum distance R that it can travel with the remaining power i Combined with the matrix R, we can get the candidate solution set Cand for the charging station of the electric vehicle. i .

[0012] S2. Recommend the best charging station for electric vehicles in the candidate charging station solution set.

[0013] S2-1. To achieve spatial load balancing, this method selects to assign electric vehicles to the charging station with the lowest load. Assuming that electric vehicles travel to charging station j at a constant speed v on the road, the travel time for the i-th electric vehicle to reach the j-th charging station is

[0014] S2-2, based on driving time and the time when the electric vehicle requests charging The time when electric vehicle i arrives at charging station j can be obtained.

[0015] S3. According to the information of electric vehicles in each charging station, the electric vehicle charging resource allocation problem is transformed into a Markov process problem.

[0016] S3-1. According to the user's charging needs i and charging parking time p i Calculate the number of charging time slots c of an electric vehicle in a fully charged state i,t and the estimated number of parking slots e i,t , and calculate the user's urgency based on these two data.

[0017] S3-2. Count the number of electric vehicles with the same number of charging time slots and expected parking time slots in each charging station and the state S at the current time t as the Markov process problem t .

[0018] S3-3, select the charging action a of the current charging station at the current moment j,t , and perform charging actions based on the user's urgency.

[0019] S3-4. Calculate the reward function whose optimization goal is to minimize the user cost DRL_Cost The reward function for minimizing the load gap DRL_Space between charging stations and the reward function r that comprehensively considers user cost and spatial load balancing DRL_Both j,t .

[0020] S3-5, according to the state S t , action a j,t And different reward functions, calculate the Q value function Q under the current state-action j (s,a)

[0021] S4. Based on the information of electric vehicles arriving at the charging station, the number of charging vehicles is selected using a deep reinforcement learning algorithm.

[0022] S4-1. Randomly initialize the parameters of the training network and initialize the parameters of the target network so that the parameters in the two networks are the same.

[0023] S4-2. Generate the initial state s of the charging station at the current moment according to the information of the electric vehicles in the charging station t

[0024] S4-3. Select charging action a according to current network parameters and greedy strategy t , and performs charging actions based on the user's urgency.

[0025] S4-4. Calculate the instant reward r t , and obtain the Markov state s at the next moment based on the information of the electric vehicles that have just arrived at the charging station and the electric vehicles that have not yet completed the charging demand t+1 .

[0026] S4-5, the sample (s t, ,a t ,r t ,s t+1 ) is put into the playback buffer pool, and a certain number of samples are selected from the buffer pool for training and updating the network parameters.

[0027] S4-6, repeat the operations from S4-1 to S4-6 until the episode reaches the preset maximum value, and set the action a of the episode episode As the final power distribution method.

[0028] The present invention has the following characteristics and beneficial effects:

[0029] The above technical solution can make full use of the flexibility of electric vehicle users' charging behavior, and effectively optimize the selection of charging stations and the charging process within the station by combining the load balancing matching strategy with deep reinforcement learning. Compared with the greedy algorithm, DRL_Cost and DRL_Space strategies, DRL_Both can more effectively reduce the user's charging cost, significantly improve the load balancing problem between multiple distribution networks, and improve the system's operating stability. In addition, the DRL_Both strategy can still achieve the best balance between charging cost and load balancing under different user participation levels, which is conducive to the safe and efficient operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention.

[0032] Figure 2 This is the convergence of the algorithm when the optimization goal is to minimize the load difference between charging stations in the embodiment of the present invention.

[0033] Figure 3 This is the convergence of the algorithm when the optimization goal is to minimize the user's charging cost in the embodiment of the present invention.

[0034] Figure 4 This is the convergence of the algorithm when comprehensively considering the load gap between charging stations and user costs in the embodiment of the present invention.

[0035] Figure 5The figure is a comparison of the valley-to-peak ratio of the spatial load under different numbers of vehicles in the embodiment of the present invention.

[0036] Figure 6 1 is a comparison of the standard deviation of space load under different numbers of vehicles in the embodiments of the present invention.

[0037] Figure 7 The figure is a comparison of the total charging cost for users with different numbers of vehicles in the embodiments of the present invention.

[0038] Figure 8 This is a comprehensive performance comparison considering the load gap of charging stations and user costs under different numbers of vehicles in the embodiments of the present invention.

[0039] Fig. 9 The figure is a comparison of the valley-to-peak ratio of spatial load under different user participation levels in the embodiment of the present invention.

[0040] Fig.10 The figure is a comparison of the standard deviation of the spatial load under different user participation levels in the embodiments of the present invention.

[0041] Fig.11 The figure is a comparison of the total charging cost of users at different user participation levels in the embodiments of the present invention.

[0042] Fig.12 This is a comprehensive performance comparison considering the charging station load gap and user cost under different user participation levels in the embodiments of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0044] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0045] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0046] Example

[0047] The present invention provides a method for recommending electric vehicle charging stations and allocating charging power based on deep reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0048] S1. Select a suitable charging station for the user based on the location of the charging station, the initial power level and location of the electric vehicle;

[0049] S1-1. Calculate the distance matrix R of the user to each charging station according to the location and initial power of the electric vehicle. In the present invention, the matrix R is defined as follows:

[0050]

[0051] Where R i,j represents the distance for the i-th vehicle to reach the j-th charging station, and N represents the total number of electric vehicles requesting charging.

[0052] S1-2, according to the initial battery charge state of the i-th electric vehicle and the maximum distance that an electric vehicle can travel R max , calculate the maximum distance R that can be traveled with the remaining power i , R i satisfy

[0053]

[0054] Combined with the matrix R, we can get the candidate solution set Cand for the charging station of the electric vehicle. i . Cand i Should meet

[0055]

[0056] in It is a collection of all charging stations that meet the listing.

[0057] S2. Recommend the best charging station for electric vehicles in the candidate charging station solution set.

[0058] S2-1. To achieve spatial load balancing, this method chooses to assign electric vehicles to the charging station with the lowest load, i.e.

[0059]

[0060] Where P j,t It represents the load of the jth charging station at time t.

[0061] Assuming that the electric vehicle travels on the road at a constant speed v to the charging station j, the travel time of the i-th electric vehicle to the j-th charging station is Satisfy the formula

[0062]

[0063] S2-2, based on driving time and the time when the electric vehicle requests charging The time it takes for electric vehicle i to reach charging station j can be obtained. The formula can be expressed as

[0064]

[0065] S3. According to the information of electric vehicles in each charging station, the electric vehicle charging resource allocation problem is transformed into a Markov process problem.

[0066] S3-1. According to the user's charging needs i and charging parking time p i Calculate the number of charging time slots c of an electric vehicle in a fully charged state i,t and the estimated number of parking slots e i,t , the formula can be expressed as

[0067]

[0068]

[0069] Where ΔT is the time difference between two adjacent time slots, and p is the charging power of the charging station. The user's urgency is calculated based on these two data The calculation formula can be expressed as

[0070]

[0071] S3-2. Count the number of electric vehicles with the same number of charging time slots and expected parking time slots in each charging station and the state S at the current time t as the Markov process problem t , then S t It can be expressed as

[0072]

[0073] in It represents the total number of electric vehicles in the charging station with a charging time slot number of c and an expected parking time slot number of e. max Represents the maximum charging time required for electric vehicles, E max represents the maximum parking time of the electric vehicle. From this, S t The dimension is

[0074]

[0075] In this embodiment, the Markov state dimension is 64. The actions are divided into 11 equally spaced discrete values ​​between 0 and 1.

[0076] S3-3, select the charging action a of the current charging station at the current moment j,t , that is, the action value of the Markov process problem. In the present invention, a j,t It can be expressed as

[0077]

[0078] in represents the number of electric vehicles being charged at the current time t, The total number of electric vehicles at the current moment. And perform charging actions according to the user's urgency.

[0079] Understandably, The larger the value, the more urgent the user's charging needs are, and charging should be given priority. The same user instructions have the same urgency. In order to minimize the risks brought by unknown future vehicles, priority should be given to charging vehicles with less remaining parking time.

[0080] S3-4. Calculate the reward function whose optimization goal is to minimize the user cost DRL_Cost It can be expressed as

[0081]

[0082] where p j,t It represents the charging price of the jth charging station at time t.

[0083] The reward function for minimizing the load gap DRL_Space between charging stations It can be expressed as

[0084]

[0085] in represents the base load of the j-th distribution network when there is no electric vehicle charging at time t. The total load of all distribution networks when there are no electric vehicles charging at time t.

[0086] because and The quantization levels are different. Therefore, in order to optimize the user's charging cost and the load difference between charging stations, the two need to be normalized. The normalization formula is as follows

[0087]

[0088] Among them, x t represents the value of the element at time t, x max Indicates the maximum value of the same type of elements, x min represents the minimum value of the same type of elements. Then the reward function r of DRL_Both considering user cost and spatial load balancing is j,t It can be calculated using the following formula

[0089]

[0090] Among them, α and β represent the weight factors when the reward function is to minimize the user charging cost and the charging station load balance, respectively. It represents the normalized value of the price. It represents the normalized value of the load difference.

[0091] S3-5, according to the state S t , action a j,t And different reward functions, calculate the Q value function Q under the current state-action j (s,a), the calculation formula is

[0092]

[0093] Among them, E represents the expected function, γ represents the discount factor, and r j,t+k Represents the agent’s reward at the future moment, Q j (s,a) represents the expected cumulative reward for taking action a in state s.

[0094] S4. Based on the information of electric vehicles arriving at the charging station, the number of charging vehicles is selected using a deep reinforcement learning algorithm.

[0095] S4-1. Build two deep neural networks, randomly initialize the parameters θ of the training network, and initialize the parameters of the target network Make the parameters in the two networks the same.

[0096] Specifically, in this embodiment, both the training network and the target network are set with two hidden layers, each layer has 128 neurons, the activation function is the Relu function, the discount factor γ, the learning rate μ, the batch size, the update frequency and the number of iteration rounds are set to 0.95, 0.0005, 64, 10000 and 80000 respectively.

[0097] S4-2, initialize the number of training iterations episode = 0. Initialize t = 0, generate the initial state s0 of the current charging station according to the information of the electric vehicles in the charging station;

[0098] S4-3. Select charging action a according to the current training network parameters and greedy strategy argmaxQ(s,a|θ) t , the strategy for selecting actions is as follows

[0099]

[0100] Where ε is the exploration rate, λ is a random number that decreases from 1 to 0, and A is the set of actions that can be selected. And the charging action is performed according to the urgency of the user.

[0101] S4-4. Calculate the instant reward r t , and obtain the Markov state s at the next moment based on the information of the electric vehicles that have just arrived at the charging station and the electric vehicles that have not yet completed the charging demand t+1 .

[0102] S4-5, the sample (s t, ,a t ,r t ,s t+1 ) is put into the playback buffer pool, and a certain number of samples are selected from the buffer pool for training, and the network parameters are updated. The update formula is

[0103]

[0104] L(θ)=E[(q target -Q(s,a|θ)) 2 ] (20)

[0105]

[0106] where q target The target Q value, s' represents the next state, a' represents all possible actions in the next state, μ is the learning rate, is the gradient operator

[0107] S4-6, repeat the operations from S4-1 to S4-6 until the episode reaches the preset maximum value, and set the action a of the episode episodeAs the final power distribution method.

[0108] Comparative Example

[0109] In this comparative example, the greedy, DRL_Cost, DRL_Space and DRL_Both strategies are compared.

[0110] First set the simulation parameters:

[0111] The basic load data of the distribution network comes from the actual power demand data of California. After being reduced by a certain ratio, the data is randomly distributed to the distribution network. Then, according to the basic load of the distribution network, the time-of-use electricity price is formulated in reference [1]. This paper adopts the DC charging method. According to the current charging rate of the charging pile, the maximum charging time required for electric vehicles is set to 7 time slots, and the maximum parking time of electric vehicles in the charging station is set to 12 time slots. The sampling interval of the data is 15 minutes, that is, ΔT = 15. In the simulation, the total number of time samples T d is 96, and the sampling time window is 0:00-23:55. All charging requests of electric vehicles are sent in this time window. The number of distribution networks M is set to 4, evenly distributed in the range of 30×30km, and the charging power of the charging station in the distribution network is p=20kW. The speed of electric vehicle users in the city is v=30km / h, and the maximum driving distance of electric vehicles is R max =300km.

[0112] Then, the convergence of DRL_Cost, DRL_Space and DRL_Both strategies is analyzed through simulation results.

[0113] The neural network is configured using the above parameters, and the convergence performance of the DDQN algorithm under different optimization objectives is compared in the comparative example. Figure 2 , Figure 3 and Figure 4 The convergence of the algorithms under the DRL_Space, DRL_Cost, and DRL_Both strategies is shown when 750 electric vehicles request charging. It can be observed from the figure that the exploration rate is high at the beginning of the algorithm, which encourages the agent to explore, so that the reward value continues to increase. Over time, the exploration rate gradually decreases, the reward value also tends to stabilize, and finally convergence is achieved. The increase in reward value and the final stability show that the MDP model used in this paper performs well when applying the DDQN algorithm.

[0114] (1) Through simulation results analysis, the spatial load balancing performance of the four algorithms

[0115] In order to compare the performance of different algorithms in spatial load optimization, we introduce the valley-to-peak ratio η(t) to measure the spatial load balancing degree of the system, and comprehensively evaluate the impact of electric vehicle charging on spatial load by calculating the average spatial load valley-to-peak ratio (SLVTPR) and the average spatial load standard deviation. Specifically, the calculation formulas for the spatial load valley-to-peak ratio and SLVTPR are:

[0116]

[0117] However, It is difficult to capture the overall load gap between all distribution networks by only reflecting the extreme differences between them. Therefore, it is necessary to introduce the average spatial load standard deviation to better show the load differences between all distribution networks. Specifically, the average spatial load standard deviation The calculation formula

[0118]

[0119] Where D(t) is the load standard deviation between distribution networks at time t.

[0120] Figure 5 and Figure 6 The performance of the average spatial load valley-peak ratio and the average spatial load standard deviation in the distribution network under different numbers of electric vehicles requesting charging is shown. As can be seen from these figures, the comparison between the greedy strategy and the DDQN-based strategy under different optimization functions shows that although the greedy strategy can achieve spatial load balancing to a certain extent, its effect is not as good as the DRL_Space strategy and the DRL_Both strategy. This is because the greedy strategy only selects charging stations for users based on the distribution network load when the user makes a charging request. However, when the user arrives at the charging station, the load of the station may no longer be the lowest among all distribution networks. In addition, the greedy strategy prioritizes the user's immediate charging needs, which, although it achieves spatial load balancing to a certain extent, fails to achieve the best effect.

[0121] In contrast, the DRL_Space strategy and the DRL_Both strategy perform better in spatial load balancing. Although the DDQN algorithm uses the same charging station allocation strategy as the greedy strategy when allocating charging stations to users, it optimizes the charging power selection based on the basic load conditions of the distribution network when the user arrives at the charging station. Under this charging power allocation strategy, the actions taken by the agent can make the load distribution of the distribution network more balanced.

[0122] On the other hand, the DRL_Cost strategy does not consider spatial load balancing in its charging strategy, so its load balancing effect is weaker than the other three strategies.

[0123] (2) Analysis of the charging price performance of the four algorithms through simulation results

[0124] from Figure 7 It can be seen that the greedy strategy leads to the highest user charging cost compared with the strategy based on the DDQN algorithm. This is because the greedy strategy satisfies the user's charging needs as quickly as possible, ignoring the user's flexibility during parking, which leads to an increase in charging costs.

[0125] In contrast, the DDQN-based strategy takes advantage of the user's flexibility and can effectively adjust the charging power according to different load conditions and electricity prices. Therefore, the user charging cost under the three DDQN algorithms is lower than that of the greedy strategy. However, different reward functions have different effects on the user charging cost. In the DRL_Space strategy, the agent does not consider the electricity price information, so it cannot adjust the charging power according to the electricity price of the charging station, resulting in its performance on the user charging cost being inferior to the other two DDQN algorithms that include cost considerations.

[0126] For the DRL_Both strategy, the design of its reward function includes the consideration of user charging cost, so the user charging cost is lower than that of the DRL_Space strategy. However, this strategy needs to strike a balance between minimizing the charging cost and the standard deviation of the space load, so its cost is higher than that of the DRL_Cost strategy.

[0127] In the case of the DRL_Cost strategy, the information input to the agent only includes the charging price for each time period. After multiple iterations, the agent will choose a charging strategy that minimizes the user's charging cost. Therefore, this reward function achieves the lowest user charging cost.

[0128] (3) Through simulation results analysis, the comprehensive performance of DRL_Cost, DRL_Space and DRL_Both strategies

[0129] like Figure 8 As shown in the figure, the normalized reward values ​​of the DRL_Space strategy and the DRL_Cost strategy are both 1. In the DRL_Cost strategy, the method selects the charging power that can minimize the charging price of electric vehicles, while ignoring the load distribution between power grids. In contrast, the DRL_Space strategy prioritizes the charging power that can balance the load between power grids, but does not consider the charging cost of users. These two strategies only focus on one aspect, which limits the overall performance of the algorithm.

[0130] However, under the DRL_Both strategy, the normalized reward value is the highest. This is because the strategy considers both the user charging cost and load balancing when selecting the charging power in each time period. After balancing these two factors, the actions selected by the strategy can both reduce the user charging cost and minimize the load gap between distribution networks. Therefore, this strategy achieves the highest reward value and the overall performance of the system is the most balanced.

[0131] from Fig. 9 and Fig.10 It can be seen that as the number of users participating in the scheduling increases, the SLVTPR gradually increases, while the standard deviation of the spatial load gradually decreases. This shows that the more users participating in the scheduling, the more balanced the spatial load.

[0132] In the case of low user engagement, only a small number of users choose the charging station with the lowest load, while most users randomly choose one of the four charging stations. In this case, the greedy strategy exhibits the worst spatial load balancing performance. This is because the greedy strategy starts charging immediately after the user arrives, and this method is highly dependent on the user's engagement. If the user does not accept the assignment of the charging station, the strategy may aggravate the load imbalance. In contrast, the DDQN-based charging strategy is generally better than the greedy strategy. This is because the DDQN strategy adjusts the charging power according to different objective functions, and the agent is still able to find the most suitable charging rate for the charging station even in the case of low user engagement.

[0133] As user participation increases, the greedy strategy outperforms the DRL_Cost strategy in spatial load balancing. This is because, under high user participation, most users will choose charging stations with lower loads when making requests, while the DRL_Cost strategy selects actions to reduce charging costs, but it does not consider the loads of other charging stations, so it is weaker than the greedy strategy, DRL_Space strategy, and DRL_Both strategy in spatial load balancing performance.

[0134] from Fig.11 It can be seen that as the number of users participating in the scheduling increases, the overall charging cost of users gradually decreases. This is because higher user participation makes users choose charging stations with lower loads, and the prices of these charging stations are also lower, thereby reducing the total charging cost. Under different user participation, it is consistent with the previous simulation results: the DRL_Cost strategy has the lowest charging cost, followed by the DRL_Both strategy, and then the DRL_Space strategy, while the greedy strategy has the highest cost.

[0135] (4) Through simulation results analysis, the performance of the four algorithms under different user participation levels

[0136] Finally, from Fig.12It can be seen that under different user participation levels, the DDQN-based algorithm still maintains the same rules as the previous simulation results under different optimization functions. The DRL_Both strategy achieves the best system performance, indicating that this weighted method enables the system to select the optimal charging strategy.

[0137] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A charging power allocation method for electric vehicle charging stations based on deep reinforcement learning, characterized in that: The steps include: Step 1: According to the location of the charging station, the initial power and location of the electric vehicle, a suitable charging station is selected for the user, thereby obtaining a set of candidate charging station solutions for the electric vehicle; Step 2: According to the spatial load balancing, the electric vehicles are allocated to the charging station with the lowest load, and the optimal charging station is given in combination with the time when the electric vehicles arrive at the charging station; Step 3: According to the information of the electric vehicles arriving at the charging station, the problem of allocating charging resources for the electric vehicles is converted into a Markov process problem. The information of the electric vehicles includes the charging demand and charging parking time of the electric vehicles. The Markov process problem includes solving the action value, state value and reward value. Step 4: Use the deep reinforcement learning algorithm to iteratively solve the Markov process problem, so that the final action value is used as the final power allocation method.

2. According to the method of charging power allocation of electric vehicle charging station based on deep reinforcement learning in claim 1, it is characterized in that: In step 1, the method for constructing a charging station candidate solution set is: First, according to the location and initial power of the electric vehicle, the distance matrix R of the user to each charging station is calculated. The definition of matrix R is as follows: Where R i,j represents the distance from the i-th vehicle to the j-th charging station, and N represents the total number of electric vehicles requesting charging; According to the initial battery charge state of the i-th electric vehicle and the maximum distance that an electric vehicle can travel R max , calculate the maximum distance R that can be traveled with the remaining power i , R i satisfy: Combined with the matrix R, we can get the candidate solution set Cand for the charging station of the electric vehicle. i , Cand i Should meet: in It is a collection of all charging stations that meet the listing.

3. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, the number of electric vehicles with the same number of charging time slots and expected parking time slots in each charging station and the current time t are counted as the state S of the Markov process problem. t , then S t It can be expressed as: in It represents the total number of electric vehicles in the charging station with a charging time slot number of c and an expected parking time slot number of e. max Represents the maximum charging time required for electric vehicles, E max It represents the maximum parking time of electric vehicles.

4. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 3 is characterized in that: The state S of the Markov process problem t The dimension calculation method is:

5. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 4 is characterized in that: In step 3, select the charging action a of the current charging station at the current moment. j,t As an action of the Markov process problem, a j,t It can be expressed as: in, represents the number of electric vehicles that choose to charge at the charging station at the current time t, The total number of electric vehicles at the current moment; and perform charging actions based on the user's urgency.

6. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 5 is characterized in that: The calculation method of the user urgency is: According to the user's charging needs i and charging parking time p i Calculate the number of charging time slots c of an electric vehicle in a fully charged state i,t and the estimated number of parking slots e i,t , the formula can be expressed as: Where ΔT is the time difference between two adjacent time slots, p is the charging power of the charging station, and the user's urgency is calculated based on these two data The calculation formula can be expressed as:

7. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 5 is characterized in that: In step 3, the optimization goal is to calculate the reward function that minimizes the user cost DRL_Cost The expression is as follows: where p j,t represents the charging price of the jth charging station at time t; Calculate the reward function that minimizes the load gap DRL_Space between charging stations The expression is as follows: in represents the base load of the j-th distribution network when there is no electric vehicle charging at time t, P t tot The total load of all distribution networks when there are no electric vehicles charging at time t.

8. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 7 is characterized in that: In step 3, the final Markov process problem is constructed as follows: because and The quantization levels of are different. Therefore, in order to optimize the user's charging cost and the load difference between charging stations, the two need to be normalized. The normalization formula is as follows: Among them, x t represents the value of the element at time t, x max Indicates the maximum value of the same type of elements, x min represents the minimum value of the same type of elements, then the reward function r of DRL_Both considering user cost and spatial load balancing j,t It can be calculated using the following formula: Among them, α and β represent the weight factors when the reward function is to minimize the user charging cost and the charging station load balance, respectively. It represents the normalized value of the price. It represents the value after normalizing the load gap; According to the state S t , action a j,t And different reward functions, calculate the Q value function Q under the current state-action j (s,a), the calculation formula is: Among them, E represents the expected function, γ represents the discount factor, and r j,t+k Represents the agent’s reward at a future time, Q j (s,a) represents the expected cumulative reward for taking action a in state s.

9. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 8, characterized in that: In step 4, a deep reinforcement learning algorithm is implemented by constructing two deep neural networks with the same structure, wherein one deep neural network is a training network and the other deep neural network is a target network.

10. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 9, characterized in that: The step 4 includes the following sub-steps: Step 4-1: Randomly initialize the parameters θ of the training network and initialize the parameters of the target network Make the parameters in the two networks the same; Step 4-2, initialize the number of training iterations episode = 0, initialize t = 0, and generate the initial state s0 of the current charging station according to the information of the electric vehicles in the charging station; Step 4-3: Select charging action a according to the current training network parameters and greedy strategy argmaxQ(s,a|θ) t , the strategy for selecting actions is as follows: Step 4-4. Calculate the instant reward r t , and obtain the Markov state s at the next moment based on the information of the electric vehicles that have just arrived at the charging station and the electric vehicles that have not yet completed the charging demand t+1 ; Step 4-5: Sample (s t, ,a t ,r t ,s t+1 ) into the playback buffer pool, and select a certain number of samples from the buffer pool for training, and update the network parameters; Step 4-6: Repeat steps 4-1 to 4-6 until the episode reaches the preset maximum value, and set the action a of the episode to episode As the final power distribution method.

11. The method for allocating charging power of an electric vehicle charging station based on deep reinforcement learning according to claim 10, characterized in that: In the steps 4-5, the method for updating the network parameters is: L(θ)=E[(q target -Q(s,a|θ)) 2 ] where q target The target Q value, s' represents the next state, a' represents all possible actions in the next state, μ is the learning rate, is the gradient operator.

Images