Electric vehicle charging and discharging decision-making method based on mixed action space

Abstract

The invention provides an electric vehicle charging and discharging decision-making method based on a hybrid action space, and relates to the technical field of electric vehicles. The method specifically comprises the following steps: constructing a multi-agent charging system comprising a plurality of electric vehicle charging models; respectively constructing a state space, a mixed action space and a reward function; a mixed action space multi-agent soft action-comment algorithm is adopted to construct a multi-agent reinforcement learning framework for the multi-agent charging system, and initialization is carried out; the state of the electric vehicle at the moment t is obtained, and then the initialized multi-agent reinforcement learning framework is used for obtaining the continuous action selected by the electric vehicle at the moment t, the instantaneous reward of the electric vehicle at the moment t and the state of the electric vehicle at the moment t + 1, and the continuous action, the instantaneous reward and the state are put into an experience pool; and randomly extracting a data sample from the experience pool and inputting the data sample into the multi-agent reinforcement learning framework to train network parameters to obtain a trained multi-agent reinforcement learning framework for solving the charging and discharging decision problem of the electric vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of electric vehicles, and in particular to a charging and discharging decision method for electric vehicles based on a hybrid action space. Background Art

[0002] Electric vehicles are battery-powered vehicles. Compared with traditional internal combustion engine vehicles, electric vehicles have important advantages in environmental protection, energy utilization, economy and technological development.

[0003] The surge in the number of electric vehicles has brought new challenges to the strengthening of charging facilities. However, the existing public charging infrastructure is far from meeting the existing charging needs, and a large number of users still choose home charging piles for charging. The exchange of electric energy between electric vehicles and the power grid involves two modes: Grid-to-Vehicle (G2V) and Vehicle-to-Grid (V2G). The former is a charging mode that obtains electric energy from the grid, while the latter is a discharge mode that transmits electric energy to the grid. Reasonable control of the charging and discharging power of electric vehicles can bring certain economic benefits to both users and the power grid. Therefore, it is very necessary to implement charging management for electric vehicles.

[0004] In the problem of electric vehicle charging management, the randomness of user behavior includes variables such as charging start time, charging end time and battery status. This randomness increases the uncertainty of the electric vehicle charging management system and the difficulty of solving the problem. In addition, the problem of electric vehicle charging management is difficult to describe with an accurate mathematical model, so it is difficult to solve it with traditional optimization methods. For sequential decision-making problems in uncertain environments that are difficult to model, reinforcement learning algorithms will become a very effective solution, but existing studies have not considered the impact of traffic congestion on the charging cycle, and traditional reinforcement learning algorithms only control the charging power of electric vehicles and are not suitable for discrete-continuous hybrid control problems. Summary of the invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention proposes an electric vehicle charging and discharging decision-making method based on hybrid action space based on multi-agent reinforcement learning, which can realize discrete control of the charging end time of multiple electric vehicles under the same transformer and continuous control of the charging power at the same time.

[0006] The present invention proposes a charging and discharging decision method for an electric vehicle based on a hybrid action space, the method comprising the following steps:

[0007] Step 1: Consider an electric vehicle as an intelligent agent and construct a multi-agent charging system including several electric vehicle charging models;

[0008] Step 2: Construct state space;

[0009] Step 3: Construct a mixed action space;

[0010] Step 4: Construct a reward function;

[0011] Step 5: Use the hybrid action space multi-agent soft action-criticism HAS-MASAC algorithm to build a multi-agent reinforcement learning framework for the multi-agent charging system and initialize it;

[0012] Step 6: Obtain the state of the electric car at time t, and then use the initialized multi-agent reinforcement learning framework to obtain the continuous actions selected by the electric car at time t, the instantaneous reward of the electric car at time t, and the state of the electric car at time t+1, and put the above data into the experience pool D;

[0013] Step 7: Randomly extract data samples from the experience pool D, and input the extracted data samples into the multi-agent reinforcement learning framework to train the network parameters in the multi-agent reinforcement learning framework to obtain a trained multi-agent reinforcement learning framework;

[0014] Step 8: Call the trained multi-agent reinforcement learning framework to solve the charging and discharging decision-making problem of electric vehicles;

[0015] The electric vehicle charging model described in step 1 includes: electric vehicle charging behavior selection and charging end time behavior selection;

[0016] The electric vehicle charging behavior selection is: selecting a charging mode from the grid to the vehicle G2V or a discharging mode from the vehicle to the grid V2G according to the electricity price and the electric vehicle battery power, and continuously controlling the charging power or discharging power of the electric vehicle;

[0017] The charging mode is to obtain electricity from the power grid to supplement the battery power of the electric vehicle; the discharging mode is for the electric vehicle to gain profit by feeding back electric energy to the power grid;

[0018] The charging end time behavior is selected as follows: a charging cycle includes several charging time slots, and the total profit obtained by the electric vehicle in a charging cycle is calculated. The total profit is the difference between the profit obtained by the electric vehicle in the discharge mode and the cost incurred by the electric vehicle in the charging mode. When the total profit is maximized in the charging cycle, the charging action of the electric vehicle is terminated;

[0019] The state space in step 2 is: assuming that the state space includes the states of N electric vehicles, for any electric vehicle i, the state of the electric vehicle i at time t includes: environmental state and electric vehicle state; the environmental state includes: the real-time electricity price at time t, the historical electricity price in the past several hours based on time t, and the INRIX index calculated based on the traffic congestion situation in the area where the electric vehicle i is located at time t; the electric vehicle state includes: the state of charge SoC of the electric vehicle i at time t and the expected remaining charging time of the electric vehicle i, and the expected remaining charging time is the difference between the expected charging end time of the electric vehicle i and the current time t;

[0020] The state of electric vehicle i at time t is represented by a vector:

[0021]

[0022] in represents the environmental state of electric vehicle i at time t; p t represents the real-time electricity price at time t; p t-23 It represents the historical electricity price in the past 23 hours based on time t; represents the state of charge of electric vehicle i at time t; I represents the expected remaining charging time of electric vehicle i at time t; t represents the INRIX index of the region where electric vehicle i is located at time t;

[0023] The construction method of the hybrid action space in step 3 is as follows: Assuming that the hybrid action space includes N actions selected by electric vehicles, a discrete action space K is defined to describe whether the electric vehicle has finished charging at the current moment, which is denoted as K = {0, 1}, where 1 indicates that the electric vehicle has finished charging at the current moment; 0 indicates that the electric vehicle has not finished charging at the current moment; for any electric vehicle i, the action selected by electric vehicle i at time t is expressed as:

[0024]

[0025] in The action chosen by electric car i at time t; is the discrete action of electric vehicle i at time t, which is used to represent the charging state of electric vehicle i at time t, and in Indicates that the electric vehicle ends the current charging round; otherwise, it indicates that the electric vehicle continues the current charging round; is the continuous action of electric vehicle i at time t, which is used to represent the charging and discharging power of electric vehicle i at time t, that is, As a continuous variable, where cmax It is the maximum charge and discharge power boundary of the charging equipment;

[0026] The reward function in step 4 is expressed as:

[0027]

[0028] where r i t represents the instantaneous reward value of electric vehicle i at time t; PO is the overload penalty, which is used to coordinate the charging power of multiple electric vehicles to prevent overload; AC i is the level of anxiety caused by traffic congestion to electric vehicles; AT i The time anxiety is caused by the deviation between the actual charging end time of the electric vehicle i and the expected charging end time; AR i It is the range anxiety of electric vehicles; PC i is the charging cost of electric vehicle i at time t; soc max Indicates the maximum battery capacity of an electric vehicle; k po Indicates the transformer overload penalty coefficient; po max Indicates the maximum capacity of the transformer; is a parameter used to reflect the degree of concern of electric vehicle i to electricity prices; is a parameter used to reflect the degree of concern of electric vehicle i about traffic congestion; is a parameter used to reflect the degree of concern of electric vehicle i to time deviation; is a parameter used to reflect the degree of concern of electric vehicles i about range anxiety;

[0029] The multi-agent reinforcement learning framework in step 5 is: for N electric vehicles in the multi-agent charging system, a reinforcement learning network is constructed for each electric vehicle, and the multi-agent reinforcement learning framework is composed of N reinforcement learning networks; the reinforcement learning network is composed of an Actor network, a first Critic network, a second Critic network, a first target Critic network, and a second target Critic network;

[0030] The method for initializing the multi-agent reinforcement learning framework is: respectively initializing N Actor networks in the multi-agent reinforcement learning framework N first critic networks N second critic networks N first target critic networks and N second target critic networks Where i = 1, 2, ..., N; θ represents the network parameters of the Actor network; φ represents the network parameters of the Critic network; Represents the network parameters of the target Critic network;

[0031] The step 6 further comprises:

[0032] Step 6.1: Get the state of electric car i at time t and the state of electric car i at time t+1 And the status Enter the initialized Actor network The continuous action of electric car i at time t is obtained Recorded as

[0033] Step 6.2: Construct the connected action space of the multi-agent charging system based on the continuous actions of all electric vehicles at time t, denoted as in represents the continuous action of the first electric car at time t; represents the continuous action of the Nth electric car at time t;

[0034] Step 6.3: Change the status After concatenating the action space with the connected action space, we input it into the initialized first Critic network to get K n The Q value of a discrete action is denoted by

[0035] Step 6.4: According to K n The Q value of a discrete action is used to select the discrete action of electric car i at time t using a greedy strategy

[0036] Step 6.5: Utilizing Discrete Actions of EV i and continuous action Get the action chosen by electric car i at time t And the action Interact with the environment to obtain the instantaneous reward value r of electric car i at time t t i ;

[0037] Step 6.6: Repeat steps 6.1-6.5 to obtain the state s of all electric vehicles at time t t , continuous action c t , instantaneous reward value r t and the state s at time t+1 t+1 And put it into the experience pool D as a set of data samples;

[0038] Step 6.7: Repeat steps 6.1-6.6 until a preset number of data samples are obtained and put into the experience pool D;

[0039] The step 7 further comprises:

[0040] Step 7.1: Randomly extract data samples of group B at different times from the experience pool D and input them into the multi-agent reinforcement learning framework;

[0041] Step 7.2: For the reinforcement learning network of electric vehicle i in the multi-agent reinforcement learning framework, the network parameters of the two critic networks in the reinforcement learning network are updated using the temporal difference method;

[0042] Step 7.3: Input the states and continuous actions in the data sample into the two updated Critic networks to obtain K n The Q value of each discrete action is obtained, and the Q values ​​obtained by the two Critic networks are summed up respectively, and then the smallest of the two summation results is selected as the corresponding Q value of the continuous action;

[0043] Step 7.4: According to the corresponding Q value of the continuous action, the maximum entropy method is used to calculate the loss function value of the Actor network and update the network parameters of the Actor network;

[0044] Step 7.5: Use the soft update method to update the network parameters of the two target critic networks;

[0045] Step 7.6: Repeat steps 7.1-7.5 to iteratively train the multi-agent reinforcement learning framework until the preset training rounds are completed, and the trained multi-agent reinforcement learning framework is obtained and saved;

[0046] The update of the network parameters of the two Critic networks described in step 7.2 is implemented using the loss function of the Critic network, which is expressed as:

[0047]

[0048] in represents the loss function used to update the Critic network; j represents the serial number of the Critic network, and j∈{1,2}, that is, when j=1, represents the first Critic network; when j = 2, represents the second Critic network; ω is the network parameter of the Critic network; Indicates input status and continuous action The Q value output by the post-Critic network; Indicates extracting data samples from the experience pool D and calculating the Q value output by the Critic network and the true Q value y t The expected value of the mean square error; where the true Q value y t The calculation process is as follows:

[0049]

[0050] Where γ represents the discount factor; represents the maximum Q value output by the Critic network at time t+1; α represents the temperature coefficient; represents the logarithmic action probability;

[0051] The expression of the corresponding Q value of the continuous action described in step 7.3 is:

[0052]

[0053] Where Q(s t ,c t ) indicates continuous action c t The corresponding Q value; k represents the discrete action that maximizes the Q value.

[0054] The beneficial effects of adopting the above technical solution are:

[0055] The method of the present invention proposes a charging management problem of an electric vehicle under a dynamic charging cycle with a variable charging end time under the influence of traffic congestion, and at the same time, discrete control is performed on the charging end time of the electric vehicle and continuous control is performed on the charging power of the electric vehicle. The method of the present invention reduces the total cost of electric vehicle users by controlling the charging power and charging end time of the electric vehicle respectively.

[0056] The method of the present invention takes into account the uncertainty of real-time electricity prices and the randomness of user behavior, and on this basis, proposes a dynamic charging cycle with a variable charging end time, designs an electric vehicle charging management method under a dynamic charging cycle, and guides the charging and discharging decisions of electric vehicles by constructing a discrete hybrid action space and introducing traffic congestion anxiety into the reward function. The multi-agent model of the hybrid action space constructed by the method of the present invention adopts an improved Critic network structure to improve the original network output layer. The improved Critic network can select discrete actions and calculate the value of continuous actions at the same time, thereby solving the EV charging management problem under the hybrid action space.

[0057] Under the same conditions, the multi-agent reinforcement learning algorithm in the hybrid action space proposed in the method of the present invention reduces the total cost including charging cost and traffic congestion anxiety by about 9.83% in total compared with the MASAC algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a method for charging and discharging decision of an electric vehicle based on a hybrid action space in this embodiment;

[0059] Figure 2is a schematic diagram of a charging and discharging decision method for an electric vehicle based on a hybrid action space in this embodiment;

[0060] Figure 3 It is a structural diagram of the Actor network in the reinforcement learning network of the intelligent agent in this implementation mode;

[0061] Figure 4 is a structural diagram of the first Critic network in the reinforcement learning network of the intelligent agent in this embodiment;

[0062] Figure 5 is a structural diagram of the second Critic network in the reinforcement learning network of the agent in this embodiment;

[0063] Figure 6 It is a structural diagram of the first target Critic network in the reinforcement learning network of the intelligent agent in this embodiment;

[0064] Figure 7 Schematic diagram of the second target Critic network in the reinforcement learning network of the intelligent agent in this embodiment. DETAILED DESCRIPTION

[0065] In order to facilitate the understanding of the present application, the specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thoroughly understood.

[0066] In this embodiment, a charging and discharging decision method for electric vehicles based on a hybrid action space is provided. Figure 1 As shown, the method comprises the following steps:

[0067] Step 1: Consider an electric vehicle as an intelligent agent and construct a multi-agent charging system including several electric vehicle charging models;

[0068] In this embodiment, if Figure 2 As shown in the figure, each electric car is an agent with a clear state space, action space and reward function. The overall scheme is divided into the environment and multi-agent reinforcement learning parts, in which the agent interacts with the environment and obtains state information from the environment, including electric vehicle state information, electricity price information, traffic information, etc. In addition to obtaining state information, the environment will feedback instantaneous rewards after the agent interacts and calculate the state at the next moment. The reinforcement learning part includes multiple agent structures, each of which contains multiple networks, which are used to take actions on the current state and approximate the value function, thereby interacting with the environment.

[0069] The electric vehicle charging model includes: electric vehicle charging behavior selection and charging end time behavior selection.

[0070] The electric vehicle charging behavior selection is: selecting a charging mode from the grid to the vehicle (Grid-to-Vehicle, G2V) or a discharging mode from the vehicle to the grid (Vehicle-to-Grid, V2G) according to the electricity price and the electric vehicle battery power, and continuously controlling the charging power or discharging power of the electric vehicle.

[0071] The charging mode is to obtain electricity from the power grid to supplement the battery power of the electric vehicle; the discharging mode is for the electric vehicle to gain profit by feeding back electric energy to the power grid.

[0072] In this embodiment, from the principle analysis, the power quantity and electricity price are comprehensively considered, among which the electricity price affects the charging cost and the discharging profit, so an electricity price threshold is set in advance. When the real-time electricity price is lower than the electricity price threshold, the charging mode from the power grid to the vehicle G2V is selected, that is, the charging mode is selected at low electricity prices; when the real-time electricity price is higher than the electricity price threshold, the discharging mode from the vehicle to the power grid V2G is selected, that is, the discharging mode is selected at high electricity prices, thereby obtaining certain benefits; and the power quantity provides restrictions on charging and discharging, and users expect to retain a higher power quantity at the end of charging to meet subsequent travel needs.

[0073] The charging end time behavior selection is: a charging cycle includes several charging time slots, and the total profit obtained by the electric vehicle in a charging cycle is calculated. The total profit is the difference between the profit obtained by the electric vehicle in the discharge mode and the cost generated by the electric vehicle in the charging mode. When the total profit is maximized in the charging cycle, the charging action of the electric vehicle is terminated; the charging end time behavior selection is a discrete control.

[0074] In this embodiment, it is assumed that the arrival time of a certain electric vehicle, that is, the charging start time, is , and the departure time, that is, the charging end time, is . The traditional electric vehicle charging management decision problem focuses on fixed charging time periods. Considering that the traffic congestion when the user travels affects the travel experience, the choice of charging end behavior will be affected by the traffic conditions, resulting in changes in the charging cycle. Therefore, the above problem is transformed into a charging management problem with a dynamic charging cycle, and its goal is to make the best choice of charging in each time slot to maximize the user's total profit. Each charging cycle is a continuous charging segment, with one hour as a time slot. Each charging cycle often contains several or more than a dozen charging time slots, so the total profit is the difference between the profit obtained by the electric vehicle in the discharge mode and the cost incurred by the electric vehicle in the charging mode in all time slots.

[0075] Step 2: Construct the state space.

[0076] In this embodiment, the state is generated by the environment, and the electric vehicle, i.e., the agent, selects actions according to the corresponding state.

[0077] The state space is: assuming that the state space includes the states of N electric vehicles, for any electric vehicle i, the state of the electric vehicle i at time t includes: environmental state and electric vehicle state; the environmental state includes: the real-time electricity price at time t, the historical electricity price in the past several hours based on time t, and the INRIX index calculated based on the traffic congestion situation in the area where the electric vehicle i is located at time t; the electric vehicle state includes: the state of charge (SoC) of the electric vehicle i at time t and the expected remaining charging time of the electric vehicle i, and the expected remaining charging time is the difference between the expected charging end time of the electric vehicle i and the current time t.

[0078] The state of electric vehicle i at time t is represented by a vector:

[0079]

[0080] in represents the environmental state of electric vehicle i at time t; p t represents the real-time electricity price at time t; p t-23 It represents the historical electricity price in the past 23 hours based on time t; represents the state of charge of electric vehicle i at time t; I represents the expected remaining charging time of electric vehicle i at time t; t Represents the INRIX index of the region where electric vehicle i is located at time t.

[0081] In this embodiment, if Figure 2 As shown, considering the periodicity of electricity prices, the electricity prices of the past 24 moments are selected. In addition, considering the impact of traffic congestion on user travel patterns, the index for quantifying traffic congestion released by INRIX is introduced to quantitatively evaluate traffic conditions, namely the INRIX index. This index is a traffic congestion evaluation index that quantifies congestion status based on road speed and road attributes. The INRIX index is a typical congestion indicator based on road speed. The specific calculation method is: assuming that the historical electricity price data for one year and the real-time traffic speed of all sections in a target area are obtained, the historical electricity price data and the real-time traffic speed are both in hours. The historical electricity price data of the first 20 days of each month and the real-time traffic speed of all sections in a target area are taken as the training set, and the rest are the test set; the INRIX index of the target area at time n is calculated as:

[0082]

[0083] Among them A mn Represents the INRIX index of road section m at time n; RS mn represents the free speed of road section m at time n; CS mn represents the actual vehicle speed on road section m at time n; B n represents the INRIX index of the target area at time n; L m represents the length of road segment m; M represents the number of road segments in the target area.

[0084] In this embodiment, the traffic congestion situation of the area where electric vehicle i is located at time t is substituted into formula (2) and formula (3) to calculate the INRIX index of the area where electric vehicle i is located at time t. The travel patterns of electric vehicle users are collected and sampled to obtain the initial electric vehicle status of all electric vehicles. in Indicates the charge state of the first electric vehicle at the initial moment; Indicates the expected remaining charging time of the first electric vehicle at the initial moment; represents the state of charge of the i-th electric vehicle at the initial moment; represents the expected remaining charging time of the i-th electric vehicle at the initial moment; Indicates the state of charge of the Nth electric vehicle at the initial moment; represents the expected remaining charging time of the Nth electric vehicle at the initial moment; extract the environmental states of all electric vehicles from the test set, and concatenate the initial electric vehicle state and environmental state of each electric vehicle into the form of formula (1) and put them into the state space.

[0085] Step 3: Construct a mixed action space;

[0086] In this implementation, a suggestion on adjusting the charging end time is proposed, and an electric vehicle charging problem under a dynamic charging duration considering a variable charging end time is constructed, and the above problem is described as a Markov decision process with a continuous-discrete mixed action space.

[0087] The construction method of the hybrid action space is as follows: Assuming that the hybrid action space includes N actions selected by electric vehicles, a discrete action space K is defined to describe whether the electric vehicle has finished charging at the current moment, which is recorded as K={0,1}, where 1 indicates that the electric vehicle has finished charging at the current moment; 0 indicates that the electric vehicle has not finished charging at the current moment; for any electric vehicle i, the action selected by electric vehicle i at time t is expressed as:

[0088]

[0089] in The action chosen by electric car i at time t; is the discrete action of electric vehicle i at time t, which is used to represent the charging state of electric vehicle i at time t, and in Indicates that the electric vehicle ends the current charging round; otherwise, it indicates that the electric vehicle continues the current charging round; is the continuous action of electric vehicle i at time t, which is used to represent the charging and discharging power of electric vehicle i at time t, that is, As a continuous variable, where c max It is the maximum charge and discharge power limit of the charging equipment.

[0090] Step 4: Construct a reward function;

[0091] The reward function is expressed as:

[0092]

[0093] where r t i represents the instantaneous reward value of electric vehicle i at time t; PO is the overload penalty, which is used to coordinate the charging power of multiple electric vehicles to prevent overload; AC i is the level of anxiety caused by traffic congestion to electric vehicles; AT i The time anxiety is caused by the deviation between the actual charging end time of the electric vehicle i and the expected charging end time; AR i It is the range anxiety of electric vehicles; PC i is the charging cost of electric vehicle i at time t; soc max Indicates the maximum battery capacity of an electric vehicle; k po Indicates the transformer overload penalty coefficient; po max Indicates the maximum capacity of the transformer; is a parameter used to reflect the degree of concern of electric vehicle i to electricity prices; is a parameter used to reflect the degree of concern of electric vehicle i about traffic congestion; is a parameter used to reflect the degree of concern of electric vehicle i to time deviation; It is a parameter used to reflect the degree of concern of electric vehicles about range anxiety.

[0094] The reward function shows that the technical solution proposed in this embodiment aims to balance the charging cost and various anxieties, that is, if a larger parameter k is used in the training process to reflect the degree of concern of electric vehicles to electricity prices, c , users will pay more attention to charging costs. If users adjust the charging end time t dThe tolerance for time deviation is low, while the tolerance for anxiety related to traffic congestion is high, which can be achieved by setting a larger parameter k that reflects the degree of concern of electric vehicles for time deviation. m and a smaller parameter k that reflects the concern of electric vehicles about traffic congestion x To ensure a higher SoC at the end of the charging process, the parameter k that reflects the degree of concern about range anxiety in electric vehicles should be set appropriately. p The above parameters can be adjusted according to different user needs.

[0095] Step 5: Use the Hybrid Action Space MultiAgent Soft Actor-Critic (HAS-MASAC) algorithm to build a multi-agent reinforcement learning framework for the multi-agent charging system and initialize it.

[0096] In this embodiment, due to the existence of hybrid action space, traditional multi-agent reinforcement learning algorithms such as Multi Agent Soft Actor-Critic (MASAC) algorithm are difficult to solve. Therefore, this embodiment adopts the improved MASAC algorithm, that is, the Hybrid ActionSpace Multi Agent Soft Actor-Critic (HAS-MASAC) algorithm to construct a reinforcement learning network structure, and improves the Critic network structure. The number of neurons output by the improved Critic network corresponds to the number of elements in the discrete action space K, so as to obtain the Q value under different discrete actions. Discrete actions are selected according to the greedy strategy, and the continuous action value output by the Actor network and the discrete action after the decision of the Critic network are combined to form the action of the agent in each iteration round, and the obtained action interacts with the environment, and the next state and reward value of the agent are calculated, and then the next iteration is entered until the end of the round.

[0097] The multi-agent reinforcement learning framework is as follows: for N electric vehicles in a multi-agent charging system, a reinforcement learning network is constructed for each electric vehicle, and the multi-agent reinforcement learning framework is composed of N reinforcement learning networks; the reinforcement learning network is composed of an Actor network, a first Critic network, a second Critic network, a first target Critic network, and a second target Critic network.

[0098] In this embodiment, if Figure 3-Figure 7As shown in the figure, each agent contains an Actor network, two Critic networks and two target Critic networks. The structures of the Critic network and the target Critic network are the same, and the parameters of the target Critic network are obtained by soft updating the parameters of the corresponding Critic network. Each network contains three hidden layers.

[0099] The method for initializing the multi-agent reinforcement learning framework is: respectively initializing N Actor networks in the multi-agent reinforcement learning framework N first critic networks N second critic networks N first target critic networks and N second target critic networks Where θ represents the network parameters of the Actor network; φ represents the network parameters of the Critic network; Represents the network parameters of the target Critic network.

[0100] In this embodiment, the Actor network corresponds to the strategy of continuous action, which is used to calculate the charging and discharging power of the electric vehicle according to the state input of the intelligent agent; the Critic network calculates the Q value of the discrete action according to the network input, and takes the action corresponding to the maximum Q value as the discrete action, thereby obtaining a complete hybrid action space electric vehicle charging and discharging decision-making process.

[0101] Step 6: Obtain the state of the electric car at time t, and then use the initialized multi-agent reinforcement learning framework to obtain the continuous actions selected by the electric car at time t, the instantaneous reward of the electric car at time t, and the state of the electric car at time t+1, and put the above data into the experience pool D;

[0102] Step 6.1: Get the state of electric car i at time t and the state of electric car i at time t+1 And the status Enter the initialized Actor network The continuous action of electric car i at time t is obtained Recorded as

[0103] In this embodiment, if Figure 3 As shown, the state of electric vehicle i at time t is input into the Actor network index of electric vehicle i after initialization, and the continuous action of electric vehicle i at time t is obtained.

[0104] Step 6.2: Construct the connected action space of the multi-agent charging system based on the continuous actions of all electric vehicles at time t, denoted as in represents the continuous action of the first electric car at time t; Represents the continuous actions of the Nth electric car at time t.

[0105] Step 6.3: Change the status After concatenating the action space with the connected action space, we input it into the initialized first Critic network to get K n The Q value of a discrete action is denoted by

[0106] In this embodiment, if Figure 4 As shown, the status The first critic network initialized after concatenating the action space with the connected action space is input. The number of neurons output by the first critic network corresponds to the number of elements in the discrete action space K. After concatenating the action space with the connection space, the Q values ​​of the two discrete actions are obtained by inputting them into the Critic1 network.

[0107] Step 6.4: According to K n The Q value of a discrete action is used to select the discrete action of electric car i at time t using a greedy strategy

[0108] The discrete actions It is expressed as:

[0109]

[0110] in Indicates that from K n The discrete action corresponding to the maximum Q value is selected from the Q values ​​of discrete actions; k represents the discrete action that maximizes the Q value; ε is the probability of the greedy strategy, which is used to indicate that there is a probability of ε to randomly select a discrete action in the discrete set, and the remaining probability of 1-ε is to select the discrete action corresponding to the maximum Q value.

[0111] In this embodiment, the greedy strategy refers to selecting the discrete action with the maximum Q value with a certain probability, otherwise it is randomly selected.

[0112] Step 6.5: Utilizing Discrete Actions of EV i and continuous action Get the action chosen by electric car i at time t And the action Interact with the environment to obtain the instantaneous reward value r of electric car i at time t t i .

[0113] Step 6.6: Repeat steps 6.1-6.5 to obtain the state s of all electric vehicles at time t t, continuous action c t , instantaneous reward value r t and the state s at time t+1 t+1 And put it into the experience pool D as a set of data samples.

[0114] In this embodiment, the state s of all electric vehicles at time t is t , continuous action c t , instantaneous reward value r t and the state s at time t t+1 Put it into the experience pool D, recorded as t ,c t ,r t ,s t+1 >.

[0115] Step 6.7: Repeat steps 6.1-6.6 until a preset number of data samples are obtained and placed into the experience pool D.

[0116] In this embodiment, data samples corresponding to different moments are obtained and put into the experience pool D.

[0117] Step 7: Randomly extract data samples from the experience pool D, and input the extracted data samples into the multi-agent reinforcement learning framework, train the network parameters in the multi-agent reinforcement learning framework, and obtain a trained multi-agent reinforcement learning framework.

[0118] Step 7.1: Randomly extract data samples of group B at different times from the experience pool D and input them into the multi-agent reinforcement learning framework.

[0119] In this embodiment, data is extracted from the experience pool D for updating the network parameters of the multi-agent reinforcement learning framework.

[0120] Step 7.2: For the reinforcement learning network of electric vehicle i in the multi-agent reinforcement learning framework, the temporal-difference (TD) method is used to update the network parameters of the two critic networks in the reinforcement learning network.

[0121] The update of the network parameters of the two Critic networks is achieved by using the loss function of the Critic network, which is expressed as:

[0122]

[0123] in represents the loss function used to update the Critic network; j represents the serial number of the Critic network, and j∈{1,2}, that is, when j=1, represents the first Critic network; when j = 2, ​represents the second Critic network; ω is the network parameter of the Critic network; Indicates input status and continuous action The Q value output by the post-Critic network; Indicates extracting data samples from the experience pool D and calculating the Q value output by the Critic network and the true Q value y t The expected value of the mean square error; where the true Q value y t The calculation process is as follows:

[0124]

[0125] Where γ represents the discount factor; represents the maximum Q value output by the Critic network at time t+1; α represents the temperature coefficient; represents the logarithmic action probability.

[0126] Step 7.3: Input the states and continuous actions in the data sample into the two updated Critic networks to obtain K n The Q value of a discrete action is obtained, and the Q values ​​obtained by the two Critic networks are summed up respectively, and then the smallest of the two summation results is selected as the corresponding Q value of the continuous action;

[0127]

[0128] Where Q(s t ,c t ) indicates continuous action c t The corresponding Q value of .

[0129] In this embodiment, if Figure 4 and Figure 5 As shown in the figure, Critic 1 is used to select discrete actions, and Critic 2 is used to evaluate the value. The significance of using two Critic networks is to avoid overestimation of the Q value.

[0130] Step 7.4: According to the corresponding Q value of the continuous action, the maximum entropy method is used to calculate the loss function value of the Actor network and update the network parameters of the Actor network;

[0131] The loss function value of the Actor network is:

[0132]

[0133] Where L π (θ) represents the loss function value of the Actor network; θ represents the network parameters of the Actor network; Indicates that the Actor network is in the input state Output continuous action probability; Indicates extracting data samples from the experience pool D and calculating expected value.

[0134] Step 7.5: Use the soft update method to update the network parameters of the two target critic networks;

[0135] The update formula of the first target Critic network is:

[0136]

[0137] in represents the updated network parameters of the first target Critic network; τ represents the soft update coefficient; Represents the network parameters of the first Critic network; Represents the network parameters of the first target Critic network before updating.

[0138] The update formula of the second target Critic network is:

[0139]

[0140] in Represents the updated network parameters of the second target Critic network; Represents the network parameters of the second Critic network; Represents the network parameters of the second target Critic network before updating.

[0141] In this embodiment, if Figure 6 and Figure 7 As shown, the network parameters of the first target critic network and the second target critic network are updated respectively using the network parameters of the two critic networks.

[0142] Step 7.6: Repeat steps 7.1-7.5 to iteratively train the multi-agent reinforcement learning framework until the preset training rounds are completed, obtain the trained multi-agent reinforcement learning framework and save it.

[0143] In this embodiment, the training rounds are set to 2000 times, the discount factor is 0.995, the soft update coefficient = 0.005, the size of the experience pool sampling is 128, the learning rate of the Actor network is 0.005, the learning rate of the Critic network is 0.001, and the number of hidden layer neurons is 256. After all the training rounds are completed, the trained Actor network and Critic network are saved, where the Actor network corresponds to the strategy of continuous action and can calculate the charge and discharge power according to the state input of the intelligent body; the Critic network calculates the value of the discrete action according to the network input, and the action corresponding to the maximum value is used as the discrete action; thus, a complete hybrid action space electric vehicle charging and discharging decision process is obtained.

[0144] Step 8: Call the trained multi-agent reinforcement learning framework to solve the charging and discharging decision-making problem of electric vehicles.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A charging and discharging decision method for electric vehicles based on hybrid action space, characterized in that: The method comprises the following steps: Step 1: Consider an electric vehicle as an intelligent agent and construct a multi-agent charging system including several electric vehicle charging models; Step 2: Construct state space; Step 3: Construct a mixed action space; Step 4: Construct a reward function; Step 5: Use the hybrid action space multi-agent soft action-criticism HAS-MASAC algorithm to build a multi-agent reinforcement learning framework for the multi-agent charging system and initialize it; Step 6: Obtain the state of the electric car at time t, and then use the initialized multi-agent reinforcement learning framework to obtain the continuous actions selected by the electric car at time t, the instantaneous reward of the electric car at time t, and the state of the electric car at time t+1, and put the above data into the experience pool D; Step 7: Randomly extract data samples from the experience pool D, and input the extracted data samples into the multi-agent reinforcement learning framework to train the network parameters in the multi-agent reinforcement learning framework to obtain a trained multi-agent reinforcement learning framework; Step 8: Call the trained multi-agent reinforcement learning framework to solve the charging and discharging decision-making problem of electric vehicles.

2. According to claim 1, a method for charging and discharging decision of electric vehicles based on hybrid action space is characterized in that: The electric vehicle charging model described in step 1 includes: electric vehicle charging behavior selection and charging end time behavior selection; The electric vehicle charging behavior selection is: selecting a charging mode from the grid to the vehicle G2V or a discharging mode from the vehicle to the grid V2G according to the electricity price and the electric vehicle battery power, and continuously controlling the charging power or discharging power of the electric vehicle; The charging mode is to obtain electricity from the power grid to supplement the battery power of the electric vehicle; the discharging mode is for the electric vehicle to gain profit by feeding back electric energy to the power grid; The charging end time behavior is selected as follows: a charging cycle includes several charging time slots, and the total profit obtained by the electric vehicle in a charging cycle is calculated. The total profit is the difference between the profit obtained by the electric vehicle in the discharge mode and the cost incurred by the electric vehicle in the charging mode. When the total profit is maximized in the charging cycle, the charging action of the electric vehicle is terminated.

3. According to claim 1, a method for charging and discharging electric vehicles based on a hybrid action space is characterized in that: The state space in step 2 is: assuming that the state space includes the states of N electric vehicles, for any electric vehicle i, the state of the electric vehicle i at time t includes: environmental state and electric vehicle state; the environmental state includes: the real-time electricity price at time t, the historical electricity price in the past several hours based on time t, and the INRIX index calculated based on the traffic congestion situation in the area where the electric vehicle i is located at time t; the electric vehicle state includes: the state of charge SoC of the electric vehicle i at time t and the expected remaining charging time of the electric vehicle i, and the expected remaining charging time is the difference between the expected charging end time of the electric vehicle i and the current time t; The state of electric vehicle i at time t is represented by a vector: in represents the environmental state of electric vehicle i at time t; p t represents the real-time electricity price at time t; p t-23 It represents the historical electricity price in the past 23 hours based on time t; represents the state of charge of electric vehicle i at time t; I represents the expected remaining charging time of electric vehicle i at time t; t Represents the INRIX index of the region where electric vehicle i is located at time t.

4. According to claim 3, a method for charging and discharging decision of electric vehicles based on hybrid action space is characterized in that: The construction method of the hybrid action space in step 3 is as follows: Assuming that the hybrid action space includes N actions selected by electric vehicles, a discrete action space K is defined to describe whether the electric vehicle has finished charging at the current moment, which is denoted as K = {0, 1}, where 1 indicates that the electric vehicle has finished charging at the current moment; 0 indicates that the electric vehicle has not finished charging at the current moment; for any electric vehicle i, the action selected by electric vehicle i at time t is expressed as: in is the action chosen by electric car i at time t; is the discrete action of electric vehicle i at time t, which is used to represent the charging state of electric vehicle i at time t, and in Indicates that the electric vehicle ends the current charging round; Otherwise, it means that the electric vehicle continues the current charging round; is the continuous action of electric vehicle i at time t, which is used to represent the charging and discharging power of electric vehicle i at time t, that is, As a continuous variable, where c max It is the maximum charge and discharge power limit of the charging equipment.

5. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 4 is characterized in that: The reward function in step 4 is expressed as: where r i t represents the instantaneous reward value of electric vehicle i at time t; PO is the overload penalty, which is used to coordinate the charging power of multiple electric vehicles to prevent overload; AC i is the level of anxiety caused by traffic congestion to electric vehicles; AT i The time anxiety is caused by the deviation between the actual charging end time of the electric vehicle i and the expected charging end time; AR i It is the range anxiety of electric vehicles; PC i is the charging cost of electric vehicle i at time t; soc max Indicates the maximum battery capacity of an electric vehicle; k po Indicates the transformer overload penalty coefficient; po max Indicates the maximum capacity of the transformer; is a parameter used to reflect the degree of concern of electric vehicle i to electricity prices; is a parameter used to reflect the degree of concern of electric vehicle i about traffic congestion; is a parameter used to reflect the degree of concern of electric vehicle i to time deviation; It is a parameter used to reflect the degree of concern of electric vehicles about range anxiety.

6. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 5 is characterized in that: The multi-agent reinforcement learning framework in step 5 is: for N electric vehicles in the multi-agent charging system, a reinforcement learning network is constructed for each electric vehicle, and the multi-agent reinforcement learning framework is composed of N reinforcement learning networks; the reinforcement learning network is composed of an Actor network, a first Critic network, a second Critic network, a first target Critic network, and a second target Critic network; The method for initializing the multi-agent reinforcement learning framework is: respectively initializing N Actor networks in the multi-agent reinforcement learning framework N first critic networks N second critic networks N first target critic networks and N second target critic networks Where i = 1, 2,, N; θ represents the network parameters of the Actor network; φ represents the network parameters of the Critic network; Represents the network parameters of the target Critic network.

7. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 6 is characterized in that: The step 6 further comprises: Step 6.1: Get the state of electric car i at time t and the state of electric car i at time t+1 And the status Enter the initialized Actor network The continuous action of electric car i at time t is obtained Recorded as Step 6.2: Construct the connected action space of the multi-agent charging system based on the continuous actions of all electric vehicles at time t, denoted as in represents the continuous action of the first electric car at time t; represents the continuous action of the Nth electric car at time t; Step 6.3: Change the status After concatenating the action space with the connected action space, we input it into the initialized first Critic network to get K n The Q value of a discrete action is denoted by Step 6.4: According to K n The Q value of a discrete action is used to select the discrete action of electric car i at time t using a greedy strategy Step 6.5: Utilizing Discrete Actions of EV i and continuous action Get the action chosen by electric car i at time t And the action Interact with the environment to get the instantaneous reward value of electric car i at time t Step 6.6: Repeat steps 6.1-6.5 to obtain the state s of all electric vehicles at time t t , continuous action c t , instantaneous reward value r t and the state s at time t+1 t+1 And put it into the experience pool D as a set of data samples; Step 6.7: Repeat steps 6.1-6.6 until a preset number of data samples are obtained and placed in the experience pool D.

8. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 7 is characterized in that: The step 7 further comprises: Step 7.1: Randomly extract data samples of group B at different times from the experience pool D and input them into the multi-agent reinforcement learning framework; Step 7.2: For the reinforcement learning network of electric vehicle i in the multi-agent reinforcement learning framework, the network parameters of the two critic networks in the reinforcement learning network are updated using the temporal difference method; Step 7.3: Input the states and continuous actions in the data sample into the two updated Critic networks to obtain K n The Q value of each discrete action is obtained, and the Q values ​​obtained by the two Critic networks are summed up respectively, and then the smallest of the two summation results is selected as the corresponding Q value of the continuous action; Step 7.4: According to the corresponding Q value of the continuous action, the maximum entropy method is used to calculate the loss function value of the Actor network and update the network parameters of the Actor network; Step 7.5: Use the soft update method to update the network parameters of the two target critic networks; Step 7.6: Repeat steps 7.1-7.5 to iteratively train the multi-agent reinforcement learning framework until the preset training rounds are completed, obtain the trained multi-agent reinforcement learning framework and save it.

9. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 8, characterized in that: The update of the network parameters of the two Critic networks described in step 7.2 is implemented using the loss function of the Critic network, which is expressed as: in represents the loss function used to update the Critic network; j represents the serial number of the Critic network, and j∈{1,2}, that is, when j=1, represents the first Critic network; when j = 2, represents the second Critic network; ω is the network parameter of the Critic network; Indicates input status and continuous action The Q value output by the post-Critic network; Indicates extracting data samples from the experience pool D and calculating the Q value output by the Critic network and the true Q value y t The expected value of the mean square error; where the true Q value y t The calculation process is as follows: Where γ represents the discount factor; represents the maximum Q value output by the Critic network at time t+1; α represents the temperature coefficient; represents the logarithmic action probability.

10. The electric vehicle charging and discharging decision method based on hybrid action space according to claim 9, characterized in that: The expression of the corresponding Q value of the continuous action described in step 7.3 is: Where Q(s t ,c t ) indicates continuous action c t The corresponding Q value; k represents the discrete action that maximizes the Q value.

Images