Electric vehicle-photovoltaic charging station coordination control method based on reinforcement learning

By introducing Markov game process and Multi-Agent Transformer algorithm in electric vehicle charging stations, the use of electric vehicles and photovoltaic charging stations is solved, and the problems of unstable grid load and high operating costs are achieved, and more efficient energy use is achieved and faster training convergence.

CN118915431APending Publication Date: 2024-11-08HANGZHOU DIANZI UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410836718.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively coordinate the use of electric vehicle charging stations and renewable energy, resulting in unstable grid load and high operating costs, and the multi-agent reinforcement learning algorithm has problems such as convergence challenges and slow training speed.

Method used

A coordinated control method for electric vehicle-photovoltaic charging stations based on multi-agent reinforcement learning is proposed. By establishing Markov game process and Multi-Agent Transformer (MAT) algorithm, the decision-making process of charging stations is optimized and electric vehicles and photovoltaic charging is realized. Coordinated control of the station reduces grid load and operating costs.

Benefits of technology

Improve the operational efficiency of electric vehicle charging stations, promote the consumption of renewable energy, reduce grid load, and solve the complexity and convergence problems in multi-agent reinforcement learning through the MAT algorithm, achieving faster training and more High reward benefits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915431A_ABST
    Figure CN118915431A_ABST
Patent Text Reader

Abstract

The invention discloses an electric vehicle-photovoltaic charging station coordination control method based on reinforcement learning. The method comprises the following steps: firstly, establishing a charging and discharging model of an electric vehicle charging station equipped with photovoltaic (PV); electric vehicle-photovoltaic charging station coordination control is converted into a Markov game process, and observation, actions and rewards in the Markov game process are determined. A new multi-agent reinforcement learning algorithm MAT is established to solve the electric vehicle-photovoltaic charging station coordination problem; and training is carried out on real data by using an MAT algorithm, and the precision of the algorithm is tested, so that electric vehicle-photovoltaic charging station coordination control is realized. According to the method, the multi-agent reinforcement learning algorithm MAT based on the Transform architecture is used, so that the algorithm complexity increase is reduced from multiplication to addition, the reward is improved, it is ensured that the obtained reward is globally optimal, the operation benefits of the electric vehicle charging station are improved, and the consumption of renewable energy sources is promoted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things, and in particular relates to an electric vehicle-photovoltaic charging station coordinated control method based on multi-agent reinforcement learning. Background Art

[0002] In recent years, people have been increasingly inclined to low-carbon living due to the intensification of the greenhouse effect. Many countries promote electric vehicles (EVs) to solve energy and environmental problems, while interest in renewable energy sources such as solar and wind power is also growing. The combination of electric vehicles and renewable energy is seen as the key to reducing carbon emissions. However, large-scale electric vehicle charging affects grid load and voltage, while the integration of distributed renewable energy increases operational uncertainty. Energy storage technology can help, but its cost and safety limit its use. Advances in organized electric vehicle charging technology allow charging stations to operate as energy storage systems. As the number of electric vehicles increases, the scale of charging stations will also expand. Coordinating electric vehicle charging stations with renewable energy provides a promising way to achieve a safer, more reliable and more cost-effective energy system.

[0003] Traditional model-based optimization methods require accurate modeling and precise variable relationships, which often makes them difficult to solve and has a significant impact on the type and parameters of the model. Therefore, model-free methods, such as reinforcement learning (RL), are becoming increasingly popular among researchers. As a powerful data-centric decision-making and control technique, reinforcement learning (RL) learns optimal policies in dynamic systems through interactions with the environment to maximize rewards.

[0004] Existing research works mainly focus on the charging and discharging control of electric vehicles in specific market environments. In most studies, a single reinforcement learning (RL) agent usually controls an electric vehicle. On the one hand, multiple agents are difficult to control collaboratively, which may cause the electric vehicle to frequently switch between charging and discharging states or power levels, ultimately shortening the battery life. On the other hand, although some studies adopt multi-agent RL algorithms for agent coordination, increasing the number of agents may lead to convergence challenges and slow training speed. In addition, the actual collaborative control of electric vehicle charging stations with renewable energy sources has not been widely studied. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an electric vehicle-photovoltaic charging station coordinated control method based on reinforcement learning, the purpose of which is to maximize the operational efficiency of the electric vehicle charging station and promote the consumption of renewable energy.

[0006] A coordinated control method for electric vehicles and photovoltaic charging stations based on reinforcement learning comprises the following steps:

[0007] Step (1) establishes a charging and discharging model for an electric vehicle charging station equipped with photovoltaic (PV). By controlling the charging piles, the maximum utilization of PV energy is achieved, the income of the charging station is increased, the consumption of renewable energy is promoted, and the load on the power grid is reduced.

[0008] Step (2) transforms the coordinated control of electric vehicles and photovoltaic charging stations into a Markov game process and determines the observations, actions and rewards in the Markov game process.

[0009] Step (3) Considering the problems existing in the existing multi-agent reinforcement learning algorithms, such as the curse of dimensionality and policy gradient update conflicts, a new multi-agent reinforcement learning algorithm - MAT (Multi-Agent Transformer) is established to solve the electric vehicle-photovoltaic charging station coordination problem.

[0010] Step (4) uses the MAT algorithm to train on real data and test the accuracy of the algorithm to achieve coordinated control of electric vehicles and photovoltaic charging stations.

[0011] In step (1), when building a model of an electric vehicle charging station equipped with photovoltaics, the charging station controls the charging / discharging of electric vehicles by making decisions on charging piles, thereby maximizing the benefits of the charging station.

[0012] The model of an electric vehicle charging station equipped with photovoltaics is as follows:

[0013] (1-1) The profit of charging can be expressed as:

[0014] Z=I c +I d -C g -C p

[0015] Among them I c is the income from charging electric vehicles (EVs), I d is the income from selling electricity to the grid, C g is the income from purchasing electricity from the power grid, C p It is the penalty fee for electric vehicles not fully charged.

[0016] (1-2) The main revenue of charging stations comes from charging EVs, which is defined as follows:

[0017] I c =λ c p c

[0018] Among them, λ c represents the charging fee per kilowatt-hour (KW) paid by electric vehicle users to the charging station, pc It represents the charging power of electric vehicles, and the charging income obtained by the charging station is expressed as I c .

[0019] (1-3) Another part of the charging station’s revenue comes from selling electricity to the grid, which is defined as follows:

[0020] I d =λ g max(0, p d +p pv -p c )-λ d p d

[0021] During periods of high electricity prices, charging stations can control electric vehicle users to discharge and sell electricity to the grid. It is worth noting that discharge can only be performed at the discharge power p of the electric vehicle. d Plus the photovoltaic power generation power p pv Greater than the charging power p of the electric vehicle c The charging station is charged at a price of d Payment is made to the user of the electric vehicle that discharges the battery at a price λ g Discharge to the grid, thereby obtaining discharge income I d .

[0022] (1-4) The main cost of a charging station comes from purchasing electricity from the grid, which is defined as follows:

[0023] C g =λ g max(0, p c -p d -p pv )

[0024] Among them, when the demand for electric vehicle charging is high, photovoltaic (PV) power generation is not enough to meet the charging demand. At this time, the charging station g Purchase electricity from the grid.

[0025] (1-5) Another part of the cost of charging stations comes from penalty fees, which are defined as follows:

[0026] C p =λ p (e tgt -e d )

[0027] When an electric vehicle arrives at a charging station, it submits its charging specifications to the station management system, including the arrival time t a , the amount of electricity upon arrival a , expected departure time t d and the expected amount of electricity when leavingtgt The charging station must ensure that the charge is sufficient when leaving the station. d Not less than expected power tgt If the user's energy demand cannot be met, the charging station will pay a penalty fee C p to compensate users.

[0028] (1-6) Considering the charging behavior of EV, the following constraints should be followed:

[0029]

[0030] e min <e t <e max

[0031] e t+1 =e t +p t Δt

[0032]

[0033] Among them, for each electric vehicle at time t, the power pt of the electric vehicle should be limited by the maximum charging power and maximum discharge power At the same time, considering the battery capacity limitation of electric vehicles, the battery power of electric vehicles e t Should be limited by minimum battery charge min and maximum battery charge max In addition, e d The amount of electricity in the electric car when it leaves the vehicle.

[0034] In step (2), the coordinated control of electric vehicles and photovoltaic charging stations is transformed into a Markov game process. The observation definition of the Markov game process is as follows:

[0035]

[0036] Among them, Y i,t represents the connection status between the electric vehicle and the charging pile i, t represents the current time step, is the remaining residence time of the electric vehicle currently connected to the corresponding charging pile, e i,t is the battery charge currently connected to the electric vehicle, Expressed as the remaining power required to charge the battery, λ g represents the grid electricity price, p pv is the currently available photovoltaic power.

[0037] The actions of the Markov game process are defined as follows:

[0038] a i,t=[-1, 1]

[0039] Among them, a i,t Representing the action of each agent (indicated by index i), the interval represents the magnitude of the charging (positive) and discharging (negative) power of agent i as its power capacity in the power network percentage.

[0040] The reward of the Markov game process is defined as follows:

[0041]

[0042] Among them, the first part Represents the net income obtained by charging electric vehicles (EV) at the charging pile, where is the charging power. Part 2

[0043] Represents the net income obtained by the charging pile for electric vehicles, where is the discharge power. Part III Represents the cost of the charging pile purchasing electricity from the grid during the Δt period, where Indicates the power purchased by the charging pile from the power grid. Part 4 Represents the cost of penalties for failing to meet EV charging requirements.

[0044] In step (3), a new multi-agent reinforcement learning algorithm, MAT, is established to solve the electric vehicle-photovoltaic charging station coordination problem. The MAT algorithm is mainly based on the multi-agent advantage decomposition theorem and defines i 1:n represents a permutation of n agents, o represents joint observation, Represents agent i 1:n If the two actions are combined, then without any additional assumptions, the following equation always holds:

[0045]

[0046] in, For agent i 1:n The joint advantage function of For agent i m The advantage function of For agent i m action.

[0047] If each agent i knows the actions of the previous i-1 agents, it can take actions to maximize its own multi-agent advantage function. Each agent only needs to maximize its own multi-agent advantage to optimize the joint advantage function. Therefore, by using the above decomposition theorem, the joint decision-making problem can be transformed into a sequential decision-making problem, and the optimization of the joint advantage function can be transformed into the optimization of the local advantage function.

[0048] Therefore, the MAT (Multi-Agent Transformer) method converts the process of taking the observation of each agent as input and outputting actions into a process similar to machine translation, using the Transformer (Encoder-Decoder) structure. By inputting the observation of each agent, it is encoded into a series of potential representations through the encoding layer, and then passed to the decoding layer, and then the optimal action of each agent is output in a sequential and autoregressive manner.

[0049] In step (4), the electric vehicle-photovoltaic charging station coordinated control agent is trained based on the MAT algorithm. The MAT algorithm mainly consists of two parts, the encoder and the decoder. The specific goals are as follows:

[0050] The encoder's goal is to minimize the empirical Bellman error:

[0051]

[0052] where φ is the parameter of the target network, which is non-differentiable and updated every few epochs, Tn represents the total number of time steps of n agents, and R(o t , a t ) is the state o observed at time step t t and the action performed t The instant reward obtained, γ is the discount factor, is the state calculated using the current encoder parameters φ The value of is the next time step state calculated using the target network value.

[0053] The decoder's goal is to minimize the following Clip PPO objective:

[0054]

[0055]

[0056] in, is the time step t by agent i m The probability distribution of the new strategy and the probability distribution of the old strategy The ratio of is the agent i at time step t m The action performed, is the encoded observation value of all agents at time step t, are the actions of the first m-1 agents after encoding at time step t. is an estimate of the joint advantage function, and represents the global value function, and then uses the generalized advantage estimation (GAE) to calculate ∈ is a hyperparameter of the clipping range, which is used to control the range of policy updates. This is to cut the ratio to prevent the policy update from being too large.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention proposes an electric vehicle-photovoltaic charging station coordination control method based on multi-agent reinforcement learning. The coordination problem of EV-PV charging stations is described as a Markov game process. Aiming at the problems existing in the existing multi-agent reinforcement learning methods, a multi-agent reinforcement learning algorithm based on the Transformer architecture - MAT is used to reduce the algorithm complexity growth from multiplication to addition, and improve the reward to ensure that the reward (profit) obtained is globally optimal, thereby improving the operational efficiency of electric vehicle charging stations and promoting the consumption of renewable energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is an algorithm structure diagram of an electric vehicle-photovoltaic charging station coordinated control method based on multi-agent reinforcement learning of the present invention;

[0060] Figure 2 A schematic diagram of the system architecture of an electric vehicle-photovoltaic charging station coordinated control method based on multi-agent reinforcement learning according to the present invention;

[0061] Figure 3 Schematic diagram of the training process of different multi-agent reinforcement learning algorithms in an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of the testing process of different multi-agent reinforcement learning algorithms in an embodiment of the present invention;

[0063] Figure 5 Schematic diagram of the test results of the electric vehicle-photovoltaic charging station intelligent body within one day in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable relevant personnel to more clearly understand the technical content of the present invention, the present invention is further described in detail below in conjunction with the drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on it.

[0065] like Figure 1 As shown, a coordinated control method of an electric vehicle and a photovoltaic charging station based on reinforcement learning, the implementation example includes the following steps:

[0066] Step 1: Establish a charging and discharging model for an electric vehicle charging station equipped with photovoltaic (PV). Figure 2 As shown, by controlling the charging piles, the maximum utilization of photovoltaic energy is achieved, the income of the charging station is increased, the consumption of renewable energy is promoted, and the load on the power grid is reduced.

[0067] The model of an electric vehicle charging station equipped with photovoltaics is as follows:

[0068] The profit of charging can be expressed as:

[0069] Z=I c +I d -C g -C p

[0070] Among them I c is the income from charging electric vehicles, I d is the income from selling electricity to the grid, C g is the income from purchasing electricity from the power grid, C p It is the penalty fee for electric vehicles not fully charged.

[0071] The main revenue of charging stations comes from charging EVs, which is defined as follows:

[0072] I c =λ c p c

[0073] Among them, λ c represents the charging fee per kilowatt-hour (KW) paid by electric vehicle users to the charging station, p c It represents the charging power of electric vehicles, and the charging income obtained by the charging station is expressed as I c .

[0074] Another portion of the charging station’s revenue comes from selling electricity to the grid, which is defined as follows:

[0075] I d =λ g max(0, p d +p pv -p c )-λd p d

[0076] During periods of high electricity prices, charging stations can control electric vehicle users to discharge and sell electricity to the grid. It is worth noting that discharge can only be performed at the discharge power p of the electric vehicle. d Plus the photovoltaic power generation power p pv Greater than the charging power p of the electric vehicle c The charging station is charged at a price of d Payment is made to the user of the electric vehicle that discharges the battery at a price λ g Discharge to the grid, thereby obtaining discharge income I d

[0077] The main cost of a charging station comes from purchasing electricity from the grid, which is defined as follows:

[0078] C g =λ g max(0, p c -p d -p pv )

[0079] Among them, when the demand for electric vehicle charging is high, photovoltaic (PV) power generation is not enough to meet the charging demand. At this time, the charging station g Purchase electricity from the grid.

[0080] Another portion of the cost of charging stations comes from penalty fees, which are defined as follows:

[0081] C p =λ p (e tgt -e d )

[0082] When an electric vehicle arrives at a charging station, it submits its charging specifications to the station management system, including the arrival time t a , the amount of electricity upon arrival a , expected departure time t d and the expected amount of electricity when leaving tgt The charging station must ensure that the charge is sufficient when leaving the station. d Not less than expected power tgt If the user's energy demand cannot be met, the charging station will pay a penalty fee C p to compensate users.

[0083] Considering the charging behavior of EV, the following constraints should be followed:

[0084]

[0085] e min<e t <e max

[0086] e t+1 =e t +p t Δt

[0087]

[0088] Among them, for each electric vehicle at time t, the power pt of the electric vehicle should be limited by the maximum charging power and maximum discharge power At the same time, considering the battery capacity limitation of electric vehicles, the battery power of electric vehicles e t Should be limited by minimum battery charge min and maximum battery charge max In addition, e d The amount of electricity in the electric car when it leaves the vehicle.

[0089] Step 2: Convert the coordinated control of electric vehicles and photovoltaic charging stations into a Markov game process and determine the observations, actions, and rewards in the Markov game process.

[0090] The observation definition of the Markov game process is as follows:

[0091]

[0092] Among them, γ i,t represents the connection status between the electric vehicle and the charging pile i, t represents the current time step, is the remaining residence time of the electric vehicle currently connected to the corresponding charging pile, e i,t is the battery charge currently connected to the electric vehicle, Expressed as the remaining power required to charge the battery, λ g represents the grid electricity price, p pv is the currently available photovoltaic power.

[0093] The actions of the Markov game process are defined as follows:

[0094] a i,t =[-1, 1]

[0095] Among them, a i,t Representing the action of each agent (indicated by index i), the interval represents the magnitude of the charging (positive) and discharging (negative) power of agent i as its power capacity in the power network Percentage

[0096] The reward of the Markov game process is defined as follows:

[0097]

[0098] Among them, the first part Represents the net income obtained by charging electric vehicles (EV) at the charging pile, where is the charging power. Part 2

[0099] Represents the net income obtained by the charging pile for electric vehicles, where is the discharge power. Part III Represents the cost of the charging pile purchasing electricity from the grid during the Δt period, where Indicates the power purchased by the charging pile from the power grid. Part 4 Represents the cost of penalties for failing to meet EV charging requirements.

[0100] Step 3, considering the problems existing in the existing multi-agent reinforcement learning algorithms, such as the curse of dimensionality and policy gradient update conflicts, a new multi-agent reinforcement learning algorithm - MAT (Multi-Agent Transformer) is established to solve the electric vehicle-photovoltaic charging station coordination problem.

[0101] The MAT algorithm is mainly based on the multi-agent advantage decomposition theorem, defining i1:n as an arrangement of n agents, o as a joint observation, Represents agent i 1:n If the two actions are combined, then without any additional assumptions, the following equation always holds:

[0102]

[0103] in, For agent i 1:n The joint advantage function of For agent i m The advantage function of For agent i m action.

[0104] If each agent i knows the actions of the previous i-1 agents, then it can take actions to maximize its own multi-agent advantage function. Each agent only needs to maximize its own multi-agent advantage to optimize the joint advantage function. Therefore, by using the above decomposition theorem, the joint decision-making problem can be transformed into a sequential decision-making problem, and the optimization of the joint advantage function can be transformed into the optimization of the local advantage function.

[0105] Therefore, the MAT (Multi-Agent Transformer) method converts the process of taking the observations of each agent as input and outputting actions into a process similar to machine translation, adopting the Transformer (Encoder-Decoder), that is, the encoder-decoder structure. By inputting the observations of each agent, it is encoded into a series of latent representations through the encoding layer and then passed to the decoding layer, and the optimal actions of each agent can be output in a sequential and autoregressive manner.

[0106] As Figure 1 shown, first, each agent (charging pile) i obtains a local observation value o by observing its local state i , thus obtaining an observation sequence Then, the observation sequence is input into the encoder in any order through the embedding layer, and the parameters of the encoder are represented by φ. The encoder includes a self-attention mechanism, a multi-layer perceptron (MLP), and a residual connection. These components are used to alleviate the problems of gradient disappearance and network degradation when the network depth increases. Subsequently, we obtain the output of the observation sequence and encode it into In addition to encoding the data of agents (i1, i2,..., i n ), it also encodes the high-level dependency or interaction relationship representing the interaction between agents. In the training phase, the encoder is trained to approximate the value function, thereby learning representation learning. Specifically, we process the encoded observation sequence through an MLP layer to obtain the value function.

[0107] Next, the encoded observation sequence is input into the decoder, and the parameters of the decoder are represented by θ. The decoder uses a masked self-attention mechanism, and this mask ensures that the attention is only calculated between the i m and i n action heads (where m < n), thus maintaining the sequential update scheme. Subsequently, the decoder uses a second masked attention function to calculate the attention between the action head and the observation representation. Finally, this process is completed through the MLP and skip connection.

[0108] The decoder embeds the joint action sequence and inputs it into the decoder, and then outputs a series of representations of the joint action These representations are then processed through an MLP layer to output the probability distribution of the action m of agent i , that is, the policy

[0109] Finally, through the autoregressive method, the joint action sequence is obtained. By executing the joint action, the joint reward R(ot ,a t ), and (o t ,a t ,R(ot,at)) is inserted into the replay buffer In the calculation of loss function L Encoder (φ)+L Decoder (θ) and update the network parameters.

[0110] Step 4: Use the MAT algorithm to train on real data, evaluate the training performance of different algorithms, test the rewards obtained by different algorithms in each round, analyze the specific performance of the MAT algorithm in a round, and finally explain the effectiveness of the proposed algorithm, so as to achieve coordinated control of electric vehicles and photovoltaic charging stations. The MAT algorithm mainly consists of two parts, encoder and decoder, with the following specific goals:

[0111] The encoder's goal is to minimize the empirical Bellman error:

[0112]

[0113] where φ is the parameter of the target network, which is non-differentiable and updated every few epochs, Tn represents the total number of time steps of n agents, and R(o t , a t ) is the state o observed at time step t t and the action performed t The instant reward obtained, γ is the discount factor, is the state calculated using the current encoder parameters φ The value of is the next time step state calculated using the target network value.

[0114] The decoder's goal is to minimize the following clipping PPO objective:

[0115]

[0116]

[0117] in, is the time step t by agent i m The probability distribution of the new strategy and the probability distribution of the old strategy The ratio of is the agent i at time step t m The action performed, is the encoded observation value of all agents at time step t, are the actions of the first m-1 agents after encoding at time step t. is an estimate of the joint advantage function, and represents the global value function, and then uses the generalized advantage estimation (GAE) to calculate ∈ is a hyperparameter of the clipping range, which is used to control the range of policy updates. This is to cut the ratio to prevent the policy update from being too large.

[0118] In the test case, the electricity price data comes from San Francisco in the California Independent System Operator and is divided into three periods: peak period, mid-peak period and off-peak period. The electric vehicle data includes arrival time, departure time and energy demand, while the photovoltaic data describes the available photovoltaic power generation per hour. The time span of the experiment is T = 24 hours, each time step is t = 1 hour, corresponding to a period of 24 hours. In order to evaluate the effectiveness of the proposed algorithm, it is compared with two well-known multi-agent reinforcement learning algorithms, namely MADDPG and MAPPO, as baseline algorithms.

[0119] During the training process, Figure 3 As shown in the figure, as the number of training rounds increases, the reward gradually stabilizes. Compared with the MADDPG and MAPPO algorithms, the proposed method MAT significantly obtains higher rewards and converges faster. This is because the method considers the global reward and reduces the complexity growth from multiplication to addition compared to MADDPG and MAPPO, resulting in faster convergence, highlighting the effectiveness of the proposed method.

[0120] A test was conducted for 30 training rounds, and the rewards in each round using different algorithms were recorded, such as Figure 4 As shown in Figure 2, it is clear that the proposed algorithm is always higher than MAPPO and MADDPG in terms of rewards.

[0121] Figure 5 The performance of the proposed algorithm in a typical cycle is shown, including photovoltaic (PV) energy output and charging power during the day. It can be observed that the charging power of the charging station drops significantly during the period with higher electricity prices, especially during the peak electricity price period, where the charging power drops to a level close to the PV output. This is because there is a significant cost to purchase electricity from the grid during the peak electricity price period. In addition, during the second period with higher electricity prices, the gap between the charging power of the charging station and the PV output is smaller, indicating that charging EVs is more profitable as electricity prices are expected to drop. These observations further highlight the effectiveness of the proposed algorithm.

[0122] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A coordinated control method for electric vehicles and photovoltaic charging stations based on reinforcement learning, characterized in that: The steps include: Step (1) establishing a charging and discharging model of an electric vehicle charging station equipped with photovoltaics (PV); Step (2) converting the coordinated control of electric vehicles and photovoltaic charging stations into a Markov game process, and determining the observations, actions and rewards in the Markov game process; Step (3) establish a new multi-agent reinforcement learning algorithm - MAT to solve the electric vehicle-photovoltaic charging station coordination problem; Step (4) uses the MAT algorithm to train on real data and test the accuracy of the algorithm to achieve coordinated control of electric vehicles and photovoltaic charging stations.

2. According to claim 1, a method for coordinated control of electric vehicles and photovoltaic charging stations based on reinforcement learning is characterized in that: In step (1), when building a model of an electric vehicle charging station equipped with photovoltaics, the charging station controls the charging / discharging of electric vehicles by making decisions on charging piles, thereby maximizing the benefits of the charging station; The model of an electric vehicle charging station equipped with photovoltaics is as follows: (1-1) The profit of charging can be expressed as: Z=I c +I d -C g -C p Among them I c is the income from charging electric vehicles (EVs), I d is the income from selling electricity to the grid, C g is the income from purchasing electricity from the power grid, C p It is the penalty fee for electric vehicles that are not fully charged; (1-2) The main revenue of charging stations comes from charging EVs, which is defined as follows: I c =λ c p c Among them, λ c represents the charging fee per kilowatt-hour (KW) paid by electric vehicle users to the charging station, p c It represents the charging power of electric vehicles, and the charging income obtained by the charging station is expressed as I c ; (1-3) Another part of the charging station’s revenue comes from selling electricity to the grid, which is defined as follows: I d =λ g max(0,p d +p pv -p c )-l d p d During periods of high electricity prices, charging stations can control electric vehicle users to discharge and sell electricity to the grid; it is worth noting that discharge can only be carried out within the discharge power p of the electric vehicle. d Plus the photovoltaic power generation power p pv Greater than the charging power p of the electric vehicle c When the charging station is charged d Payment is made to the user of the electric vehicle that discharges the battery at a price λ g Discharge to the grid, thereby obtaining discharge income I d ; (1-4) The main cost of a charging station comes from purchasing electricity from the grid, which is defined as follows: C g =λgmax(0,p c -p d -p pv ) Among them, when the demand for electric vehicle charging is high, photovoltaic PV power generation is not enough to meet the charging demand; at this time, the charging station g Purchase electricity from the grid; (1-5) Another part of the cost of charging stations comes from penalty fees, which are defined as follows: C p =λ p (And tgt -And d ) When an electric vehicle arrives at a charging station, it submits its charging specifications to the station management system, including the arrival time t a , the amount of electricity upon arrival a , expected departure time t d and the expected amount of electricity when leaving tgt ; The charging station must ensure that the power is sufficient when leaving d Not less than expected power tgt If the user's energy demand cannot be met, the charging station will pay a penalty fee C p To compensate users; (1-6) Considering the charging behavior of EV, the following constraints should be followed: and min <e t <e max And t+1 =and t +p t Δt Among them, for each electric car at time t, the power p of the electric car is t Should be limited by the maximum charging power and maximum discharge power At the same time, considering the battery capacity limitation of electric vehicles, the battery power of electric vehicles e t Should be limited by minimum battery charge min and maximum battery charge max In addition, e d The amount of electricity in the electric car when it leaves the vehicle.

3. The electric vehicle-photovoltaic charging station coordinated control method based on reinforcement learning according to claim 2 is characterized in that: In step (2), the coordinated control of electric vehicles and photovoltaic charging stations is transformed into a Markov game process. The observation definition of the Markov game process is as follows: Among them, γ i,t represents the connection status between the electric vehicle and the charging pile i, t represents the current time step, is the remaining residence time of the electric vehicle currently connected to the corresponding charging pile, e i,t is the battery charge currently connected to the electric vehicle, Expressed as the remaining power required to charge the battery, λ g represents the grid electricity price, p pv is the currently available PV power; The actions of the Markov game process are defined as follows: a i,t =[-1,1] Among them, a i,t represents the action of each agent i, and the interval represents the size of the charging and discharging power of agent i as its power capacity in the power network percentage of The reward of the Markov game process is defined as follows: Among them, the first part Represents the net income obtained by charging electric vehicles (EV) at the charging pile, where is the charging power; the second part Represents the net income obtained by the charging pile for electric vehicles, where is the discharge power; the third part Represents the cost of the charging pile purchasing electricity from the grid during the Δt period, where Indicates the power purchased by the charging pile from the power grid; Part 4 Represents the cost of penalties for failing to meet EV charging requirements.

4. The electric vehicle-photovoltaic charging station coordinated control method based on reinforcement learning according to claim 3 is characterized in that: In step (3), a new multi-agent reinforcement learning algorithm - MAT is established to solve the electric vehicle-photovoltaic charging station coordination problem; The MAT algorithm is based on the multi-agent advantage decomposition theorem, defining i 1:n represents a permutation of n agents, o represents joint observation, Represents agent i 1:n If the two actions are combined, then without any additional assumptions, the following equation always holds: in, For agent i 1:n The joint advantage function of For agent i m The advantage function of For agent i m Actions; If each agent i knows the actions of the previous i-1 agents, it can take actions to maximize its own multi-agent advantage function; each agent only needs to maximize its own multi-agent advantage to optimize the joint advantage function; therefore, by using the above decomposition theorem, the joint decision-making problem can be transformed into a sequential decision-making problem, and the optimization of the joint advantage function can be transformed into the optimization of the local advantage function; Therefore, the MAT algorithm converts the process of taking the observation of each agent as input and outputting actions into a process similar to machine translation, using the Transformer, or encoder-decoder structure; by inputting the observation of each agent, it is encoded into a series of potential representations through the encoding layer, and then passed to the decoding layer, and then the optimal action of each agent is output in a sequential and autoregressive manner.

5. The electric vehicle-photovoltaic charging station coordinated control method based on reinforcement learning according to claim 4 is characterized in that: In step (4), the electric vehicle-photovoltaic charging station coordinated control agent is trained based on the MAT algorithm. The MAT algorithm mainly consists of two parts, the encoder and the decoder. The specific goals are as follows: The encoder's goal is to minimize the empirical Bellman error: where φ is the parameter of the target network, which is non-differentiable and updated every few epochs, Tn represents the total number of time steps of n agents, and R(o t ,a t ) is the state o observed at time step t t and the action performed t The instant reward obtained, γ is the discount factor, is the state calculated using the current encoder parameters φ The value of is the next time step state calculated using the target network the value of The decoder's goal is to minimize the following Clip PPO objective: in, is the time step t by agent i m The probability distribution of the new strategy and the probability distribution of the old strategy The ratio, is the agent i at time step t m The action performed, is the encoded observation value of all agents at time step t, is the actions of the first m-1 agents after encoding at time step t; is an estimate of the joint advantage function, and Denote the global value function, and then use the generalized advantage estimate GAE to calculate ∈ is a hyperparameter of the clipping range, which is used to control the range of policy updates. This is to cut the ratio to prevent the policy update from being too large.

6. The electric vehicle-photovoltaic charging station coordinated control method based on reinforcement learning according to claim 4 or 5, characterized in that: The specific implementation of the MAT algorithm is as follows: First, each agent i obtains a local observation value o by observing its local state i , thus obtaining an observation sequence Then, the observation sequence is input into the encoder in any order through the embedding layer. The parameters of the encoder are represented by φ. The encoder includes a self-attention mechanism, a multi-layer perceptron MLP, and a residual connection. Then, the output of the observation sequence is obtained and encoded as In the training phase, representation learning is learned by training the encoder to approximate the value function; Specifically, the encoded observation sequence is processed through an MLP layer to obtain the value function; Next, the encoded observation sequence is input into the decoder, and the parameters of the decoder are represented by θ; the decoder uses a masked self-attention mechanism, and this mask ensures that the attention is only calculated m between the i n and i action heads (where m < n), thus maintaining the sequential update scheme; subsequently, the decoder uses a second masked attention function to calculate the attention between the action head and the observation representation; finally, this process is completed through an MLP and skip connections; The decoder embeds the joint action sequence And input it into the decoder, which then outputs a series of representations of the joint action These representations are then processed through an MLP layer, outputting agent i m Action The probability distribution of strategy Finally, the joint action sequence is obtained through the autoregressive method By performing joint actions, we obtain joint rewards R(o t ,a t ), and (o t ,a t ,R(ot,a)) is inserted into the replay buffer In the calculation of loss function L Encoder (φ)+L Decoder (θ) and update the network parameters.

Citation Information

Cited By

  • Power data scheduling method, system and equipment based on deep reinforcement learning

    CN119623981A

  • Multi-charging-station electric vehicle charging and discharging scheduling method based on reinforcement learning

    CN119831240A

  • Photovoltaic power station parameter identification method based on Transform-PPO deep reinforcement learning

    CN119940134A

  • Privacy protection type multi-agent electric vehicle charging scheduling method and system

    CN121481179A