Electric vehicle charging station charging control method and system based on deep reinforcement learning

Abstract

The invention provides an electric vehicle charging station charging control method and system based on deep reinforcement learning, and relates to the technical field of electric vehicle charging control. Setting a reward function and constructing an electric vehicle charging coordination control model by taking a control objective of realizing profit maximization of a charging station while meeting the charging demand of an electric vehicle driver; converting the electric vehicle coordinated charging coordination control model in the charging station into a Markov random game model; a multi-agent reinforcement learning algorithm of a centralized Critic network with an attention mechanism is introduced, a Markov random game model is solved, and through continuous interaction with dynamic environment information, an updating strategy of each charger agent is optimized so as to maximize an expected accumulated reward, so that the charging efficiency is improved. And the optimal electric vehicle charging coordination control strategy is obtained.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electric vehicle charging control, and in particular to a charging control method and system for an electric vehicle charging station based on deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] The core goal of the decarbonization revolution is to achieve a low-carbon or carbon-free economy to reduce dependence on fossil fuels and shift to renewable energy and low-carbon technologies. Electric vehicle fleets can fully tap their potential as flexible distributed energy storage, achieve grid peak load regulation, integrate renewable energy, and delay the need for grid capacity upgrades through efficient electric vehicle charging management.

[0004] Most of the existing technologies for charging control of electric vehicle charging stations use robust optimization or chance constraints to characterize the uncertainty associated with the charging environment. However, in the real charging environment, due to the complex dynamic characteristics and mutual coupling of charging stations, traffic and user behavior, it is unrealistic to perfectly characterize them using mathematical models. In addition, traditional mathematical model solving techniques are also difficult to cope with real-time charging control requirements. Summary of the invention

[0005] In order to solve the above problems, the present disclosure proposes a charging control method and system for electric vehicle charging stations based on deep reinforcement learning. The chargers in the charging station are regarded as multi-intelligent agents. By optimizing the charging power action and using the charging regulation strategy, orderly charging control of electric vehicles is achieved. The charging power can be dispatched within the available charging time of the electric vehicle. While ensuring that the charging needs of the electric vehicle are met, the revenue of the charging station can be maximized.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions: The charging control method of electric vehicle charging station based on deep reinforcement learning includes: Obtain charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; According to the charging station information, electric vehicle information and electric vehicle charging power, the control goal is to meet the charging needs of electric vehicle drivers while maximizing the profit of the charging station, set the reward function, and build an electric vehicle charging coordination control model; The coordinated charging control model of electric vehicles in the charging station is transformed into a Markov random game model; A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is adopted to solve the Markov random game model. Each charger in the charging station is taken as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with the dynamic environment information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordination control strategy is obtained.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: Electric vehicle charging station charging control system based on deep reinforcement learning, including: The data acquisition module is used to obtain the charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; A reward function building module is used to set a reward function and build an electric vehicle charging coordination control model based on the charging station information, electric vehicle information and electric vehicle charging power, with the control goal of satisfying the charging needs of electric vehicle drivers while maximizing the profit of the charging station; The decision-making update module is used to transform the coordinated charging control model of electric vehicles inside the charging station into a Markov random game model. A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is used to solve the Markov random game model. Each charger in the charging station is used as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with dynamic environmental information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordinated control strategy is obtained.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program, and when the computer program is executed by a processor, the charging control method of an electric vehicle charging station based on deep reinforcement learning is implemented.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the charging control method of an electric vehicle charging station based on deep reinforcement learning is implemented.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the charging control method of an electric vehicle charging station based on deep reinforcement learning.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The electric vehicle charging station charging control method based on deep reinforcement learning disclosed in the present invention adopts a centralized training and decentralized execution framework for the coordinated control problem of electric vehicle charging. Each charger in the charging station is used as an intelligent agent and has control autonomy over the charging power of the connected electric vehicle. The chargers in the charging station are used as multiple intelligent agents and the charging power actions are optimized. In the centralized training stage, multiple intelligent agents observe their own states and obtain the state information of other intelligent agents through communication. Then each intelligent agent makes a specific action, that is, the charging power executed. The environment receives a t , and the transition state s t+1 , and reward r t Feedback is given to the agent, and by constantly interacting with the dynamic environment information, the strategy of each charger agent is continuously optimized to maximize the expected cumulative reward. In the decentralized execution phase, each agent makes decisions based only on its own observations, and there is no communication between agents. This solves the charging scheduling problem of electric vehicles arriving and leaving randomly, and alleviates the shortcomings of traditional mathematical models and single-agent reinforcement learning in dealing with the uncertainty of the charging environment and the scalability of the agent scale.

[0012] The disclosed charging control method for electric vehicle charging stations based on deep reinforcement learning can dispatch charging power within the available charging time of electric vehicles, and can maximize the revenue of charging stations while ensuring that the charging needs of electric vehicles are met. In addition, the model test time of the disclosed model at different scales is maintained at an average decision of about 23 ms per electric vehicle, which shows that the model has the ability to make real-time decisions and online recommendations. At the same time, the present invention can also be used to guide the planning of charging stations. Unplanned increase in the number of chargers does not necessarily improve the operating efficiency of charging stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0014] Figure 1 The overall process framework of the charging control method of the embodiment of the present disclosure; Figure 2This is a framework diagram of the multi-agent deep reinforcement learning method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0016] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0017] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0018] Example 1 In one embodiment of the present disclosure, a charging control method for an electric vehicle charging station based on deep reinforcement learning is provided, which realizes orderly charging control of electric vehicles through a charging regulation strategy, and has important theoretical value and practical significance for releasing the adjustment capability of massive flexible resources and promoting good interaction between vehicles and users. The steps include: Step 1: Obtain charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; Step 2: According to the charging station information, electric vehicle information and electric vehicle charging power, the reward function is set to build an electric vehicle charging coordination control model with the control goal of satisfying the charging needs of electric vehicle drivers while maximizing the profit of the charging station; Step 3: Convert the coordinated charging control model of electric vehicles inside the charging station into a Markov random game model; Step 4: A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is used to solve the Markov random game model. Each charger in the charging station is used as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with dynamic environmental information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordination control strategy is obtained.

[0019] As an embodiment, the specific implementation process of the electric vehicle charging station charging control method based on deep reinforcement learning disclosed in the present invention is as follows: Step 1: Obtain the charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; Specifically, taking a single charging station as an example, the charging station information and electric vehicle information are obtained. Electric vehicles include three categories: vehicles that are being charged, vehicles waiting to be charged, and vehicles that are about to arrive. Electric vehicle information includes vehicle arrival time, latest departure time, battery charge state, and battery capacity; charging station information includes the number of charging piles and charging power. With the development of intelligent transportation technology and machine learning, based on navigation, traffic congestion, user driving behavior and other information, the time when electric vehicles arrive at the charging station can be predicted and estimated using existing algorithms.

[0020] Furthermore, the adjustable range of charging power and the adjustment time interval of the electric vehicle are determined according to the charging power of the charging pile in the charging station, wherein the adjustable range of charging power is specifically -Pmax~Pmax, Pmax is the maximum charging power allowed by the charging pile and the electric vehicle, and the adjustment time interval can be set to minutes~hours.

[0021] Step 2: According to the charging station information, electric vehicle information and electric vehicle charging power, the reward function is set to build an electric vehicle charging coordination control model with the control goal of satisfying the charging needs of electric vehicle drivers while maximizing the profit of the charging station; Specifically, the control goal is to meet the charging needs of electric vehicle drivers while maximizing the profit of charging stations. An electric vehicle charging coordination control model is constructed. A reward function is set based on the control goal of the electric vehicle charging coordination control model. The setting indicators of the reward function may include charging station charging income, penalties for not meeting user charging expectations, waiting penalties, and charging station overload penalties. The calculation basis of each indicator of the reward function is as follows: 1) Charging station charging income: Electric vehicle owners pay the charging station for each kilowatt-hour of charging , charging stations pay for each kilowatt-hour of energy they purchase from the power company . Then the charging station i exist t The net income from time-slot charging is:

[0022] in, represents the charging operation income of the charging station, For charger i The operating power, ∆ t From the period t To time period t +1 time interval, Pay per kWh of energy, Pay charging stations for each kilowatt-hour charged; Representative charger i Whether to connect electric vehicles. =1 means charging, =0 means the charger is idle.

[0023] 2) User undercharging penalty: When an electric vehicle t d When the battery capacity is less than the expected battery capacity, it will lead to a poor experience for the electric vehicle driver. The mathematical model is used to describe the discomfort penalty of the electric vehicle driver during the charging process:

[0024] in, , k 1 and k 2 is the shape parameter. , . k 1 and k 2The larger the value, the higher the SoC during charging. Will continue to increase. The battery charge of the EV when it is expected to leave.

[0025] Specifically, at the arrival time t a hour, is equal to 0. When the expected departure time is t d The driver's expectation of battery charge is equal to the driver's expected battery charge. t Comfort Rewards It can be defined as:

[0026] in, Indicated in t The actual battery charge of the EV being charged during this time period.

[0027] When the electric vehicle is not charged to the required amount of electricity within the expected time, i Each charger will be subject to an undercharge penalty, defined as:

[0028] in, is the insufficient charging penalty term. The sub-reward function of the electric vehicle driver’s charging satisfaction is .

[0029] 3) Waiting penalty: To avoid the decline in user satisfaction caused by vehicles waiting too long at charging stations, each charger i Set waiting penalty Since the length of the electric vehicle’s waiting time is not directly related to a specific charger agent, the corresponding waiting penalty is shared by all chargers.

[0030]

[0031] in, For charger i In the period t Make a move The waiting reward (punishment) obtained after that. For charger i The maximum allowed charging power. is the unit time conversion factor. for t The number of vehicles waiting at the charging station during the time period. t From the period t To time period t +1 for the time interval.

[0032] 4) Overload penalty: When the total charging power of all chargers exceeds the maximum load of the charging station During the usage period, the charging station needs to pay additional electricity purchase fees. The price paid for each kilowatt peak demand is . Then the charger i The overload penalty is expressed as:

[0033] in, For charger i In the period t Make a move The overload reward (penalty) obtained after the charging station is charged. When the total action power of all chargers does not exceed the maximum load of the charging station, the overload reward (penalty) is 0. When the total action power of all chargers in the charging station is greater than the maximum load of the charging station, the charger i The overload reward is related to the power of the action selected by the charging station. The larger the action value, the greater the contribution to the additional electricity purchase cost that the charging station needs to pay. for t The maximum load allowed at the charging station at any time.

[0034] Step 3: Convert the coordinated charging control model (process) of electric vehicles inside the charging station into a Markov random game model (process); Specifically, the coordinated charging control problem of electric vehicles within a charging station is formulated as a stochastic game process. E = {1, 2, i , ..., N}express N A collection of chargers (agents), i ∈ E Indicates the first i chargers. Markov decision process It is composed of state, action, reward, state transition probability and discount factor. Five-tuple description.

[0035] The state space is the number of charging stations, and the speed of power will affect the charging time and whether to wait. N chargers, then the observation quantity of each charger agent is The information included is:

[0036] in, t is the global time. Representative charger i Whether to connect electric vehicles. =1 means charging, =0 means the charger is idle. δ i,t =1, Indicated in t The SOC of the EV being charged during the period, is the SOC when the EV is expected to leave, To connect the charger i EV battery capacity, for t Time-limited charger i The charging power, The maximum charging power allowed by the charger.

[0037] The maximum charging power allowed for electric vehicles. For charging stations t The number of waiting vehicles at the time. for t The total charging power allowed by the charging station at all times.

[0038] 2) Action space: The charging action is set as a discrete value. t Time-limited charger i The action is set to , the action value is the charging power of the charger. The action needs to meet the following conditions: a. The operating power cannot exceed the maximum charging power allowed by the charger or electric vehicle.

[0039] b. When the charger i When no electric vehicle is connected, the operating power is 0.

[0040] 3) Reward space: The coordinated control objective of electric vehicle charging is simulated as maximizing the profit of charging stations while meeting the charging needs of electric vehicle drivers. Therefore, the indicators of the reward function include the sum of charging income, insufficient charging penalty when the driver's charging expectations are not met, waiting penalty, and overload penalty. It can be obtained according to the calculation process in step 2.

[0041] 4) State transition probability: During the state transition process of the system, the increment of the SOC of the electric vehicle receiving charging can be expressed with high precision. i After the charger agent determines the charging power of the connected electric vehicle, the charging time interval of the electric vehicle changes from t arrive t +1, calculated based on the relationship between unit charging power and battery capacity. i If the electric vehicle with a charger fails to complete charging within the dwell time, the information is updated as follows:

[0042] If you connect i The charging process of the electric vehicles of the chargers is completed, and there are no electric vehicles waiting or driving into the charging station. Then the status information is updated as follows:

[0043] Step 4: A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is used to solve the Markov random game model. Each charger in the charging station is used as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with dynamic environmental information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordination control strategy is obtained.

[0044] Specifically, we will construct a Markov decision process The five-tuple The constructed data set is input into the multi-agent deep reinforcement learning method. The multi-agent deep reinforcement learning method is as follows Figure 2 As shown, the following steps are included: The multi-agent reinforcement learning algorithm consists of agents represented by deep neural networks and a model-free system environment. The system environment is a dynamic environment coupled to the electric transportation network, including the operation status of charging stations with randomness and uncertainty, dynamic user charging behavior, and potential future charging demand. The deep neural network uses two sets of actor-critic structures. Each agent has its own independent parameters. θμ i Actor Network The charger agent takes actions based on its own state observations. By generating an action mask vector, the mask segments of the valid action dimension are encoded as 1, otherwise 0. When the charger is not connected to the EV, the action decision of its operation should be zero power. When the charger is connected to the EV, the action decision of its operation should be constrained to be within the minimum value of the maximum allowed power of the EV and the charger.

[0045] The multi-agent reinforcement learning algorithm disclosed in this paper introduces a centralized critic network with an attention mechanism, which enhances the scalability of the algorithm. The centralized critic network is represented as:

[0046] in, f i is a two-layer multi-layer perceptron (MLP), g i is a single-layer MLP encoder. At the beginning of training, the agent i Observed value o i and actions a i Input encoder g i ( o i , a i ), and obtain the agent i Encoded information e i By performing multi-head attention calculation on the encoded information of other agents, we can obtain the attention of other charger agents to the agent. i Total Contribution x i , which is specifically expressed as follows:

[0047] in, w j is the attention weight. v j Representing an Agent j Embedded functions. h (.) is a non-linear activation function. Vis the shared change matrix of the agent. The attention weight is calculated by the following formula. By comparing the encoded information e j and e i The similarity between i For Agents j the value of attention. W k and W q They are e j and e i The linear transformation matrix of .

[0048]

[0049] Agent i The Critic network of uses the attention mechanism to improve the scalability problem that the input of the Critic network increases exponentially with the number of agents. Its output is estimated Q i Values ​​are used to evaluate actions a i In order to stabilize the training, each agent has its own parameters and The target Actor network and target Critic network. Each time the Actor network and Critic network parameters are updated, the target Actor and target Critic network need to be soft updated. τ It is a soft update parameter.

[0050]

[0051]

[0052] Agent i Goal Q The values ​​are:

[0053] in, γ is the discount factor. It is an intelligent agent i The target action is the output of the target Actor network. α is the temperature parameter that determines the balance between entropy maximization and reward.

[0054] The estimate output by the Critic network Q Value and target Critic network output target Q The value defines the loss function:

[0055] The critic network parameters are updated by gradient descent as shown below:

[0056] In the formula, σ is the learning rate of the Critic network.

[0057] Agent i The gradient calculation of the Actor network is determined by the following formula:

[0058] Among them, the counterfactual baseline is used to evaluate the contribution of a single agent action to the charging station system reward, that is, the global Q The value minus the expected value of the random selection of the agent's action is defined as follows:

[0059]

[0060] in,\ i Indicates except i Other intelligent agents.

[0061] Actor Network Parameters The update is as follows:

[0062] Based on the above neural network structure, a centralized training and decentralized execution framework is adopted in the coordinated control of electric vehicle charging. Each charger in the charging station acts as an intelligent agent and has control autonomy over the charging power of the connected electric vehicles. Specifically, in the centralized training stage, multiple agents observe their own status and obtain the status information of other agents through communication. Then each agent makes a specific action, that is, the charging power executed. a t , and the transition state s t+1 , and reward r t Feedback to the agent. By continuously interacting with the dynamic environment information, the strategy of each charger agent is continuously optimized to maximize the expected cumulative reward. In the decentralized execution phase, each agent makes decisions based only on its own observations, and there is no communication between agents.

[0063] As an embodiment, in this embodiment, the charging control time interval is set to be executed once every 10 minutes, and the optimization period is 24 hours.

[0064] Example 2 In one embodiment of the present disclosure, a charging control system for an electric vehicle charging station based on deep reinforcement learning is provided, comprising: The data acquisition module is used to obtain the charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; A reward function building module is used to set a reward function and build an electric vehicle charging coordination control model based on the charging station information, electric vehicle information and electric vehicle charging power, with the control goal of satisfying the charging needs of electric vehicle drivers while maximizing the profit of the charging station; The decision-making update module is used to transform the coordinated charging control model of electric vehicles inside the charging station into a Markov random game model. A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is used to solve the Markov random game model. Each charger in the charging station is used as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with dynamic environmental information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordinated control strategy is obtained.

[0065] Example 3 In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the charging control method for an electric vehicle charging station based on deep reinforcement learning.

[0066] Example 4 In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the charging control method of an electric vehicle charging station based on deep reinforcement learning is implemented.

[0067] Example 5 In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the charging control method of an electric vehicle charging station based on deep reinforcement learning.

[0068] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A charging control method for electric vehicle charging stations based on deep reinforcement learning, characterized in that: include: Obtain charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; According to the charging station information, electric vehicle information and electric vehicle charging power, the control goal is to meet the charging needs of electric vehicle drivers while maximizing the profit of the charging station, set the reward function, and build an electric vehicle charging coordination control model; The coordinated charging control model of electric vehicles in the charging station is transformed into a Markov random game model; A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is adopted to solve the Markov random game model. Each charger in the charging station is taken as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with the dynamic environment information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordination control strategy is obtained.

2. The charging control method for an electric vehicle charging station based on deep reinforcement learning as claimed in claim 1, characterized in that: Electric vehicles include vehicles that are being charged, vehicles waiting to be charged, and vehicles that are about to arrive. Electric vehicle information includes vehicle arrival time, latest departure time, battery charge status, and battery capacity. Charging station information includes the number of charging piles and charging power. The adjustable range and adjustment time interval of the electric vehicle charging power are set according to the charging power of the charging piles in the charging station. The adjustable range is specifically - P max ~ P max , P max The maximum charging power allowed by the charging pile and electric vehicle is adjusted within a time interval ranging from minutes to hours.

3. The charging control method for an electric vehicle charging station based on deep reinforcement learning as claimed in claim 1, characterized in that: The indicators of the reward function of the electric vehicle charging coordination control model include charging income of the charging pile, user undercharging penalty, waiting penalty and overload penalty; Charging income from charging piles is, in, represents the charging operation income of the charging station, For charger i The operating power, ∆ t From the period t To time period t +1 time interval, Pay per kWh of energy, Pay charging stations for each kilowatt-hour charged; Representative charger i Whether to connect electric vehicles. =1 means charging, =0 means the charger is idle. The user's insufficient charging penalty includes: when the battery capacity t of the electric vehicle at the time of departure is less than the expected battery capacity, a mathematical model is used to describe the discomfort penalty of the electric vehicle driver during the charging process, the comfort reward, and when the electric vehicle is not charged to the required amount of power within the expected time. i The under-charging penalty that each charger will receive.

4. The electric vehicle charging station charging control method based on deep reinforcement learning as claimed in claim 3 is characterized in that: The overload penalty is when the total charging power of all chargers exceeds the maximum load of the charging station, and the charging station needs to pay additional electricity purchase fees. When the sum of the total action power of all chargers in the charging station is greater than the maximum load of the charging station, the overload reward of the charger is related to the size of the action power selected by itself. The larger the action value, the greater the contribution to the additional electricity purchase cost that the charging station needs to pay.

5. The charging control method for an electric vehicle charging station based on deep reinforcement learning as claimed in claim 1, characterized in that: The chargers in the charging station are regarded as multi-agents, and the coordinated control model of electric vehicle charging in the charging station is expressed as a Markov random game process. E = {1, 2, i , ..., N }express N A collection of chargers, i ∈ E Indicates the first i The Markov random game process consists of states, actions, rewards, state transition probabilities and discount factors. Five-tuple description.

6. The charging control method for an electric vehicle charging station based on deep reinforcement learning as claimed in claim 1, characterized in that: The multi-agent reinforcement learning algorithm consists of agents represented by deep neural networks and a model-free system environment. The system environment is a dynamic environment coupled with an electric transportation network, including the random and uncertain operating status of charging stations, dynamic user charging behavior, and potential future charging demand. The deep neural network uses two sets of Actor-critic structures. Each agent has an Actor network with independent parameters. The charger agent takes actions based on its own state observations. By generating action mask vectors, a centralized Critic network with an attention mechanism is introduced. The loss function is defined by the estimated value output by the Critic network and the target value output by the target Critic network. The strategy of each charger agent is continuously optimized to maximize the expected cumulative reward.

7. Electric vehicle charging station charging control system based on deep reinforcement learning, characterized in that: include: The data acquisition module is used to obtain the charging station information and electric vehicle information, and determine the adjustable range and adjustment time interval of the electric vehicle charging power according to the charging power of the charging pile in the charging station; A reward function building module is used to set a reward function and build an electric vehicle charging coordination control model based on the charging station information, electric vehicle information and electric vehicle charging power, with the control goal of satisfying the charging needs of electric vehicle drivers while maximizing the profit of the charging station; The decision-making update module is used to transform the coordinated charging control model of electric vehicles inside the charging station into a Markov random game model. A multi-agent reinforcement learning algorithm with a centralized critic network with an attention mechanism is used to solve the Markov random game model. Each charger in the charging station is used as an agent. Multiple agents observe their own status and obtain the status information of other agents to make specific actions. By continuously interacting with dynamic environmental information, the update strategy of each charger agent is optimized to maximize the expected cumulative reward until the optimal electric vehicle charging coordinated control strategy is obtained.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the charging control method for an electric vehicle charging station based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the charging control method of the electric vehicle charging station based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the charging control method for an electric vehicle charging station based on deep reinforcement learning as described in any one of claims 1-6.

Images