Power data scheduling method, system and equipment based on deep reinforcement learning

Abstract

The invention belongs to the technical field of vehicle charging, and particularly relates to an electric power data scheduling method, system and device based on deep reinforcement learning. The method comprises the following steps: S1, establishing a mathematical model representing a charging station charging and discharging scheduling optimization problem; s2, modeling a mathematical problem into a Markov decision process, and constructing a scheduling optimization network based on a DQN algorithm model; and S3, training the scheduling optimization network according to the operation data of the charging station, so that the scheduling optimization network adaptively obtains a better control strategy in continuous decision making and learning to maximize the economic benefit of the charging station and the service satisfaction of a user. And S4, the control center of the charging station obtains the demand information of the current vehicle, and then generates an optimal charging and discharging scheme meeting the demand of the current user by using the trained scheduling optimization network. According to the invention, the problem that the existing charging and discharging scheduling scheme adopting the V2G technology is difficult to effectively balance customer demands and charging station benefits is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle charging, and specifically relates to a power data scheduling method, system and device based on deep reinforcement learning. Background Art

[0002] Electric vehicles (EVs) are a type of transportation vehicle that is powered by electricity. They have attracted more and more attention due to their high efficiency, zero pollution, and low emissions. As the market share of electric vehicles continues to grow, the demand for vehicle charging is also increasing. Large-scale and disorderly access to electric vehicles will have adverse effects on the power grid, such as increased peak-to-valley differences, reduced power quality, and increased difficulty in optimizing and controlling power grid operations.

[0003] In response to the burden that electric vehicle charging demands bring to the power grid, technicians have proposed vehicle-to-grid (V2G) technology, which is a new type of power grid technology that can achieve bidirectional, real-time, controllable, and high-speed energy flow between vehicles and the power grid. In this technology, EVs are disguised as energy storage devices that can both obtain electricity from the power grid and transmit electricity to the power grid. They have the dual nature of source and load, and provide energy storage backup for the power grid.

[0004] Applying V2G technology to control the charging and discharging process of electric vehicles can effectively alleviate the negative impact of large-scale access of electric vehicles on the stability of power grid operation; it plays a role in shaving peaks and filling valleys and stabilizing renewable energy. Therefore, it is necessary and urgent to develop V2G technology and implement V2G technology in EV charging and discharging scheduling strategies. However, when applying V2G technology to generate charging and discharging scheduling plans for charging stations, how to balance user needs and the economic benefits of charging stations is still a complex issue. Summary of the invention

[0005] In order to solve the problem that the existing charging and discharging scheduling scheme using V2G technology is difficult to effectively balance customer demand and charging station benefits, the present invention provides a power data scheduling method, system and device based on deep reinforcement learning.

[0006] The present invention is implemented by the following technical solutions:

[0007] A power data dispatching method based on deep reinforcement learning, comprising:

[0008] S1: Establish a mathematical model to characterize the optimization problem of charging and discharging scheduling of charging stations.

[0009] S2: Model the mathematical problem as a Markov decision process, and build a scheduling optimization network based on the DQN algorithm model to solve the mathematical model. In the scheduling optimization network, define actions, states, and rewards as follows:

[0010] (1) Action a of the control center of the charging station at time t t Including charging price, discharging price, as well as the charging power and discharging power of each charging pile.

[0011] (2) The state s of the control center of the charging station at time t t Contains global status information of the charging station And the user status information of the charging pile Both meet:

[0012] in, and They represent the charging power and discharging power of the mth charging pile at time t respectively; and They represent the charging price and discharging price of the charging station at time t respectively; m∈M, M represents the set of charging piles in the charging station; represents the expected SOC information of the i-th user; represents the expected departure time of the i-th user; represents the energy demand of user i, i∈N, and N represents the user set.

[0013] (3) The reward r of the control center of the charging station at time t t Satisfy the following formula:

[0014]

[0015] In the above formula, Indicates the economic benefits of the charging station; represents the service satisfaction of the i-th user; α and β are and The weight of represents the power purchased by the charging station from the distribution network at time t; Indicates the real-time electricity price in the electricity market; represents the time when the i-th user enters the station; represents the shortest charging time of the i-th user's vehicle.

[0016] S3: Train the dispatch optimization network based on the operating data of the charging station, so that it can adaptively obtain better control strategies in continuous decision-making and learning to maximize the economic benefits of the charging station and the service satisfaction of users.

[0017] S4: When any vehicle arrives at the charging station and makes a charging request, the control center of the charging station obtains the current vehicle demand information, including: entry time, expected exit time, charge state when the vehicle arrives, and expected charge state when leaving, and then uses the trained scheduling optimization network to generate the best charging and discharging plan that meets the current user needs.

[0018] As a further improvement of the present invention, in the mathematical model characterizing the charging and discharging scheduling optimization problem of the charging station established in step S1, the maximization of the overall profit F of the charging station is taken as the optimization goal; the charging and discharging power corresponding to each charging and discharging behavior, and the charging and discharging electricity price including the service fee are taken as decision variables; the battery capacity of the vehicle itself, the charging and discharging power constraints of the vehicle and the charging pile, and the service electricity price constraints are taken as overall constraints.

[0019] As a further improvement of the present invention, in the mathematical model, the optimization target is:

[0020]

[0021]

[0022] In the above formula, T represents the operating time of the charging station; Represents the basic load of the charging station interacting with the grid.

[0023] As a further improvement of the present invention, in the mathematical model, the expression of the battery capacity constraint of each vehicle is as follows:

[0024]

[0025] In the above formula, Cap represents the battery capacity of the current vehicle; SoC t Indicates the current real-time charge state of the vehicle; η chg and η dhg Respectively represent the charging energy efficiency and discharging energy efficiency between the vehicle and the charging pile; SoC min and SoC max Respectively represent the lower and upper limits of the vehicle's state of charge; SoC d Indicates the current desired state of charge of the vehicle.

[0026] As a further improvement of the present invention, in the mathematical model, the expressions of the charging and discharging power constraints of the vehicle and the charging pile are as follows:

[0027]

[0028] In the above formula, and They respectively represent the maximum charging power and maximum discharging power allowed by the charging pile to which the vehicle is currently connected; Indicates the maximum instantaneous power that the charging station can withstand.

[0029] As a further improvement of the present invention, in the mathematical model, the expression of the service electricity price constraint is as follows:

[0030]

[0031] In the above formula, and is the upper and lower limits of the charging electricity price; E t sc is the slack variable of charging electricity price; and is the upper and lower limits of the discharge electricity price; is the slack variable of the discharge electricity price.

[0032] As a further improvement of the present invention, in step S2, the shortest charging time T of any vehicle is need It is used to ensure that the vehicle's state of charge can meet expectations before the expected departure time. The calculation formula is as follows:

[0033]

[0034] The energy demand P of any vehicle req The calculation formula is as follows:

[0035] P req =(SoC d -SoC a )·Cap;

[0036] In the above formula, SoC a Indicates the charge status of the vehicle when it arrives at the station.

[0037] As a further improvement of the present invention, in step S3, the training process of the scheduling optimization network is as follows:

[0038] S31: Initialize the training parameters of the DQN algorithm, including learning rate, reward discount factor, exploration probability, training step size and number of iterations; define the size of the experience replay pool and small batch data; initialize the system environment and obtain the initial state of the charging station.

[0039] S32: According to the output of the online neural network and the greedy strategy, the charging and discharging behavior control is performed during the operation of the charging pile, and then the charging and discharging power and electricity price at the current moment are generated to obtain the corresponding action a t .

[0040] S33: According to the action a t Determine the state of the environment at the next moment s t+1 , and calculate the reward r of the action t .

[0041] S34: storing the associated parameters of the iteration process as experience data in the experience replay pool, the experience data includes: the current state s t+1 , action a t , reward r t and the next state s t+1 .

[0042] S35: Randomly select small batches of data from the experience replay pool and transmit them to the Online neural network and the Target neural network;

[0043] S36: The Target network takes the small batch data as input and outputs the target Q value. The Online network takes the small batch data as input and outputs the estimated Q value. The error loss is calculated based on the target Q value and the estimated Q value.

[0044] S37: Iterate steps S32-S36 and perform reverse gradient propagation according to the error loss to update the model parameters of the Online neural network and the Target neural network until a preset number of iterations is reached.

[0045] The present invention also includes a power data dispatching system based on deep reinforcement learning, which includes a demand collection unit, a decision unit and an execution unit.

[0046] The demand collection unit is used to obtain demand information of any arriving vehicle, including: the time of entry, expected exit time, the charge state of the vehicle when it arrives, and the expected charge state when it leaves.

[0047] The decision-making unit runs a dispatch optimization network trained using the aforementioned deep reinforcement learning-based power data dispatch method. The dispatch optimization network is used to generate a charging and discharging plan that can meet user needs and maximize the economic benefits of the charging station and user service satisfaction based on the operating status of each charging pile in the charging station and user demand information.

[0048] The execution unit is used to allocate an idle charging pile to the user, and after the user's vehicle is connected to the designated charging pile, provide services to the vehicle according to the charging and discharging plan generated by the decision unit.

[0049] The present invention also includes a charging station scheduling device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates an electric power data scheduling system based on deep reinforcement learning as described above, thereby interacting with users and providing customers with vehicle charging and discharging services that meet their needs.

[0050] The technical solution provided by the present invention has the following beneficial effects:

[0051] The present invention models the charging station scheduling problem in the V2G charging environment as a single-objective optimization model, and then uses the DQN algorithm to design and train a scheduling optimization network. The scheduling optimization network automatically generates a vehicle charging and discharging plan that can meet customer needs and maximize the interests and environmental benefits of the charging station based on the differentiated needs of vehicles and the real-time operating status of the charging station.

[0052] The charging pile control center using the solution of the present invention adopts a control strategy for adaptive adjustment of charging and discharging prices and charging and discharging powers, and has the characteristics of autonomous learning, dynamic adaptation and rapid convergence. Compared with the current mathematical optimization methods such as convex optimization, which have problems such as high computational complexity and poor convergence, the present invention has great advantages in terms of convergence speed and convergence performance. The proposed method models the charging station control center as an intelligent body, constructs a linear weighted reward function related to EV charging and discharging behavior, charging and discharging power, charging and discharging prices, and charging time, so that the intelligent body can adaptively improve the strategy, realize the intelligence and efficiency of EV charging and discharging decision-making schemes, thereby achieving the goal of maximizing the profit and benefit of the charging station and ensuring the performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a system architecture diagram of a charging station control system using V2G technology in Example 1 of the present invention.

[0054] Figure 2 This is a flowchart of the steps of the power data scheduling method based on deep reinforcement learning provided in Example 1 of the present invention.

[0055] Figure 3 This is a schematic diagram of the scheduling optimization network training phase in Example 1 of the present invention.

[0056] Figure 4 This is a system topology diagram of the power data dispatching system based on deep reinforcement learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] Example 1

[0059] This embodiment provides a V2G charging and discharging scheduling method based on deep reinforcement learning (DRL). Figure 1 In the charging station shown, Figure 1 As can be seen from the architecture, the control center of the charging station serves as the control center of the entire system. The control center interacts with the power grid upwards and can realize the downstream consumption of electric energy and the upstream connection to the grid; it interacts with each charging pile and the connected vehicles downwards, thereby controlling each charging pile to charge or discharge the connected vehicles. In the scheme of this embodiment, each EV connected to the charging station accepts the V2G charging and discharging protocol by default. Under this protocol, the charging station can realize flexible control of the charging and discharging behavior of EVs. While accessing the national time-of-uses (TOU) market, it introduces a dynamic charging and discharging price pricing strategy that considers the addition of service fees, thereby helping users obtain more favorable charging prices and more discharge benefits, so as to encourage EV users to participate in the V2G scheduling process. At the same time, the scheme of this embodiment realizes the flexible regulation of the charging and discharging power of EVs in combination with customer needs and the operating status of the charging station, and maximizes the economic benefits of the charging station on the premise of meeting the charge state requirements of customers when leaving the station. In addition to maximizing the benefits of users and charging stations, the charging and discharging scheduling plan finally generated by this embodiment can also fully release the energy storage backup function of EV, contribute to peak load shaving and valley filling for the power grid, and smooth out renewable energy, thereby achieving greater social benefits.

[0060] Specifically, Figure 2 As shown, the power data scheduling method based on deep reinforcement learning provided in this embodiment includes the following steps:

[0061] S1: Establish a mathematical model to characterize the optimization problem of charging and discharging scheduling of charging stations.

[0062] The technical problem to be solved by the technical solution provided in this embodiment is to scientifically dispatch the charging and discharging capacity of the charging station to meet the charging needs of vehicles at the charging station. Under the V2G protocol, this scheduling work involves adaptively adjusting the charging and discharging power and charging and discharging price at different times in the user's charging behavior. From the perspective of charging station management, the essence of solving this problem is to maximize the revenue of the charging station while protecting the interests of customers (lower charging costs, charging volume and time meet requirements). Therefore, this embodiment treats this problem as a single-objective optimization problem (maximizing charging station revenue) and performs mathematical modeling.

[0063] The mathematical model of a typical single-objective optimization problem consists of three elements: decision variables, optimization objectives, and constraints. In the mathematical model that characterizes the charging and discharging scheduling optimization problem of a charging station constructed in this embodiment, the decision variables are the charging and discharging power corresponding to each charging and discharging behavior, and the charging and discharging electricity price including the service fee. This decision variable is the part that the charging station can flexibly adjust according to the operating conditions, among which the charging and discharging power and charging and discharging electricity price corresponding to the charging and discharging behavior can be adjusted through the designated charging pile used by the user. That is, the charging station can dynamically adjust the charging and discharging status and power of the charging pile used by each user according to the actual needs of each user and the overall operating status of the charging station, and provide different users with differentiated electricity prices after the guidance price range when the users settle accounts.

[0064] 1. Optimization goals

[0065] In the mathematical model that characterizes the charging and discharging scheduling optimization problem of the charging station established in this embodiment, the optimization goal is to maximize the overall revenue F of the charging station. The overall revenue F of the charging station is obtained by adding up the fees (income) collected by all charging piles in the station when providing charging services to each user during the operating period, the incentive income (expenditure) paid by the charging piles when providing discharge services to users, and the electricity fee (expenditure) paid when obtaining electricity from the power grid. Therefore, the expression of the optimization goal is as follows:

[0066]

[0067] In the above formula, represents the charging power of the mth charging pile at time t; represents the charging price of the charging station at time t; represents the discharge power of the mth charging pile at time t; represents the discharge price of the charging station at time t; where m∈M, M represents the set of charging piles in the charging station; t∈T, T represents the operating time of the charging station; Indicates the real-time electricity price (TOU) in the electricity market; It represents the power purchased by the charging station from the distribution network at time t.

[0068] In the charging station based on V2G technology in this embodiment, the charging station purchases power from the distribution network. The charging power and discharge power of each charging pile in the charging station are related to the real-time charging power and discharge power of each charging pile in the charging station. The charging station should ensure that the charging and discharging power of the entire power grid is balanced with the energy supply of the power grid. Satisfy the following formula:

[0069]

[0070] In the above formula, represents the basic load of the interaction between the charging station and the power grid, represents the sum of the charging power of all charging piles in the station at time t, It represents the sum of the discharge power of all charging devices in the station at time t.

[0071] 2. Constraints

[0072] In the mathematical model for characterizing the optimization problem of charging and discharging scheduling of charging stations established in this embodiment, the constraints are divided into three categories, namely: (1) the capacity constraints that the battery of the vehicle needs to meet when enjoying the charging and discharging service; (2) the charging and discharging power constraints that the charging pile and the entire charging station need to meet when the charging pile provides charging and discharging services for the vehicle; (3) the service electricity price constraints that the time-of-use electricity price adopted by the system needs to meet when the charging station settles service fees with the user. The specific contents of the three types of constraints are as follows:

[0073] 2.1. Battery capacity constraints

[0074] When driving, the vehicle (EV) needs to consume electric energy, so the state of charge (SoC) of the electric vehicle gradually decreases. After entering the charging station, the battery is no longer driving, and the state of charge of the vehicle is related to the charging or discharging behavior of the vehicle. In the charging mode, the state of charge of the vehicle gradually increases with the charging time, and in the discharging mode, the state of charge of the vehicle gradually decreases with the discharging time. Therefore, considering that the V2G charging station can provide both charging and discharging services, the SoC update expression of the EV is:

[0075]

[0076] Among them, Cap represents the battery capacity of the current vehicle; SoC t-1 Indicates the state of charge of the vehicle before entering the station; SoC t Indicates the state of charge of the vehicle when it leaves the station; Δt indicates the time the vehicle stays in the charging station. η chg Represents the charging energy efficiency between the vehicle and the charging pile; η dhg Indicates the discharge energy efficiency between the vehicle and the charging pile.

[0077] In addition, in order to avoid over-charging and over-discharging of the EV battery pack, the EV's state of charge must also meet the following constraints:

[0078]

[0079] Among them, SoC min Indicates the lower limit of the vehicle's state of charge; SoC max Indicates the upper limit of the vehicle's state of charge.

[0080] This constraint indicates that when the charging pile provides charging services for the vehicle, the charging service should be based on the vehicle's real-time charge state SoC. t Reaching the upper limit of the vehicle's state of charge SoC max When the charging pile provides discharge service for the vehicle, the discharge service should be based on the vehicle's real-time charge state SoC t Reach the lower limit of vehicle state of charge SoC min Stop when.

[0081] Finally, after each EV arrives at the charging station, it needs to provide demand information to the control center of the charging station, including the vehicle's progress time t a , expected departure time t d 、State of charge SoC when the vehicle arrives a , Expected state of charge SoC when leaving d This information is used to control the charging and discharging process between the charging pile and the EV. According to the battery capacity Cap of the EV, the energy demand P after the EV enters the station can be obtained: req The calculation formula is as follows:

[0082]

[0083] At this time, the control center of the charging station can add EVs to the waiting service queue according to the order in which they arrive, and then provide services to vehicles in a timely manner based on different needs. For example, for a vehicle entering the station, the departure time t d The earlier (more urgent the need) and the energy requirement P req The more EVs there are (and the more revenue they generate), the higher their priority in obtaining charging services.

[0084] Therefore, in order to ensure that the charge state of each user entering the station can meet the expected state when leaving the station, the charge state constraint of the EV when arriving at the charging station and leaving can be expressed as follows: Assume that when t = t a When SoC t =SoC a ; then when t=t d When SoC t =SoC d On this basis, in order to ensure that when t = t d When SoC t =SoC d In this embodiment, a minimum charging time T is set for each vehicle's charging and discharging service. need , which satisfies the following formula:

[0085]

[0086] Based on the determined minimum charging time Tneed , the charging station can be used before the EV leaves d -T need At all times, the EV only needs to keep charging without participating in the V2G scheduling of the charging station, thereby ensuring that the vehicle's charge state always meets expectations before leaving.

[0087] 2.2 Charging power constraints

[0088] When a charging station provides charging and discharging services to vehicles, the charging piles and other equipment in the station have their own electrical characteristics. For example, for each charging pile and the vehicle connected to it, there is a maximum charging and discharging power limit when charging and discharging.

[0089] Therefore, the charging pile has the following maximum power constraints:

[0090]

[0091]

[0092] In the above formula, Indicates the maximum charging power allowed by the charging pile to which the vehicle is currently connected; Indicates the maximum discharge power allowed by the charging pile to which the vehicle is currently connected.

[0093] For each charging pile, it can only provide one of the charging and discharging services for the same connected vehicle, and the charging and discharging behaviors of the charging pile cannot exist at the same time. Therefore, this embodiment adds the following charging and discharging behavior constraints of the charging pile based on the characteristic that the charging power and the generated power are not positive at the same time:

[0094]

[0095] This constraint is used to force the product of the charging power and discharging power of each charging pile to be 0, indicating that the same charging pile cannot be charged and discharged at the same time.

[0096] Each charging station is equipped with multiple charging piles. In order to protect the charging facilities of the charging station and the battery life of the EV, the instantaneous power of the charging station should be less than the maximum value it can withstand. Therefore, for the entire charging station, it also needs to meet the following total requirements:

[0097]

[0098]

[0099] In the above formula, Indicates the maximum instantaneous power that the charging station can withstand.

[0100] 2.3. Service electricity price constraints

[0101] In the solution of this embodiment, the charging station encourages EVs to participate in V2G services by adjusting the charging and discharging prices, thereby achieving the purpose of promoting the consumption of new energy and shifting peaks and filling valleys. The charging and discharging behaviors of EV users are all dispatched by the charging station, so the charging station needs to charge users for part of the service fee. Taking into account market factors and user tolerance, this embodiment specifies differentiated charging and discharging electricity prices.

[0102] In actual operation, charging stations should not unilaterally increase charging service electricity prices or reduce discharge service electricity prices in pursuit of corporate profits. Therefore, this embodiment sets the following service electricity price constraints:

[0103] For the user's charging behavior, the corresponding service price constraint expression is:

[0104]

[0105] In the above formula, and The upper and lower limits of the charging electricity price; is the slack variable of charging electricity price.

[0106] For the user's discharge behavior, the corresponding service price constraint expression is:

[0107]

[0108]

[0109] In the above formula, and is the upper and lower limits of the discharge electricity price; is the slack variable of the discharge electricity price.

[0110] S2: Model the mathematical problem as a Markov decision process, and build a scheduling optimization network based on the DQN algorithm model to solve the mathematical model.

[0111] Based on the mathematical model established above to characterize the optimization problem of charging and discharging scheduling of charging stations, this embodiment intends to seek the best solution to this problem through an intelligent method based on the reinforcement learning DQN algorithm. Specifically, this embodiment models the optimization problem as a Markov decision process. At the beginning of each time slot, the charging station control center determines the charging and discharging behavior decisions (including charging and discharging behavior control and power size formulation) and charging and discharging price formulation of all charging piles in the charging station according to the observed state at the previous moment. The solution process of the optimization problem is a series of time series decisions. The decision at the previous moment will affect the behavior of the agent at the next moment, so the optimal decision problem is transformed into the solution of the Markov decision process. Specifically, the control center is modeled as an agent, and its actions include six parts: charging and discharging behavior control (timing of charging and discharging behavior), charging and discharging power formulation (charging and discharging power at different times), and charging and discharging price formulation (time-sharing service electricity price). The agent optimizes actions under different states by maximizing the long-term average return. The reward consists of two parts: charging station revenue and EV user service satisfaction. In particular, the timing of charging and discharging behavior can actually be managed by dynamically adjusting the charging and discharging power. For example, when the timing of charging and discharging is not reached, it means that the charging power or the generating power at this time is 0.

[0112] In summary, in the scheduling optimization network of this embodiment, the actions, states, and rewards of the agent are defined as follows:

[0113] (1) Action

[0114] Action a of the control center of the charging station at time t t Including charging price, discharging price, as well as the charging power and discharging power of each charging pile.

[0115] Specifically, the action a of the control center at time t t The charging and discharging price vector of the charging station and the charging and discharging power vector of the charging pile Composition, that is The two vectors satisfy:

[0116]

[0117]

[0118] in, represents the charging price vector; represents the discharge price vector; represents the charging power vector; represents the discharge power vector.

[0119] In addition, considering that the DQN algorithm is suitable for discrete action space tasks, the above actions all need to be processed discretely.

[0120] (2) Status

[0121] The state s of the control center of the charging station at time t t Contains global status information of the charging station And the user status information of the charging pile Among them, the global status information of the charging station It is reflected in the total charging power, total discharging power, and real-time charging and discharging prices of the charging pile. It is reflected in the three aspects of the vehicle's expected charge state, expected departure time and energy demand. and user status information Satisfy the following formulas respectively:

[0122]

[0123]

[0124] in, and They represent the charging power and discharging power of the mth charging pile at time t respectively; and They represent the charging price and discharging price of the charging station at time t respectively; m∈M, M represents the set of charging piles in the charging station; represents the expected SOC information of the i-th user; represents the expected departure time of the i-th user; represents the energy demand of user i, i∈N, and N represents the user set.

[0125] (3) Rewards

[0126] In the scheme of this embodiment, in order to make the optimized scheduling scheme provide high-quality services to users, enable charging piles to obtain higher profits, and take into account the ecological value of the V2G system, the reward is divided into the economic benefits of the charging station and user service satisfaction Two aspects. That is, the optimized dispatching plan should not only provide revenue for charging stations, but also improve the user experience.

[0127] In particular, in order to quantify the abstract indicator of user satisfaction, this embodiment uses the waiting time of the user in the charging station as an indicator for evaluating user satisfaction. That is, the sooner the user completes the service and leaves, the better the user experience, which is also in line with the actual feelings of most users.

[0128] In summary, in this embodiment, the reward r of the control center of the charging station at time t is defined as t Satisfy the following formula:

[0129]

[0130] In the above formula, Indicates the economic benefits of the charging station; represents the service satisfaction of the i-th user; α and β are and The weight of represents the power purchased by the charging station from the distribution network at time t; Indicates the real-time electricity price in the electricity market; represents the time when the i-th user enters the station; represents the shortest charging time of the i-th user's vehicle.

[0131] S3: Train the dispatch optimization network based on the operating data of the charging station, so that it can adaptively obtain better control strategies in continuous decision-making and learning to maximize the economic benefits of the charging station and the service satisfaction of users.

[0132] like Figure 3 As shown, in this embodiment, the training process of the scheduling optimization network is as follows:

[0133] S31: Initialize the training parameters of the DQN algorithm, including learning rate, reward discount factor, exploration probability, training step size and number of iterations; define the size of the experience replay pool and small batch data; initialize the system environment and obtain the initial state of the charging station.

[0134] S32: According to the output of the online neural network and the greedy strategy, the charging and discharging behavior control is performed during the operation of the charging pile, and then the charging and discharging power and electricity price at the current moment are generated to obtain the corresponding action a t .

[0135] S33: According to the action a t Determine the state of the environment at the next moment s t+1 , and calculate the reward r of the action t .

[0136] S34: storing the associated parameters of the iteration process as experience data in the experience replay pool, the experience data includes: the current state s t+1 , action a t , reward r t and the next state s t+1 .

[0137] S35: Randomly select small batches of data from the experience replay pool and transmit them to the Online neural network and the Target neural network;

[0138] S36: The Target network takes the small batch data as input and outputs the target Q value. The Online network takes the small batch data as input and outputs the estimated Q value. The error loss is calculated based on the target Q value and the estimated Q value.

[0139] S37: Iterate steps S32-S36 and perform reverse gradient propagation according to the error loss to update the model parameters of the Online neural network and the Target neural network until a preset number of iterations is reached.

[0140] S4: When any vehicle arrives at the charging station and makes a charging request, the control center of the charging station obtains the current vehicle demand information, including: entry time, expected exit time, charge state when the vehicle arrives, and expected charge state when leaving, and then uses the trained scheduling optimization network to generate the best charging and discharging plan that meets the current user needs.

[0141] Example 2

[0142] Based on the solution in Example 1, this embodiment also provides a power data dispatching system based on deep reinforcement learning, such as Figure 4 As shown, it includes a demand collection unit, a decision unit and an execution unit.

[0143] The demand collection unit is used to obtain demand information of any arriving vehicle, including: the time of entry, expected exit time, the charge state of the vehicle when it arrives, and the expected charge state when it leaves.

[0144] A scheduling optimization network trained by the power data scheduling method based on deep reinforcement learning as in Example 1 is run in the decision unit. The scheduling optimization network is used to generate a charging and discharging plan that can meet user needs and maximize the economic benefits of the charging station and user service satisfaction based on the operating status of each charging pile in the charging station and user demand information.

[0145] The execution unit is used to allocate an idle charging pile to the user, and after the user's vehicle is connected to the designated charging pile, provide services to the vehicle according to the charging and discharging plan generated by the decision unit.

[0146] Example 3

[0147] Based on the solutions in Examples 1 and 2, this embodiment further provides a charging station scheduling device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a power data scheduling system based on deep reinforcement learning as in Example 2, thereby realizing interaction with users and providing customers with vehicle charging and discharging services that meet their needs.

[0148] The charging station scheduling device provided in this embodiment is essentially a computer device. The computer device can be an embedded computer device or a general-purpose computer device. For example, it can be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server, or a server cluster composed of multiple servers) that can execute programs.

[0149] In one typical application of this example solution, the charging station scheduling device can serve as the management background of the entire charging pile and run on the background server. At this time, when the user makes an appointment request through the cloud APP, the device provides the corresponding scheduling plan and service. In another typical application, the charging station scheduling device can also be embedded in each charging pile, so that when each charging pile receives a user request, it communicates with the background service and provides services according to the scheduling plan it generates.

[0150] The computer device of this embodiment at least includes but is not limited to: a memory and a processor that can be connected to each other through a system bus. In this embodiment, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0151] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of a computer device.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A power data scheduling method based on deep reinforcement learning, characterized in that: It includes: S1: Establish a mathematical model to characterize the optimization problem of charging and discharging scheduling of charging stations; S2: The charging and discharging scheduling optimization problem is modeled as a Markov decision process, and a scheduling optimization network for solving the mathematical model is constructed based on the DQN algorithm model; in the scheduling optimization network, it is defined as: (1) Action a of the control center of the charging station at time t t Including charging price, discharging price, and charging power and discharging power of each charging pile; (2) The state s of the control center of the charging station at time t t Contains global status information of the charging station And the user status information of the charging pile Both meet: in, and They represent the charging power and discharging power of the mth charging pile at time t respectively; and Respectively represent the charging price and discharging price of the charging station at time t; m∈M; represents the expected SOC information of the i-th user; represents the expected departure time of the i-th user; represents the energy demand of the i-th user, i∈N; (3) The reward r of the control center of the charging station at time t t Satisfy the following formula: In the above formula, Indicates the economic benefits of the charging station; represents the service satisfaction of the i-th user; α and β are and The weight of represents the power purchased by the charging station from the distribution network at time t; Indicates the real-time electricity price in the electricity market; represents the time when the i-th user enters the station; represents the shortest charging time of the i-th user’s vehicle; S3: training the scheduling optimization network according to the operation data of the charging station, so that it can adaptively obtain a better control strategy in continuous decision-making and learning, so as to maximize the economic benefits of the charging station and the service satisfaction of the user; S4: When any vehicle arrives at the charging station and makes a charging request, the control center of the charging station obtains the current vehicle demand information, including: the time of entry, the expected time of exit, the charge state of the vehicle when it arrives, and the expected charge state when it leaves, and then uses the trained scheduling optimization network to generate the best charging and discharging plan that meets the current user needs.

2. The power data scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The mathematical model characterizing the charging and discharging scheduling optimization problem of the charging station established in step S1 takes maximizing the overall revenue F of the charging station as the optimization goal; takes the charging and discharging power corresponding to each charging and discharging behavior, and the charging and discharging electricity price including the service fee as the decision variables; and takes the battery capacity of the vehicle itself, the charging and discharging power constraints of the vehicle and the charging pile, and the service electricity price constraints as the overall constraints.

3. The power data scheduling method based on deep reinforcement learning according to claim 2, characterized in that: In the mathematical model, the optimization objective is: In the above formula, T represents the operating time of the charging station; Represents the basic load of the charging station interacting with the grid.

4. The power data scheduling method based on deep reinforcement learning according to claim 3, characterized in that: In the mathematical model, the expression of the battery capacity constraint of each vehicle is as follows: In the above formula, Cap represents the battery capacity of the current vehicle; SoC t Indicates the current real-time charge status of the vehicle; η chg and η dhg Respectively represent the charging energy efficiency and discharging energy efficiency between the vehicle and the charging pile; SoC min and SoC max Respectively represent the lower and upper limits of the vehicle's state of charge; SoC d Indicates the current expected state of charge of the vehicle.

5. The power data scheduling method based on deep reinforcement learning according to claim 4, characterized in that: In the mathematical model, the expressions of the charging and discharging power constraints of the vehicle and the charging pile are as follows: In the above formula, and They respectively represent the maximum charging power and maximum discharging power allowed by the charging pile to which the vehicle is currently connected; Indicates the maximum instantaneous power that the charging station can withstand.

6. The power data scheduling method based on deep reinforcement learning according to claim 5, characterized in that: In the mathematical model, the expression of the service electricity price constraint is as follows: In the above formula, and The upper and lower limits of the charging electricity price; is the slack variable of charging electricity price; and is the upper and lower limits of the discharge electricity price; is the slack variable of the discharge electricity price.

7. The power data scheduling method based on deep reinforcement learning according to claim 6, characterized in that: In step S2, the shortest charging time T for any vehicle need It is used to ensure that the vehicle's state of charge can meet expectations before the expected departure time. The calculation formula is as follows: The energy demand P of any vehicle req The calculation formula is as follows: P req =(SoC d -SoC a )·Chap; In the above formula, SoC a Indicates the charge status of the vehicle when it arrives at the station.

8. The power data scheduling method based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the training process of the scheduling optimization network is as follows: S31: Initialize the training parameters of the DQN algorithm, including learning rate, reward discount factor, exploration probability, training step size and number of iterations; define the size of the experience replay pool and mini-batch data; Initialize the system environment and obtain the initial status of the charging station; S32: According to the output of the online neural network and the greedy strategy, the charging and discharging behavior control is performed during the operation of the charging pile, and then the charging and discharging power and electricity price at the current moment are generated to obtain the corresponding action a t ; S33: According to the action a t Determine the state of the environment at the next moment s t+1 , and calculate the reward r of the action t ; S34: storing the associated parameters of the iterative process as experience data in the experience playback pool, wherein the experience data includes: the current state s t+1 , action a t , reward r t and the next state s t+1 ; S35: Randomly select small batches of data from the experience replay pool and transmit them to the Online neural network and the Target neural network; S36: The Target network takes the small batch data as input and outputs the target Q value. The Online network takes the small batch data as input and outputs the estimated Q value. The error loss is calculated based on the target Q value and the estimated Q value. S37: Iterate steps S32-S36 and perform reverse gradient propagation according to the error loss to update the model parameters of the Online neural network and the Target neural network until a preset number of iterations is reached.

9. A power data dispatching system based on deep reinforcement learning, characterized in that: It includes: A demand collection unit is used to obtain demand information of any arriving vehicle, including: the time of entry, expected departure time, the state of charge of the vehicle when it arrives, and the expected state of charge when it leaves; A decision unit, in which a scheduling optimization network trained by the power data scheduling method based on deep reinforcement learning as described in any one of claims 1 to 8 is run; the scheduling optimization network is used to generate a charging and discharging plan that can meet user needs and maximize the economic benefits of the charging station and user service satisfaction according to the operating status of each charging pile in the charging station and user demand information; The execution unit is used to allocate an idle charging pile to the user, and after the user's vehicle is connected to the designated charging pile, provide services to the vehicle according to the charging and discharging plan generated by the decision unit.

10. A charging station dispatching device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and running on the processor, and is characterized in that: when the processor executes the computer program, it creates an electric power data dispatching system based on deep reinforcement learning as described in claim 9, thereby interacting with users and providing customers with vehicle charging and discharging services that meet their needs.

Images