A two-stage scheduling method for coordinated distribution network ancillary services of electric vehicles based on deep reinforcement learning

Abstract

This invention discloses a two-stage scheduling method for coordinated distribution network ancillary services for electric vehicles based on deep reinforcement learning. The method includes: establishing a scheduling strategy model based on a distribution network model, an electric vehicle battery dynamic characteristic model, a charging price incentive model, and a discharging price incentive model; training the scheduling strategy model using a two-stage training method based on a Gaussian mixture model; during scheduling, using a K-means clustering method based on electric vehicle energy availability criteria to select electric vehicles for distribution network ancillary services; and establishing a data simulator, performing pre-training using a mixture of real and simulated data in the offline stage, and training using real-time data in the online stage. The scheduling method proposed in this invention can effectively improve training speed and model robustness, and reduce accumulated voltage errors and the proportion of illegally charged electric vehicles during online startup, thereby solving the problems of real-time peak shaving and voltage regulation.

Description

Technical Field

[0001] This invention relates to the technical field of coordinated distribution network dispatching services, and in particular to a two-stage dispatching method for coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning. Background Technology

[0002] Compared to traditional fuel-powered vehicles, electric vehicles (EVs) are gaining increasing attention worldwide as an effective way to reduce greenhouse gas emissions. According to a report by the International Energy Agency, by 2030, 13% of cars globally will be electric, with demand for EVs growing at an average annual rate of 36% between 2021 and 2030. China's EV fleet is projected to reach 60 million vehicles, with peak charging load reaching 479 GW, making China the largest producer and consumer of EVs. The State Council, in its "New Energy Vehicle Industry Development Plan (2021-2035)," explicitly promotes the comprehensive and deep integration of new energy vehicles with energy, transportation, and information communication, enhances the energy and transportation system and urban intelligence, builds a new pattern of coordinated industrial development, and improves the infrastructure system, accelerating the construction of charging, battery swapping, and hydrogen refueling infrastructure. Electric vehicles, as a mobile and flexible energy resource, have become a typical representative of large-scale distributed energy and a significant feature of the next generation of power systems.

[0003] However, the widespread adoption of electric vehicles (EVs) will have significant negative impacts on the safe and economical operation of the power grid. Uncoordinated large-scale EV charging and its random behavior can severely burden the distribution network. Unmanaged EV charging increases peak-hour load demand, potentially causing equipment and line congestion and seriously threatening the stable operation of the power system. A large number of EVs within a specific timeframe and area will lead to a surge in electricity demand, increasing line losses and causing voltage drops. Once the voltage drops to a certain level, the grid will be unable to supply power. With the development of active power control technology and four-quadrant smart charger technology in Vehicle-to-Grid (V2G), EVs can also serve as flexible energy storage resources to provide ancillary services to the power grid, playing a crucial role in the stable operation of the distribution network. Therefore, designing a coordinated dispatch scheme for EVs to coordinate distribution network ancillary services, thereby aiding the economic and safe operation of the distribution network, is of great value for my country's construction of a flexible, resilient, and intelligently interactive next-generation power grid.

[0004] Regulatory schemes for electric vehicle (EV) charging ancillary services can be categorized into two types: EV charging coordination and price incentives. However, existing schemes only consider one of these methods. Due to the strong correlation between these two approaches, the proposed schemes lack feasibility. Furthermore, current technologies can only address single-service grid operations and lack a universal solution for EV participation in peak shaving, voltage regulation, and congestion management. In addition, existing schemes use a crude method to evaluate the demand response of EV users, employing only a simple linear model to describe users' actual reactions to price incentives, and neglecting the impact of EV charge state and EV owner charging demand on the willingness to participate in grid ancillary services. This makes it difficult for existing schemes to achieve good results in actual operation.

[0005] Existing methods typically use the real-time state of charge (SOC) and arrival / departure times of electric vehicles (EVs) as criteria for clustering algorithms to select EVs for participation in grid ancillary services. However, this does not accurately reflect the scheduling priority of EVs. For EVs with large battery capacities, a high SOC does not necessarily mean a short charging time, while for EVs with small capacities, a short parking time does not necessarily indicate an urgent charging task. Existing solutions fail to establish effective criteria for EVs to participate in grid ancillary services to ensure the charging needs of EV users.

[0006] Existing electric vehicle (EV) scheduling schemes are based on traditional model-based optimization methods, relying on complete and accurate information about EV states and the power grid, which is difficult to obtain in reality. Due to the dynamic nature of the operating environment over time and the uncertainty of EV charging behavior, model-based methods may not be able to effectively solve complex optimization problems. As the scale of the distribution network and the number of EVs increases, the computational complexity of model-based methods will impose significant computational costs on system operators. Summary of the Invention

[0007] The purpose of this invention is to solve the technical problems of electric vehicles participating in real-time peak shaving and voltage regulation.

[0008] To this end, the present invention proposes a two-stage scheduling method for electric vehicle coordinated distribution network auxiliary services based on deep reinforcement learning, comprising the following steps:

[0009] A scheduling strategy model is established based on a distribution network model, an electric vehicle battery dynamic characteristic model, a charging price incentive model, and a discharging price incentive model. This scheduling strategy model is trained using a two-stage training method based on a Gaussian mixture model (GMM) for coordinating the scheduling of electric vehicles providing distribution network ancillary services. During scheduling, a K-means clustering method based on electric vehicle energy availability criteria is used to select electric vehicles for distribution network ancillary services. The two-stage training method based on the GMM employs a Gaussian mixture model to fit the distribution of charging behavior characteristics for each electric vehicle user. This Gaussian mixture model decomposes the charging behavior characteristics of electric vehicle users into multiple Gaussian-based probability density functions. After establishing a data simulator based on the Gaussian mixture model, pre-training is performed offline using a mixture of real and simulated data, and online training is performed using real-time data.

[0010] In some embodiments of the present invention, real data is divided into three groups for offline training, online training, and testing processes, respectively. After the two training phases, the method is tested using untrained real data.

[0011] In some embodiments of the invention, isotropic truncated Gaussian noise is added to each feature in more than 20% of the simulated data, and selected types of data are replaced at random proportions.

[0012] In some embodiments of the present invention, a relaxation time is defined for each electric vehicle to assess its scheduling potential, and a relaxation time penalty term is set in the reward function to meet the charging needs of the electric vehicle owner.

[0013] In some embodiments of the present invention, the electric vehicle battery dynamic characteristic model implements: describing the charging / discharging process of the electric vehicle, establishing an electric vehicle energy storage model for the arrival / departure time phase of the electric vehicle, and describing the SOC at departure time to meet the energy demand of the electric vehicle user.

[0014] In some embodiments of the present invention, the charging price incentive model obtains a price-participation curve through interpolation, and uses a neural network to fit the curve to obtain a nonlinear functional relationship between charging price and electric vehicle user participation.

[0015] In some embodiments of the present invention, the discharge price incentive model calculates the degradation cost of the electric vehicle battery vehicle-to-grid (V2G) mode, the revenue of the electric vehicle vehicle-to-grid (V2G) process, and the revenue of the electric vehicle charging station, and provides a discharge price incentive scheme based on the Weber-Fechner law.

[0016] In some embodiments of the present invention, the scheduling strategy uses a real-time peak shaving and voltage regulation scheme based on deep reinforcement learning, wherein the controller sets the charging price to adjust the expected number of electric vehicles arriving, and sets the discharging price to incentivize electric vehicle users to participate in vehicle-to-grid (V2G) services.

[0017] In some embodiments of the present invention, the training based on the Gaussian mixture model (GMM) adopts a two-stage reinforcement learning framework. First, the electric vehicle coordinated distribution network auxiliary service scheduling problem is formulated as a Markov decision process to achieve peak shaving, voltage regulation and congestion management.

[0018] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the scheduling method for two-stage electric vehicle coordinated distribution network auxiliary services based on deep reinforcement learning as described above.

[0019] The present invention has the following beneficial effects:

[0020] The proposed invention presents a two-stage scheduling method for coordinated electric vehicle distribution network ancillary services based on deep reinforcement learning. By establishing a scheduling strategy model based on a distribution network model, an electric vehicle battery dynamic characteristic model, a charging price incentive model, and a discharging price incentive model, and by setting technical features such as training the scheduling strategy model using a two-stage training method based on a Gaussian mixture model (GMM), the method can effectively improve training speed and model robustness, and reduce the cumulative voltage error and the proportion of illegally charged electric vehicles during online startup, thereby solving the problems of real-time peak shaving and voltage regulation.

[0021] In some embodiments of the present invention, by adding isotropic truncated Gaussian noise to each feature in more than 20% of the simulated data and replacing certain types of data in a random proportion, the excessive concentration of the distribution can be avoided, thereby improving the robustness of the model.

[0022] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method in an embodiment of the present invention;

[0024] Figure 2 This is a graph showing the relationship between total charging price and charging willingness in Embodiment 1 of the present invention;

[0025] Figure 3 This is a graph showing the relationship between profit and willingness to discharge in Embodiment 1 of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0027] It should be noted that the directional terms such as left, right, up, down, top, and bottom used in this embodiment are only relative concepts or are based on the normal use of the product, and should not be considered as restrictive.

[0028] The following embodiments of the present invention propose a two-stage scheduling method for coordinated distribution network ancillary services for electric vehicles based on deep reinforcement learning, comprising the following steps: establishing a scheduling strategy model based on a distribution network model, an electric vehicle battery dynamic characteristic model, a charging price incentive model, and a discharging price incentive model; the scheduling strategy model is trained using a two-stage training method based on a Gaussian mixture model (GMM) for use in scheduling electric vehicles for coordinated distribution network ancillary services; during the scheduling process, a K-means clustering method based on the energy availability standard of electric vehicles is used to select electric vehicles for distribution network ancillary services; wherein, the two-stage training method based on the Gaussian mixture model adopts the Gaussian mixture method, which uses a Gaussian mixture model to fit the distribution of charging behavior characteristics of each electric vehicle user, and the Gaussian mixture model decomposes the charging behavior characteristics of electric vehicle users into multiple Gaussian-based probability density functions; after establishing a data simulator based on the Gaussian mixture model, in the offline stage, the data simulator is pre-trained using a mixture of real data and simulated data, and in the online stage, real-time data is used for training.

[0029] This invention presents a two-stage scheduling method for coordinated electric vehicle (EV) distribution network ancillary services based on deep reinforcement learning, aiming to address real-time peak shaving and voltage regulation issues. This invention proposes a novel modeling method for EV user participation intentions, measuring the impact of multi-attribute attitudes on EV user choices while simultaneously considering the profits of EV users, charging stations, and the power grid. This invention defines a relaxation time for each EV to assess its scheduling potential and ensures that EV owners' charging needs are met by setting a relaxation time penalty term in the reward function. It also proposes an EV energy availability criterion based on K-means clustering, providing an effective EV state classification standard for large-scale EV scheduling. Furthermore, this invention proposes a two-stage training method based on Gaussian Mixture Model (GMM), which effectively improves training speed and model robustness, and reduces accumulated voltage errors and the proportion of illegally charging EVs during online startup. The method flow of this invention is as follows: Figure 1 As shown, it includes the following steps:

[0030] Model 1

[0031] 1.1 Distribution Network Model:

[0032] The distribution network is defined as follows: B = {0, 1, 2, ..., b} represents the set of buses. Let L represent the set of lines. For each (m,n)∈L, m is the unique parent node of bus n. The radial distribution network model is as follows:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] Γ={t q |t q =qΔt,q∈Z *} (7)

[0040] Where E n P is the set of electric vehicles on bus n. mn,t I mn,t It represents the real-time active power and current of line mn, P nq,t This is the real-time active power of line nq. Γ and It is the discrete-time set of the power grid operation and the parking of electric vehicles at charging stations (CS). Q mn, Q is the real-time reactive power of line mn. nq, This is the real-time reactive power of line nq. Δt is the time interval. These represent the active and reactive power outputs from charging stations, non-electric vehicle loads, and local solar photovoltaic systems, respectively. Active power for charging / discharging electric vehicles. It is the reactive power consumed during the charging process, and It refers to the reactive power support provided by electric vehicles to the power grid. This refers to the number of electric vehicles in V2G and RPC (Reactive Power Control) modes. This represents the number of electric vehicles in standard charging mode. V m,n / V n,t It is the voltage of bus m / n, r mn / x mn It is the impedance / inductive reactance of the line (m,n), Smn,t The real-time apparent power of line mn. (1) and (2) are the power balance of the distribution network linearization. (3) and (4) are... and The composition. (5) and (6) are voltage and power constraints, where V min / V max It is the minimum / maximum voltage difference between the two buses.

[0041] 1.2 Dynamic Characteristic Model of Electric Vehicle Battery:

[0042] The dynamic characteristic model of each electric vehicle battery is as follows:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] in It refers to charging / discharging efficiency. It's the energy from electric vehicles. It refers to the battery capacity of the electric vehicle. It is a binary charge / discharge state. Maximum charge and discharge power at that time It is the minimum / maximum SOC of electric vehicle batteries. and For charging / discharging, reactive power, and maximum apparent power. It is the charger power factor. It is the SOC at the time of departure. This is the expected energy state after charging. (8)-(14) describe the charging / discharging process of electric vehicles and establish a system based on... Electric vehicle energy storage model during arrival / departure time phases. (15) indicates that the SOC at departure should meet the energy demand of electric vehicle users.

[0053] 1.3 Charging Price Incentive Model

[0054] This invention assumes that the charging / discharging price and service quality are the same at charging stations on the same bus route, and that electric vehicle owners will only choose charging stations on the bus routes where they work and live. The charging price reasonableness curve for each electric vehicle user is an S-shaped curve that remains constant over a period of time. The formula for the charging price reasonableness curve within time t is as follows:

[0055]

[0056]

[0057] Among them, R n,t This represents the reasonableness of the charging fee charged by the electric vehicle charging station at bus n, with a value range of [0,1]. Total charging price It is the price of charging. Service Fee The sum. Parameter a n,t ,b n,t and c n,t The shape of the curve is determined by these factors. The charging price rationality curves for each electric vehicle owner are only affected by local historical charging prices and the general price level; therefore, the rationality curves for each electric vehicle owner are generally similar. Evaluation information of electric vehicle owners regarding their respective bus charging stations is collected on each bus route. Based on the Fishbein model and the collected information, the charging preference F of electric vehicle owners for their respective bus charging stations is determined. n,i,t The calculation formula is as follows:

[0058]

[0059]

[0060]

[0061] in It is a collection of electric vehicles that participated in the survey. Let g represent the satisfaction level of electric vehicle owners with the j-th attribute of electric vehicle charging stations, where g is the number of attributes. This indicates the degree of preference of electric vehicle owners for attribute j of electric vehicle charging stations. These attributes include service fees, service quality, geographical location, and average waiting time within time period t, etc. It largely depends on the price sensitivity of electric vehicle owners.

[0062] For electric vehicle owners, the reasonableness threshold of local charging station fees is as follows: The definition is as follows:

[0063]

[0064]

[0065] Among them, S n,i,t It is the pressure on car owners to lose battery power, measuring the impact of changes in SOC on user charging behavior. This is the threshold for the battery energy level that influences the user's charging decision. n,i,t It is a coefficient that reflects a user's resilience to low battery anxiety.

[0066] Discrete total charging price point Expected charging participation rate The description is as follows:

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] H={h∈Z * h≤N h}(29)

[0073] N represents the number of car owners participating in charging at discrete price point l. n,s This represents the total number of electric vehicle owners surveyed. It is a binary variable for charging decisions. δ is the distance between adjacent discrete price points that is greater than 0. It is the lowest price at which charging stations purchase electricity from power plants. It is the highest charging price, N h This represents the number of discrete price points. An interpolation method is used to obtain the price-participation curve, and a neural network is used to fit this curve to obtain a non-linear functional relationship between charging prices and electric vehicle user participation.

[0074] 1.4 Discharge Price Incentive Model

[0075] Existing chargers not only implement V2G functionality but also bidirectional Reactive Power Control (RPC). Since electric vehicle batteries experience almost no degradation in RPC mode and can still charge at normal power, electric vehicle aggregators do not need to pay incentive fees for users' RPC mode. When setting discharge prices for V2G (Vehicle to Grid) mode, embodiments of this invention should comprehensively consider the interests of both electric vehicle charging stations and electric vehicle owners. It is assumed that charging stations can obtain some information from vehicle owners, including basic battery information such as SOC (State of Charge).

[0076] 1) Degradation costs of electric vehicle batteries in V2G (Vehicle to Grid) mode

[0077] Due to the nonlinear degradation process of lithium-ion batteries, the cost model for electric vehicle batteries is a nonlinear model related to the depth of discharge, discharge power, and other factors. This includes the degradation cost of electric vehicle batteries during the V2G (Vehicle to Grid) process. This can be simplified as follows:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] in and It is the set of electric vehicles and the discrete-time set in the V2G mode. It is the price per unit of battery wear. It is the discharge energy within time period t. It's an investment in electric vehicle batteries. and These are DOD (Depth of Discharge) and the number of battery cycles, respectively.

[0084] 2) Benefits of V2G process for electric vehicles

[0085] The charging price for electric vehicles is based on the charging price at the time of arrival, while the discharging price changes every time period t. This embodiment of the invention assumes that all charging stations have the same charging and discharging power. The benefits of each electric vehicle owner in the V2G (Vehicle to Grid) process are discussed. as follows:

[0086]

[0087]

[0088]

[0089] in, The profit comes from electric vehicle users participating in V2G services. It is the profit from electric vehicle users participating in V2G (Vehicle to Grid) services. It is the cost of recharging the released energy. It is the total charging price for the arrival time of the electric vehicle. It is the discharge price of CS. It is the charging energy during time phase t.

[0090] 3) Revenue from electric vehicle charging stations

[0091] Higher charging prices will attract more electric vehicle owners to participate in V2G (Vehicle to Grid) services, but will also directly impact the revenue of charging stations. Total revenue of charging stations. The definition is as follows:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] in and It is a collection of electric vehicles in RPC (Reactive Power Control) mode and standard charging mode. It is the total charging price when the car owner arrives. This represents the number of electric vehicles currently charging, as shown in (41). This refers to the number of electric vehicles that are in normal charging mode and do not participate in V2G or RPC modes. It is the time set of peak reduction, as shown in (48), where It is the start / end time period. It's the profit from charging. and It refers to the cost and price of electricity. It is the profit that CSs make from selling energy to the grid. It is the feedback electricity price paid by the power grid. If the actual output does not reach 70% of the bid volume The punishment It's a penalty price. It is the V2G cost of purchasing energy emitted by users. It is the number of electric vehicles that are in a charging state, as shown in (41). It is the total discharge power of the charging station. (44) At the highest price of power generation cost. This serves as a constraint to ensure the profitability of the public sector of the power grid.

[0104] 4) Discharge price incentive scheme based on Weber-Fechner law

[0105] The Weber-Fechner law accurately expresses the functional relationship between the human body's response *s* and the objective environmental stimulus *I*. It was first applied in the fields of psychology and acoustics.

[0106] The Weber-Fechner law states that the magnitude of a sensation is directly proportional to the logarithm of the stimulus intensity. As stimulus intensity increases geometrically, sensation intensity increases arithmetically. The Weber-Fechner method is defined as follows:

[0107] s=klog(I)+s0(49)

[0108] Where s is the intensity of human sensation, I is the intensity of external environmental stimuli, k is the Weber index, which is sensory specific and should be determined based on the sensation and type of the stimulus, and s0 is the integral constant of the stimulus.

[0109] Assuming that the parameters of each electric vehicle charging station are identical, the expected unit profit per user for V2G (Vehicle to Grid) services can be expressed as follows:

[0110]

[0111] The user's expected average unit profit can be expressed as:

[0112]

[0113]

[0114]

[0115] in It is the average total charging price. It is the average price per unit of battery wear. Because Generally very small, so the battery wear price of a regular electric vehicle model is w L It can be used as its numerical value.

[0116] In this embodiment of the invention, the stimulus intensity I is the average benefit per unit of discharge energy, and the human perceived intensity s represents the average willingness of electric vehicle owners to participate in V2G (Vehicle to Grid) services. According to the Weber-Fechner law, the following relationship can be obtained from the embodiments of the present invention:

[0117]

[0118]

[0119]

[0120] in It is the lower / upper limit of the effective profit incentive range. (55) is the boundary condition, where s 0n, It is the greatest desire, k n,t This is a coefficient reflecting users' sensitivity to average profit. When At that point, the average unit profit will fall into a dead zone, thus preventing users from participating in V2G services. With... As the number of people increased, their willingness gradually strengthened until... It will reach saturation at that time. If we disregard the impact of travel chains, Compared to the average income of local electric vehicle users Inversely proportional, and The opposite is true.

[0121] 2 Execution Scheme Based on Two-Stage Deep Reinforcement Learning

[0122] 2.1 Scheduling Strategy

[0123] Figure 1 This paper presents a proposed real-time peak shaving and voltage regulation scheme based on deep reinforcement learning. The controller sets charging prices to adjust for the expected number of arriving electric vehicles (EVs) and discharging prices to incentivize user participation in V2G services. For EV users, a K-means clustering method based on EV energy availability criteria is proposed to select EVs for V2G services. The controller then selects the number of EVs operating in either V2G (Vehicle to Grid) or RPC (Reactive Power Control) mode. The controller is pre-trained offline using a simulator with mixed data and then rapidly trained online using real-time data.

[0124] 2.2 K-means method based on electric vehicle energy availability standard

[0125] Traditional methods typically use real-time State of Charge (SOC) and departure / stop time as criteria for clustering algorithms to select electric vehicles for V2G (Vehicle to Grid) services. This does not accurately reflect the scheduling priority of electric vehicles. For electric vehicles with large battery capacities, a high SOC does not necessarily mean a short charging time, while for electric vehicles with small capacities, a short stopping time may not indicate an urgent charging task. To address these issues, this invention proposes an electric vehicle energy availability criterion for clustering algorithms:

[0126]

[0127]

[0128] in and χ n,i,t It consists of remaining charging energy and relaxation time. The expected State of Charge (SOC) for vehicle owners is considered. The reason this embodiment of the invention does not classify electric vehicles solely based on relaxation time is to reduce the risk of uncertainty, including user premature departures and other load fluctuations. The minimization objective function of the K-means algorithm is written as follows:

[0129]

[0130]

[0131] Where c i and u k K represents the sample and cluster center. n,t and N n,tIt represents the number of electric vehicles in the cluster centers and charging stations.

[0132] 2.3 Two-stage reinforcement learning framework

[0133] (1) The problem of scheduling ancillary services for electric vehicle distribution networks is formulated as a Markov decision process.

[0134] 1) Peak shaving and voltage regulation:

[0135] Environment variables s t :s t It refers to the status information of the charging station and bus observed during time phase t, including P. n,t Q n,t V n,t , and in It is the number of electric vehicles that arrived. It is the minimum relaxation time.

[0136] Action variable a t :a t It is the control action given by the deep reinforcement learning controller at time stage t, including The values ​​of action variables are pre-constrained based on prior knowledge.

[0137] Reward r t The reward function includes charging station profits and peak-shaving output, voltage, and SOC constraints, and consists of the following three parts:

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] (61) represents profit and peak-shaving incentives, (63) represents SOC constraints, (64) represents cumulative voltage error, and (65) represents node voltage constraints. Among these, All of these are reward functions. τ1, τ2, τ3, τ4, and τ5 are the positive coefficients designed in the reward function, and pu is the universal unit of per-unit value.

[0145] 2) Blockage Management:

[0146] Environment variables s t :st It refers to the status information of the charging station and bus observed during time phase t, including P. n,t Q n,t V n,t , and in It is the number of electric vehicles that arrived. It is the minimum relaxation time.

[0147] Action variable a t :a t It is the control action given by the deep reinforcement learning controller at time stage t, including The values ​​of action variables are pre-constrained based on prior knowledge.

[0148] Reward r t , c The reward function includes line congestion rate, voltage, and SOC (State of Charge) constraints, and consists of the following three parts:

[0149]

[0150]

[0151]

[0152]

[0153] (67) is the blocking reward, c n,t For the real-time load occupancy rate of the line, (68) is the SOC (State of Charge) constraint. (69) constrains the node voltage. υ1 and υ3 are the positive coefficients designed in the reward function under the congestion scenario. υ4 and υ5 are the positive coefficients designed in the reward function.

[0154] (2) Two-stage training

[0155] During the initial training phase of reinforcement learning, insufficient sampling data means the agent knows very little about the environment. Deep reinforcement learning algorithms require generating many random actions to explore the environment, leading to high operational costs and serious tracking errors. To overcome the problem of low sampling rates and avoid the dangers of unstable exploration in the early stages of training, one solution is to build a simulator to perform a two-stage training process. However, this raises two additional issues: how to ensure the effectiveness of the generated training data, and how to improve the robustness of the model when the training dataset is too concentrated and the proportions remain constant.

[0156] To address the aforementioned problems, this invention proposes an improved two-stage deep reinforcement learning training method.

[0157] First, to ensure the validity of the simulation data, a Gaussian mixture model is adopted. User charging behavior exhibits strong regularity and variability. To generate data that reflects the true characteristics of electric vehicles, a Gaussian Mixture Model (GMM) is used to fit the distribution of charging behavior characteristics for each electric vehicle user. The Gaussian Mixture Model decomposes the charging behavior characteristics of electric vehicle users into several Gaussian-based probability density functions. The Gaussian Mixture Model is described as follows:

[0158]

[0159] Where x is the feature, N(x|μ) v ,Σ v ) is the v-th component of the mixture model, where V is the number of components. π v This represents the weight of the v-th component. Since Gaussian mixture models are an unsupervised learning method, the optimal number of clusters cannot be automatically set. The silhouette coefficient of each feature is calculated to select the optimal number of clusters.

[0160] The profile coefficients are as follows:

[0161]

[0162] Where a(i) is the average distance between sample i and all other points in the same cluster. b(i) is the average distance between sample i and all other points in the next nearest cluster. The shading coefficient ranges from -1 to 1. A higher shading coefficient indicates better cluster performance.

[0163] After establishing and implementing a data simulator based on a Gaussian mixture model, the real data is divided into three groups for offline training, online training, and testing. During offline training, only real data is used for rapid training. A dataset with a 1:1 mixture of real and simulated data is used as the online training dataset for each set. After both training phases, untrained real data is used to test the proposed method. Preferably, to improve the model's robustness, isotropic truncated Gaussian noise is added to 20% or more of the simulated data for each feature, and certain data types are replaced at random proportions to avoid excessive concentration of the distribution.

[0164] Example 1

[0165] All tests were conducted using Python on a computer with a 3.40GHz CPU, a 1050Ti GTX graphics card, and 16GB of RAM. PyTorch was used to develop the neural network framework for the DRL (Deep Reinforcement Learning) algorithm, and Pandapower was used to build the modified IEEE 33 bus distribution network. The effectiveness of the proposed scheme has been validated in a simulation case study using real historical data on the modified IEEE 33 bus distribution network.

[0166] Three electric vehicle aggregators are located at bus stops 13, 16, and 31, respectively. Detailed parameters for Example 1 are shown in Tables 1 and 2. Based on the set parameters, the willingness evaluation model used in Example 1 is as follows: Figure 2 , Figure 3 As shown, Figure 2 This is a graph showing the relationship between total charging price and charging willingness in this embodiment. The horizontal axis represents the total charging price (yuan), and the vertical axis represents the charging willingness (%). Figure 3 This is a graph showing the relationship between profit and discharge willingness in this embodiment, with the horizontal axis representing profit (yuan) and the vertical axis representing charging willingness (%). The peak period of the urban load curve usually occurs between 11:30 and 12:00, and drops rapidly after 12:00, while the peak period for electric vehicle charging during the day is between 12:30 and 1:00. Because these two peak periods are staggered, the aggregator's bidding strategy is more conservative than expected. Based on the actual situation, the peak time for grid power consumption is set to 11:45, and 8 scheduling operations are performed starting at 10:00 in the morning, with an interval of 15 minutes. The bid volume of the charging station aggregator on the 13 bus is [0.27, 0.46, 0.68, 1.03, 1.6, 2.74, 4.1, 5.02] (unit: megawatts), and the number of electric vehicles with charging needs is [136, 208, 305, 324, 403, 510, 602, 657] (unit: vehicles). The bid quantities for charging station aggregators on the 16-bus network are [0.23, 0.34, 0.46, 0.68, 1.37, 1.6, 2.05, 2.51] (unit: megawatts), and the number of electric vehicles with charging needs is [104, 114, 126, 152, 163, 176, 181, 192] (unit: vehicles). The bid quantities for charging station aggregators on the 31-bus network are [0.34, 0.68, 1.37, 2.28, 4.1, 5.7, 7.3, 9.58] (unit: megawatts), and the number of electric vehicles with charging needs is [125, 259, 468, 505, 694, 751, 849, 923] (unit: vehicles).

[0167] Table 1. Simulation Parameters

[0168]

[0169] Table 2. Algorithm Parameters

[0170]

[0171] The following conclusions can be drawn:

[0172] 1) During peak shaving periods, the proposed solution reduced peak load by 3.43%-4.81%, brought line congestion rates back to safe levels, and avoided voltage violations. Furthermore, the charging station's total profit increased by 7.07% due to the provision of peak shaving services.

[0173] 2) The implementation of the improved two-stage method can reduce the cumulative voltage error by an average of 90.5% and the proportion of illegally charged electric vehicles by 86.54% during the online startup process, and improve the training speed by about 70% compared with the single-stage method.

[0174] 3) Compared with traditional methods, the available energy criterion and SOC (State of Charge) constraint proposed in the reward function ensure the interests of electric vehicle users, reducing the proportion of electric vehicles that do not meet charging requirements from 27.04% to 0.052%, with an average additional waiting time of 0.57 minutes and a maximum additional waiting time of 1.31 minutes.

[0175] This invention proposes a two-stage electric vehicle (EV) auxiliary service coordination and scheduling method based on deep reinforcement learning (DRL) in the above embodiments. It also proposes a detailed modeling method for EV user participation intention based on the Fishbein model and Weber-Fechner, measuring the impact of multi-attribute attitudes on EV user choices. This invention uses the K-means clustering method for EV energy availability criteria to evaluate the energy state of EVs for scheduling, and designs a relaxation time term in the reward function to reflect the charging state constraints of EVs, thus meeting the charging needs of EVs at departure. The model training process proposed in this invention is divided into two stages: in the offline stage, actors use mixed data to learn the dynamic characteristics of the virtual environment through an EV simulator based on a Gaussian mixture model; in the online stage, the trained policy is rapidly updated in the real-time environment, effectively making decisions on uncertainties. The results of Example 1 show that the proposed method fully taps the scheduling potential of flexible power resources of EVs, enabling the grid-side node voltage to stabilize and rise back to the specified range, reducing the load peak-valley difference, and avoiding transmission line congestion problems. The load volatility on the aggregator side is reduced, and profits are significantly improved. Electric vehicles can also reach the expected SOC value after the auxiliary services are completed. Therefore, the embodiments of the present invention take into account the interests of the power grid side, aggregators, and electric vehicle users.

[0176] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A scheduling method for two-stage coordinated distribution network ancillary services for electric vehicles based on deep reinforcement learning, characterized in that, Includes the following steps: A scheduling strategy model is established based on a distribution network model, an electric vehicle battery dynamic characteristic model, a charging price incentive model, and a discharging price incentive model. The scheduling strategy model is trained using a two-stage training method based on a Gaussian mixture model (GMM) for use in coordinating the scheduling of electric vehicles for distribution network ancillary services. During the scheduling process, a K-means clustering method based on the energy availability standard of electric vehicles is used to select electric vehicles for distribution network ancillary services. The two-stage training method based on Gaussian mixture model (GMM) employs the Gaussian mixture method to fit the distribution of charging behavior characteristics of each electric vehicle user. The Gaussian mixture model decomposes the charging behavior characteristics of electric vehicle users into multiple Gaussian-based probability density functions. After establishing a data simulator based on the Gaussian mixture model, in the offline stage, the data simulator is used for pre-training with a mixture of real and simulated data. In the online stage, real-time data is used for training.

2. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in claim 1, characterized in that, Real data is divided into three groups for offline training, online training, and testing, respectively. After the two training phases, the method is tested using untrained real data.

3. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in claim 1, characterized in that, In more than 20% of the simulated data, add isotropic truncated Gaussian noise to each feature and replace selected types of data at random proportions.

4. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, A relaxation time is defined for each electric vehicle to assess its scheduling potential, and a relaxation time penalty term is set in the reward function to meet the charging needs of electric vehicle owners.

5. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, The electric vehicle battery dynamic characteristic model describes the charging / discharging process of the electric vehicle, establishes an electric vehicle energy storage model for the arrival / departure time phases of the electric vehicle, and describes the SOC at departure time to ensure it meets the energy needs of the electric vehicle user.

6. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, The charging price incentive model obtains a price-participation curve through interpolation, and then uses a neural network to fit the curve to obtain a nonlinear functional relationship between charging price and electric vehicle user participation.

7. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, The discharge price incentive model calculates the degradation cost of electric vehicle battery vehicle-to-grid (V2G) mode, the revenue of electric vehicle vehicle-to-grid (V2G) process, and the revenue of electric vehicle charging station, and provides a discharge price incentive scheme based on the Weber-Fechner law.

8. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, The scheduling strategy uses a real-time peak shaving and voltage regulation scheme based on deep reinforcement learning. The controller sets the charging price to adjust the expected number of electric vehicles arriving and sets the discharging price to incentivize electric vehicle users to participate in vehicle-to-grid (V2G) services.

9. The scheduling method for two-stage coordinated distribution network auxiliary services for electric vehicles based on deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, The training based on the Gaussian mixture model (GMM) adopts a two-stage reinforcement learning framework. First, the electric vehicle coordinated distribution network auxiliary service scheduling problem is formulated as a Markov decision process to achieve peak shaving, voltage regulation and congestion management.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the scheduling method for two-stage electric vehicle coordinated distribution network auxiliary services based on deep reinforcement learning as described in any one of claims 1 to 9.

Images