Optimization system considering differentiated demand response of electric vehicle

Abstract

The invention discloses an optimization system considering differentiated demand response of electric vehicles, the electric vehicles participating in demand response are divided into an agile contract-signing EV and a stable contract-signing EV, the agile contract-signing EV aims to maximize self-income, minimize mileage guarantee and load curve variance weighted difference, and based on an automatic demand response result of the agile contract-signing EV, the agile contract-signing EV performs automatic demand response of the agile contract-signing EV. And the stable contract signing EV performs output power optimization and aggregation by taking minimization of aggregation demand response cost as a target, performs autonomous demand response decision making based on deep reinforcement learning and autonomously participates in power grid demand response. The deep reinforcement learning converts a demand response scheduling problem into a Markov decision process, and the objective of optimizing an objective function of agile contract signing EV and an objective function of stable contract signing EV is achieved through a reward function. Compared with the prior art, the method has the advantages that the differentiated demand response of the electric vehicle is optimized through deep reinforcement learning, so that peak load shifting is effectively carried out on the power grid, and the demand response cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, and specifically relates to an optimization system that takes into account the differentiated demand response of electric vehicles. Background Technology

[0002] The widespread integration of high-proportion renewable energy sources, primarily wind and solar power, into the power grid poses a significant challenge to the safe and stable operation of the grid, and significantly increases the grid's demand for agile resource regulation.

[0003] Electric vehicles (EVs) participating in grid demand response can mitigate the impact of renewable energy on grid stability and support agile grid regulation. EVs can be divided into two categories: stable contracted EVs and agile contracted EVs. Stable contracted EVs are easy to centrally control but lack agility; agile contracted EVs can achieve decentralized response and fully realize demand response potential, but coordination is difficult. Therefore, how to achieve collaborative participation of multiple types of EVs in demand response is an urgent problem to be solved.

[0004] To address the aforementioned challenges, this paper proposes a multi-type EV collaborative participation demand response method based on deep reinforcement learning. First, EVs participating in demand response are categorized into agile contracted EVs and stable contracted EVs, and a collaborative scheduling architecture for multi-type EV participation in demand response is proposed. Second, a multi-type EV collaborative demand response strategy based on deep reinforcement learning is proposed. Specifically, agile contracted EVs aim to maximize their own revenue, mileage guarantee, and the weighted difference of the grid load curve variance. They make autonomous demand response decisions based on deep reinforcement learning and autonomously participate in grid demand response. Based on the autonomous demand response results of agile contracted EVs, asymmetric consistency optimization of output power is performed on stable contracted EVs to minimize the aggregated demand response cost, and they are aggregated to participate in grid demand response based on the optimization results. Finally, experiments are conducted to verify the effectiveness and rationality of the proposed multi-type EV collaborative participation demand response method. Summary of the Invention

[0005] Purpose of the invention: To address the problems in the background art, the present invention provides an optimization system that takes into account the differentiated demand response of electric vehicles.

[0006] Technical solution: This invention discloses an optimization system that takes into account the differentiated demand response of electric vehicles. The system performs the following processing:

[0007] Step 1: Divide the electric vehicles participating in demand response into Agile Contracted EVs and Stable Contracted EVs;

[0008] Step 2: Agile contracted EVs construct objective function one with the goals of maximizing their own revenue, minimizing mileage guarantee, and minimizing the weighted difference of the load curve variance, and perform autonomous demand response optimization;

[0009] Step 3: Based on the autonomous demand response deficit optimized by agile contracted EV, trigger stable contracted EV to construct objective function 2 with the goal of minimizing aggregated demand response cost, and perform output power optimization and aggregation;

[0010] Step 4: Based on deep reinforcement learning, the demand response scheduling problem of objective function 1 and objective function 2 is transformed into a Markov decision process. The agent continuously tries different actions in the environment and gradually explores the strategy that can maximize the long-term cumulative reward based on the reward signal fed back by the environment. Finally, the optimal optimization objective of objective function 1 of agile signing EV and objective function 2 of stable signing EV is achieved.

[0011] Furthermore, by weighting the EV user benefits (i.e., the difference between charging and discharging revenue and battery loss) and travel security (i.e., mileage guarantee cost) and grid benefits (i.e., load curve variance), we design the first objective function for agile contracted EVs:

[0012]

[0013] Where st represents the constraints, constraint (1) indicates that the agile contracted EV cannot be in charging and discharging states at the same time, and must meet the autonomous demand response strategy constraints; constraint (2) represents the upper and lower limits of the agile contracted EV's state of charge; constraint (3) represents the upper and lower limits of the agile contracted EV's charging and discharging power; U m1ben and U m1tra These are the weights for agile contract EV revenue and mileage guarantee cost, respectively; A m1ben Z(t) represents the difference between the charging / discharging revenue and the battery degradation cost. m1tra (t) represents the mileage coverage cost; U b P represents the variance weight of the grid load curve; T is the total optimization time scale; M is the number of agile contracted EVs; B(t) is the variance of the grid load curve, which is inversely proportional to grid efficiency; P... m,c (t),P m,d (t) represents the charging power and discharging power, respectively, and SOC. m (t),SOC m,min SOC m,max These represent the state of charge (SOC) and its upper and lower limits for the EVm battery, respectively, and P. m,c (t),P m,c,min ,P m,c,max These are the charging power of Agile's contracted EVs and their upper and lower limits, respectively; P m,d (t),P m,d,min ,P m,d,max These are the discharge power of the Agile Contracted EV and its upper and lower limits, respectively.

[0014] Furthermore, based on the results of the autonomous demand response of agile contracted EVs, that is, under the premise of satisfying objective function one, objective function two for stable contracted EVs is designed. By optimizing the power output of stable contracted EV aggregators and minimizing the aggregation power output cost, the expression of objective function two is established as follows:

[0015]

[0016] Where st represents the constraint conditions, constraint (1) represents the stable contracted EV aggregate output balance constraint; constraint (2) represents the upper and lower limits of the stable contracted EV charge state constraint; f i (P i (t) represents the demand response cost of aggregator i; P i (t) represents the polymer output power; α i ,β i ,χ i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient for contracted EVs, P def (t) represents the vacancy in the autonomous response of Agile Contracted EVs, SOC n (t),SOC n,min SOC n,max These represent the stable charge status of contracted EVs and their upper and lower limits, respectively.

[0017] Furthermore, the specific process of deep reinforcement learning in step 4 is as follows:

[0018] The real-time pricing and charging scheduling scheme is formulated as a time-series decision problem and a discrete-time-step finite Markov decision process. A quintuple (S) is defined for each time period. t ,a t ,P,r i ,γ);

[0019] The aggregator intelligent agent observes the environmental state S during time period t. t Environmental state S t This includes market electricity prices, wind and solar power forecasts, and existing EV information at charging stations, corresponding to the agile contracted EV revenue U in objective function one. m1ben and load variance optimization U b The input of B(t);

[0020] The agent takes action a t This includes providing time-varying incentive price discounts and whether each EV is charged, corresponding to the aggregate output power P in objective function two. i (t);

[0021] In action a t After execution, the agent observes the new state S. t+1 and take action in the next time period.t+1 P represents the state from which state S is located. t Transition to state S t+1 The probability of r t The reward obtained by the agent after taking action in time period t is the higher the agile signing EV and the smaller the load variance B(t) in objective function one, the greater the reward; the aggregate cost f of stable signing EV in objective function two is... i (P i The lower the value of (t), the greater the reward; the agent indirectly optimizes objective function one and objective function two by maximizing the cumulative reward.

[0022] The discount factor γ determines whether the agent prioritizes immediate or future rewards, corresponding to the mileage guarantee U. m1tra Z m1tra (t) and immediate returns U m1ben The trade-off is achieved through γ in the quintuple: the larger γ is, the more the agent focuses on long-term rewards; the smaller γ is, the more it focuses on short-term gains.

[0023] Furthermore, each agile contracted EV is deployed with an autonomous demand-response decision-making agent. At the beginning of each moment, the agent perceives the environmental state S. t Then the agent will state S t Input a deep Q-network and select action a based on its output state-action Q-value. t :

[0024]

[0025] During the charging process, calculate the charging demand of EVi for the next time period. for:

[0026]

[0027] In the formula: This indicates the charging requirements of the EVi when it first arrives at the charging station; For the charging needs of EVi during period t, P represents the target and initial state of charge of the EVi battery, respectively. i,t Calculating the charging power of EVi in time period t allows us to quantify the charging amount required by the agile contracted EVs to meet their travel needs in the next time period. This demand directly affects the mileage guarantee cost in objective function one. Based on the charging amount in the next time period, we can make reasonable decisions about whether to charge and the specific charging power, thereby determining action a. t The autonomous response gap of agile contracted EVs depends on the actual output of autonomous optimization, that is, the degree to which charging needs are met.

[0028] Furthermore, the confidence level of aggregator i's state information is calculated based on the aggregated contract capacity, number of stable contracted EVs, and number of contracts signed by the stable contracted EV aggregator:

[0029]

[0030] In the formula, κ i Confidence coefficient; σ represents the average number of stable EV contracts aggregated by aggregator i. i (t) represents the number of stable contracted EVs; C i (t) represents the aggregated contract capacity;

[0031] Using the aggregator's incremental cost as state information, iterative optimization of state information is performed based on asymmetric communication weights, with a confidence level δ. i (t) is the underlying basis for asymmetric communication weight allocation, providing a quantitative standard for the dynamic adjustment of asymmetric weights, expressed as:

[0032]

[0033] In the formula, Y i r (t) and Y i r+1 (t) represents the iteration amount in the r-th and r+1-th iterations, taking into account the historical state information of aggregator i. and To account for the correction amounts in the r-th and r+1-th iterations of the historical state information of aggregator i and aggregator j; To account for the historical state information of aggregator j in the r-th iteration, α i Here, r represents the operating cost coefficient, and r represents the number of asymmetric consistency iterations.

[0034] After the state information of the aggregators converges through iteration, the output power of each aggregator is determined based on the convergence value of the state information. The convergence condition and the output power of the aggregator are expressed as follows:

[0035]

[0036] In the formula, σ is the convergence threshold of the iteration; Y i * (t) represents the asymmetric consistency convergence value of the aggregator state information; Y i r (t) and Y i r+1 (t) represents the iteration amount in the rth and r+1th iterations, taking into account the historical state information of aggregator i; P i (t) represents the polymer output power; α i ,β i These are the operating cost coefficients; p subTo stabilize the economic compensation coefficient of contracted EVs, the contracted EVs perform aggregated demand response actions under the aggregated control of the aggregator, calculate the aggregated output cost according to objective function two, and feed it back to the power grid dispatch center.

[0037] Furthermore, in step 4, maximizing the cumulative reward function guides the agent to optimize the charging strategy. The reward function is defined as the aggregator's operating profit A in each time period. t =r t (S t ,a t The goal of the real-time pricing charging scheduling scheme is to maximize the total expected discount reward.

[0038]

[0039] In the formula: E[·] is the expectation function; Q(·) is the state-action value function; and the instantaneous reward r at the current moment is... t (S t ,a t The sum of expected rewards at all future moments is used to weight future rewards using a discount factor γ, where 0 < γ < 1, and the weight of rewards further into the future is smaller.

[0040] Furthermore, the incremental cost of aggregators is improved by introducing an adaptive weighting factor to update the position of the incremental cost of aggregators adaptively. The improved formula is as follows:

[0041] Y i r (t)'=ωY i r (t)

[0042] In the formula, Y i r (t)' represents the improved aggregator incremental cost; ω is the adaptive weighting factor. When the grid load fluctuates greatly or the charging and discharging demand of electric vehicles changes frequently, the adaptive weighting factor ω can be dynamically adjusted according to the real-time monitored market electricity price, wind and solar power forecast information, the number of existing electric vehicles at the charging station, and their state of charge.

[0043] Furthermore, the reward function is improved as follows:

[0044]

[0045] In the formula: The total charging load of the EV during time period t; Let M represent the amount of wind and solar power curtailed during time period t, M be the aggregator profit weighting factor, and N be the penalty factor.

[0046] Beneficial effects:

[0047] 1. This invention reduces the pressure on the power grid for peak shaving, improves the agility of peak shaving and valley filling, and enhances the stability of the power grid load by coordinating and optimizing the autonomous demand response of agile contracted EVs and the aggregated demand response of stable contracted EVs.

[0048] 2. This invention integrates EV user benefits (the difference between charging / discharging revenue and battery loss), travel security (mileage guarantee cost), and grid efficiency (load curve variance) through a weighted approach, breaking through the limitations of traditional single-objective optimization (such as pursuing only EV revenue) and achieving synergistic optimization between "users and the grid." The grid load curve variance refers to the curve showing the change in electricity load over a period of time (e.g., a day, a week), with variance being a statistical indicator describing the degree of fluctuation in this curve. A larger variance indicates more severe load fluctuations (larger peak-to-valley difference, higher fluctuation frequency); a smaller variance indicates a more stable load. The core relationship between grid load curve variance and grid efficiency is: the more severe the load fluctuations (the larger the variance), the higher the grid's generation costs, transmission and distribution losses, equipment investment, and fault risk, resulting in lower overall efficiency; conversely, the more stable the load (the smaller the variance), the higher the grid efficiency. The weight values ​​directly determine the optimization direction.

[0049] 3. The deep reinforcement learning adopted in this invention is a fusion of reinforcement learning and deep learning. Its main feature is that the agent continuously tries different actions in the environment and gradually explores the strategy that can maximize long-term cumulative rewards based on the reward signals fed back by the environment. It can automatically extract effective features from the original data and has a powerful ability to handle complex tasks and high-dimensional state spaces.

[0050] Taking agile contracted electric vehicles (EVs) as an example, they are considered intelligent agents. In objective function one, the lower the "mileage guarantee cost" (i.e., better meeting user travel needs), the greater the reward; the smaller the "grid load curve variance (B(t) correlation)" (the better the smoothing effect), the greater the reward. The intelligent agent learns the "state-action" mapping through deep reinforcement learning algorithms (such as deep Q-networks) to maximize the cumulative reward, essentially solving for the maximum value of the objective function. The environment in which these EVs operate contains many elements, including fluctuations in market electricity prices and wind and solar power generation, as well as their own battery charge, location, and other state information. The actions they can take include deciding whether to charge or discharge at different times, and determining the charging and discharging power. Under this setting, the deep reinforcement learning algorithm begins to operate. Through continuous interaction with the environment, each interaction generates a quintuple, i.e. (state S... t Action a t Reward r t The next state S t+1 (Whether to terminate (done)). The agent determines the termination based on its current state S. t Using deep reinforcement learning algorithms, an action a can be selected.t After performing an action, the environment will reward the agent with a value r according to its inherent rules. t and feedback the new state S t+1 If certain conditions are met, the interaction process will terminate (done is true).

[0051] 4. This invention maximizes the cumulative reward function to guide the agent to optimize the charging strategy, and feeds back the cost optimization results of the aggregated processing to the agent's decision-making, thereby achieving collaborative optimization.

[0052] The core objective of aggregated processing of stable contracted EVs is to minimize the aggregated demand response cost. The essence of the cumulative reward function is to transform this local cost objective into a globally learnable reward signal for the agent, forming a closed loop of "cost optimization → reward feedback → strategy adjustment." When the aggregated processing of stable contracted EVs reduces costs (e.g., aggregated power output more accurately matches grid demand, reducing redundant scheduling costs), the aggregator's operating profit (A) increases. t The reward function is defined as "the aggregator's operating profit in each time period", and the cumulative reward will increase accordingly. Conversely, if the aggregation process fails to meet the cost optimization target of objective function two (such as over-output leading to increased compensation costs), the operating profit will decrease and the reward will decrease. The agent will then learn to adjust its strategy (such as optimizing the aggregation control parameters for stable contracted EV) to further optimize objective function two.

[0053] Aggregated output (output optimization of stably contracted EVs) alters the actual load state of the power grid, and this state serves as the input for the agent's decision-making (S). t (Part of this) directly affects the calculation logic of cumulative rewards: the aggregated power output of stable contracted EVs will change the grid load curve B(t), while the objective function of agile contracted EVs includes the "load curve variance weighted difference," which affects the revenue calculation of agile EVs and is reflected in the rewards through operating profit; the aggregated power output cost is directly included in the aggregator's operating cost. If the cost is too high, it will offset the revenue of agile contracted EVs, leading to a decrease in operating profit and ultimately reducing the reward r. t This reduction forces intelligent agents to balance the "autonomous benefits of agile EVs" and the "aggregated costs of stable EVs" when making decisions.

[0054] Aggregated output is not statically optimized, but dynamically adjusted over time t (e.g., updating the output plan based on real-time grid load and the autonomous response results of agile EVs). Furthermore, the "time-series decision-making" characteristic of the cumulative reward function (based on the quintuple S) is also relevant. t →S t+1 State transitions are well suited to this dynamic nature.

[0055] 5. This invention, based on a deep Q-network learning strategy algorithm, can effectively solve the problem of large action-space complexity. The proposed real-time optimized scheduling strategy does not rely on prediction information and is not limited by the timing and number of EV access points. Therefore, it has universality in the real-time scheduling problem of charging stations, can be applied to real-time charging scheduling of EVs, and can be extended to scenarios involving the joint operation of multiple charging stations. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the structure of the present invention;

[0057] Figure 2 This invention provides a multi-type EV collaborative demand response control architecture.

[0058] Figure 3 This is an experimental diagram proposed in this invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings.

[0060] like Figure 2 As shown: Based on the demand for information dissemination from renewable energy sources such as photovoltaic and wind power, as well as load data, the power grid dispatch center coordinates the participation of agile contracted EVs and stable contracted EVs in demand response, respectively, under the incentives of electricity price and compensation from the power grid dispatch center. Agile contracted EVs do not need to sign agreements with EV aggregators; instead, they make their own demand response decisions based on their own travel needs and relevant grid-side information. Stable contracted EVs, on the other hand, sign demand response agreements with EV aggregators, who then perform aggregated control and provide stable contracted EVs with certain economic compensation according to the agreement. The coordination between agile and stable contracted EVs is mainly reflected in two aspects: ① After agile contracted EVs participate in demand response independently, stable contracted EVs aggregate demand response based on their own response gaps, and the two work together to meet grid demand; ② After participating in demand response, stable contracted EVs feed back their aggregated response results to agile contracted EVs, which then dynamically adjust their independent demand response strategies based on the feedback information.

[0061] 1. Agile Contracted EV Autonomous Demand Response Model

[0062] 1.1 Energy Storage Model

[0063] Considering M agile contract EVs, the energy storage model for the m-th agile contract EV is:

[0064] E m (t+1)=E m (t)+(P m,c (t)-P m,d (t))Δt

[0065] In the formula, Em (t) represents the stored energy; P m,c (t),P m,d (t) represents the charging and discharging power, respectively; Δt represents the time interval.

[0066] The state of charge of EV m is:

[0067] In the formula, SOC m (t) represents the state of charge of the EVm battery; E m,max This refers to the rated capacity of the energy storage battery.

[0068] EVm's charging capacity C participating in demand response m,c (t) and discharge capacity C m,d (t) are respectively represented as:

[0069]

[0070] In the formula, SOC m,max and SOC m,min These are the upper and lower limits of the state of charge of EVm, respectively.

[0071] 1.2 Profit Model

[0072] Benefits of EVm participating in demand response A m,ben (t) represents the difference between the charging / discharging revenue and the battery degradation cost, expressed as:

[0073] A m,ben (t)=[(y d (t)P m,d (t)-y c (t)P m,c (t))-(ξ m P m,d (t)+P m,c (t))]Δt

[0074] In the formula, y c (t) and y d (t) represent the charge and discharge valences at time t, respectively; ξ m denoted as EVm, representing the charge / discharge loss coefficient.

[0075] 1.3 Mileage Coverage Cost Model

[0076] To ensure the mobility of EVs contracted through Agile, a mileage guarantee cost model is constructed based on their state of charge requirements upon leaving the grid. Assume the EVm's entry and exit times are as follows: and Then its mileage guarantee cost Z m,tra (t) is represented as:

[0077]

[0078] in, This is the mileage guarantee cost coefficient for EVm; a larger value indicates a greater weight in guaranteeing the EVm's travel capability. EVM off-network time Target value for state of charge.

[0079] 2. Stable Contracted EV Independent Demand Response Model

[0080] 2.1 Aggregated Contract Capacity Model

[0081] Consider I EV aggregators, and define Ω as the set of stable EV contracts aggregator i has aggregated at time t. i (t), then the aggregated contract capacity C i (t) is represented as:

[0082]

[0083] In the formula, E n,max To ensure stable contracted EVn energy storage battery rated capacity; SOC n,max and SOC n,min These are the upper and lower limits of the state of charge of EVn, respectively.

[0084] 2.2 Aggregated Output Cost Model

[0085] The costs incurred by EV aggregators in regulating and stabilizing contracted EVs to participate in demand response mainly consist of two parts: aggregator operating costs and economic compensation costs for stable contracted EVs, expressed as follows:

[0086] f i (P i (t))=α i [P i (t)] 2 +(β i +p sub )P i (t)+χ i

[0087] In the formula, f i (P i (t) represents the demand response cost of aggregator i; P i (t) represents the polymer output power; α i ,β i ,χ i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient for contracted EVs.

[0088] 3. Agile Contract Signing EV Autonomous Demand Response Strategy

[0089] Design an objective function for agile EV contracting, aiming to maximize the agile EV contracting revenue, mileage guarantee cost, and the weighted difference of the variance of the grid load curve. The objective function is as follows:

[0090]

[0091] st represents the constraints, where constraint (1) indicates that the agile contracted EV cannot be in both charging and discharging states at the same time, and must meet the autonomous demand response strategy constraints; constraint (2) represents the upper and lower limits of the agile contracted EV's state of charge; constraint (3) represents the upper and lower limits of the agile contracted EV's charging and discharging power; U m1ben and U m1tra These are the weights for agile contract EV revenue and mileage guarantee cost, respectively; A m1ben Z(t) represents the difference between the charging / discharging revenue and the battery degradation cost. m1tra (t) represents the mileage coverage cost; U b P represents the variance weight of the grid load curve; T is the total optimization time scale; M is the number of agile contracted EVs; B(t) is the variance of the grid load curve, which is inversely proportional to grid efficiency; P... m,c (t),P m,d (t) represents the charging power and discharging power, respectively, and SOC. m (t),SOC m,min SOC m,max These represent the state of charge (SOC) and its upper and lower limits for the EVm battery, respectively, and P. m,c (t),P m,c,min ,P m,c,max These are the charging power of Agile's contracted EVs and their upper and lower limits, respectively; P m,d (t),P m,d,min ,P m,d,max These are the discharge power of the Agile Contracted EV and its upper and lower limits, respectively.

[0092] 4. Stable EV self-reliant demand response strategy

[0093] The objective function for stable contracted EVs is designed by optimizing the power output of stable contracted EV aggregators and minimizing the aggregation cost. The expression for the objective function is as follows:

[0094]

[0095] st represents the constraint condition, constraint (1) represents the stable contracted EV aggregate output balance constraint; constraint (2) represents the upper and lower limits of the stable contracted EV charge state constraint; f i (P i (t) represents the demand response cost of aggregator i; P i (t) represents the polymer output power; α i ,βi ,χ i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient for contracted EVs, P def (t) represents the vacancy in the autonomous response of Agile Contracted EVs, SOC n (t),SOC n,min SOC n,max These represent the stable charge status of contracted EVs and their upper and lower limits, respectively.

[0096] 5. Deep Learning-Based Demand Response Scheduling Algorithm for Multi-Type EV Collaboration

[0097] This invention formulates real-time pricing and charging scheduling schemes as a time-series decision problem and a discrete-time finite Markov decision process. A quintuple (S) is defined for each time period. t ,a t ,P,r i ,γ).

[0098] First, the aggregator intelligent agent observes the environmental state S during time period t. t This includes market electricity prices, wind and solar power forecasts, and existing EV information at charging stations, corresponding to the agile contracted EV revenue U in the objective function. m1ben and load variance optimization U b The input is B(t). Then, the agent takes action a. t This includes providing time-varying incentive price discounts and whether each EV is charged, corresponding to the aggregated output power P. i (t). Finally, in action a t After execution, the agent observes the new state S. t+1 and take action in the next time period. t+1 P is the state from S. t Transition to state S t+1 The probability of r t The reward obtained by the agent after taking action in time period t is a quantitative feedback signal of the objective function. The higher the return of agile signing EV and the smaller the load variance B(t) in the objective function, the greater the reward; the aggregate cost f of stable signing EV in the objective function is... i (P i The lower the value of γ, the greater the reward. The agent indirectly optimizes the objective function by maximizing the cumulative reward. The discount factor γ determines whether the agent prioritizes the current immediate reward or the future reward, corresponding to the mileage guarantee U in the objective function. m1tra Z m1tra (t)(long-term travel demand) and immediate revenue U m1benThe trade-off between (short-term benefits) is achieved through γ in the quintuple: the larger γ is, the more the agent focuses on long-term rewards (such as ensuring that the SOC meets the standard when leaving the grid to meet mileage requirements); the smaller γ is, the more it focuses on short-term gains (such as profiting from discharging under high electricity prices in the current period).

[0099] Each Agile Contracted EV is deployed with an autonomous demand-response decision-making agent. At the beginning of each moment, the agent perceives the environmental state S. t Then, the agent will state S t Input a deep Q-network and select action a based on its output state-action Q-value. t :

[0100]

[0101] During the charging process, calculate the charging demand of EVi for the next time period. for:

[0102]

[0103] In the formula: This indicates the charging requirements of the EVi when it first arrives at the charging station; The charging demand of EVi during period t. P represents the target and initial state of charge of the EVi battery, respectively. i,t Calculating the charging power of EVi in time period t allows us to quantify the charging amount required by the agile contracted EVs to meet their travel needs in the next time period. This demand directly affects the mileage guarantee cost in objective function one. Based on the charging amount in the next time period, we can make reasonable decisions about whether to charge and the specific charging power, thereby determining action a. t The autonomous response gap of agile contracted EVs depends on the actual output of autonomous optimization, that is, the degree to which charging needs are met.

[0104] The confidence level δ of aggregator i's status information is calculated based on the aggregator's aggregated contract capacity, number of stable contracted EVs, and number of contracts. i (t):

[0105]

[0106] In the formula, κ i Confidence coefficient; σ represents the average number of stable EV contracts aggregated by aggregator i. i (t) represents the number of stable contracted EVs; C i (t) represents the aggregated contract capacity.

[0107] Confidence level δ i The formula for calculating (t) involves the average number of contracts signed by the aggregator. Stable number of signed EV contracts (σ) i (t)), aggregated contract capacity (C i (t)) is essentially used to evaluate the reliability of the state information (such as incremental cost) provided by the aggregator through these core parameters. This reliability assessment is an "implicit premise" for subsequent iterative optimization - only highly reliable state information is suitable as the basis for iteration. Otherwise, iteration based on unreliable information will lead to deviations in the calculation of aggregate output power, affecting the optimization effect of the objective function (minimizing aggregate cost).

[0108] "Iterative optimization of state information based on asymmetric communication weights"—the core logic of asymmetric weights is to "give higher weight to more reliable information during iteration." And the confidence level δ... i (t) is the potential basis for asymmetric weight allocation. The confidence level provides a quantitative standard for the dynamic adjustment of asymmetric weights and is the underlying support for iterative optimization to "screen high-quality information".

[0109] The role of confidence level is more like "pre-verification"—ensuring that the state information entering the iteration stage is "qualified" rather than directly participating in the formula calculation.

[0110] This invention uses the aggregator's incremental cost as state information and performs iterative optimization of the state information based on asymmetric communication weights, as expressed in:

[0111]

[0112] In the formula, Y i r (t) and Y i r+1 (t) represents the iteration amount in the r-th and r+1-th iterations, taking into account the historical state information of aggregator i. and To account for the correction amounts in the r-th and r+1-th iterations of the historical state information of aggregator i and aggregator j; To account for the historical state information of aggregator j in the r-th iteration, α i Let r be the operating cost coefficient, and r be the number of asymmetric consistency iterations.

[0113] Simultaneously, the incremental cost of aggregators is improved by introducing an adaptive weighting factor to update the position of the incremental cost of aggregators adaptively. The improved formula is as follows: Y i r (t)'=ωY i r (t), Y i r(t)' represents the improved aggregator incremental cost; ω is the adaptive weighting factor. When the grid load fluctuates greatly or the charging and discharging demand of electric vehicles changes frequently, the adaptive weighting factor ω can be dynamically adjusted based on environmental conditions such as real-time monitored market electricity prices, wind and solar power forecasts, the number of existing electric vehicles at charging stations, and their state of charge.

[0114] After the state information of the aggregators converges through iteration, the output power of each aggregator can be obtained based on the convergence value of the state information. The convergence condition and the output power of the aggregators are expressed as follows:

[0115]

[0116] In the formula, σ is the convergence threshold of the iteration; Y i * (t) represents the asymmetric consistency convergence value of the aggregator state information; Y i r (t) and Y i r+1 (t) represents the iteration amount in the rth and r+1th iterations, taking into account the historical state information of aggregator i; P i (t) represents the polymer output power; α i ,β i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient for contracted EVs, contracted EVs, under the aggregation control of aggregators, perform aggregation demand response actions, calculate aggregation output costs based on F2, and feed them back to the power grid dispatch center.

[0117] Maximizing the cumulative reward function guides the agent to optimize its charging strategy. Reward function r t (S t ,a t The aggregator's operating profit A for each time period is defined as follows: t =r t (S t ,a t ).

[0118] To maintain high scheduling agility for EVs at charging stations and thus shift more charging load to periods of frequent wind and solar curtailment, the agent needs to learn to reduce charging load when there is no wind or solar curtailment. Therefore, the reward function needs to be modified:

[0119]

[0120] In the formula: The total charging load of the EV during time period t; Let M represent the amount of wind and solar power curtailed during time period t, M be the aggregator profit weighting factor, and N be the penalty factor.

[0121] The goal of a real-time pricing charging scheduling scheme is to maximize the total expected discount reward.

[0122]

[0123] In the formula: E[·] is the expectation function; Q(·) is the state-action value function.

[0124] The formula is based on the instantaneous reward r at the current moment. t (S t ,a t The sum of expected rewards at all future moments is used to weight future rewards using a discount factor γ (0 < γ < 1) (the further into the future the reward, the smaller the weight).

[0125] Starting from the current time t, the goal is to maximize the sum of accumulated rewards over all future time periods after applying discount weights. The core of this approach is "long-term cumulative optimization" over time, rather than optimization at a single final time.

[0126] 5.1 Construction of Deep Q-Network

[0127] Since the discount information is discrete, such as 10% discount, 20% discount, etc., a deep Q network (DQN) is used to solve for the optimal strategy, and experience replay and target neural network are used to improve the stability of the training process.

[0128] (1) Experience Replay: Through each interaction with the environment, the agent replays its experience as tuples (S t a t r t S t+1 The experience samples are stored in the form of an experience buffer. When the number of experience samples exceeds the capacity of the experience buffer, the experience samples are replaced. During training, a batch of experience samples are randomly sampled from the experience buffer to update the parameters of the deep neural network.

[0129] (2) Target Network: To improve the training stability and convergence of the DQN algorithm, this paper uses two networks with identical structures: an evaluation network and a target network. During training, the evaluation network needs to be updated at each step to make real-time decisions and select the optimal action for the current state; the target network, on the other hand, needs to copy the parameters of the evaluation network and update them every μ steps. The target network is used to calculate the target Q-value, providing a stable target and mitigating target changes during training. During training, the parameters of the evaluation network are updated by minimizing the loss function L(θ).

[0130]

[0131] In the formula: y is the target reward; θ is the evaluation network parameter; θ' is the target network parameter.

[0132] DQN allows for customized charging price discounts and charging plans for each EV user. Simultaneously, to maintain high dispatch agility for the EV fleet at charging stations, charging priorities are ranked based on available dispatch time. This optimizes the economics of wind and solar power integration and aggregators, while also saving charging costs for EV users.

[0133] 6. Numerical Example Verification

[0134] This paper considers a scenario involving 500 EVs and 3 stable contracted EV aggregators participating in grid demand response, where agile contracted EVs and stable contracted EVs account for 70% and 30% of the market, respectively. The simulation duration is 12 hours with a step size of 5 minutes.

[0135] Load curve comparison Figure 3 As shown, during the approximately 15-20 hour period, the peak power of the original load (black dashed line) is higher, while the peak load after the stimulus response (purple solid line) is significantly lower. This indicates that the demand response effectively reduced the load during peak electricity consumption periods, preventing the power system from experiencing overload due to short-term high loads. Around the 5-hour mark, the original load is at its lowest point, and the load after the stimulus response relatively increases at this point, partially filling the electricity consumption trough. Simultaneously, overall, the difference between the load curve after the stimulus response during trough periods (such as from the initial period to around 5 hours) and the original load "fills" the load trough, making the load curve smoother and improving the efficiency of power resource utilization.

[0136] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent transformations or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An optimization system considering the differentiated demand response of electric vehicles, characterized in that, The system performs the following processing: Step 1: Divide the electric vehicles participating in demand response into Agile Contracted EVs and Stable Contracted EVs; Step 2: Agile contracted EVs construct objective function one with the goals of maximizing their own revenue, minimizing mileage guarantee, and minimizing the weighted difference of the load curve variance, and perform autonomous demand response optimization; Step 3: Based on the autonomous demand response deficit optimized by agile contracted EV, trigger stable contracted EV to construct objective function 2 with the goal of minimizing aggregated demand response cost, and perform output power optimization and aggregation; Step 4: Based on deep reinforcement learning, the demand response scheduling problem of objective function 1 and objective function 2 is transformed into a Markov decision process. The agent continuously tries different actions in the environment and gradually explores the strategy that can maximize the long-term cumulative reward based on the reward signal fed back by the environment. Finally, the optimal optimization objective of objective function 1 of agile signing EV and objective function 2 of stable signing EV is achieved.

2. The optimization system for considering the differentiated demand response of electric vehicles according to claim 1, characterized in that, By weighting and integrating the benefits of EV users (the difference between charging and discharging revenue and battery loss), travel security (range guarantee cost), and grid benefits (load curve variance), the objective function for agile contracted EVs is designed as follows: Where st represents the constraints, constraint (1) indicates that the agile contracted EV cannot be in charging and discharging states at the same time, and must meet the autonomous demand response strategy constraints; constraint (2) represents the upper and lower limits of the agile contracted EV's state of charge; constraint (3) represents the upper and lower limits of the agile contracted EV's charging and discharging power; U m1ben and U m1tra These are the weights for agile contract EV revenue and mileage guarantee cost, respectively; A m1ben Z(t) represents the difference between the charging / discharging revenue and the battery degradation cost. m1tra (t) represents the mileage coverage cost; U b P represents the variance weight of the grid load curve; T is the total optimization time scale; M is the number of agile contracted EVs; B(t) is the variance of the grid load curve, which is inversely proportional to grid efficiency; P... m,c (t),P m,d (t) represents the charging power and discharging power, respectively, and SOC. m (t),SOC m,min SOC m,max These represent the state of charge (SOC) and its upper and lower limits for the EVm battery, respectively, and P. m,c (t),P m,c,min ,P m,c,max These are the charging power of Agile's contracted EVs and their upper and lower limits, respectively; P m,d (t),P m,d,min ,P m,d,max These are the discharge power of the Agile Contracted EV and its upper and lower limits, respectively.

3. The optimization system for considering the differentiated demand response of electric vehicles according to claim 2, characterized in that, Based on the results of the autonomous demand response of agile contracted EVs, that is, under the premise of satisfying objective function one, objective function two for stable contracted EVs is designed. By optimizing the power output of stable contracted EV aggregators and minimizing the aggregation power output cost, the expression of objective function two is established as follows: Where st represents the constraint conditions, constraint (1) represents the stable contracted EV aggregate output balance constraint; constraint (2) represents the upper and lower limits of the stable contracted EV charge state constraint; f i (P i (t) represents the demand response cost of aggregator i; P i (t) represents the polymer output power; α i ,β i ,χ i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient for contracted EVs, P def (t) represents the vacancy in the autonomous response of Agile Contracted EVs, SOC n (t),SOC n,min SOC n,max These represent the stable charge status of contracted EVs and their upper and lower limits, respectively.

4. An optimization system considering the differentiated demand response of electric vehicles according to claim 3, characterized in that, The specific process of deep reinforcement learning in step 4 is as follows: The real-time pricing and charging scheduling scheme is formulated as a time-series decision problem and a discrete-time-step finite Markov decision process. A quintuple (S) is defined for each time period. t ,a t ,P,r i ,γ); The aggregator intelligent agent observes the environmental state S during time period t. t Environmental state S t This includes market electricity prices, wind and solar power forecasts, and existing EV information at charging stations, corresponding to the agile contracted EV revenue U in objective function one. m1ben and load variance optimization U b The input of B(t); The agent takes action a t This includes providing time-varying incentive price discounts and whether each EV is charged, corresponding to the aggregate output power P in objective function two. i (t); In action a t After execution, the agent observes the new state S. t+1 and take action in the next time period. t+1 P represents the state from which state S is located. t Transition to state S t+1 The probability of r t The reward obtained by the agent after taking action in time period t is the higher the agile signing EV and the smaller the load variance B(t) in objective function one, the greater the reward; the aggregate cost f of stable signing EV in objective function two is... i (P i The lower the value of (t), the greater the reward; the agent indirectly optimizes objective function one and objective function two by maximizing the cumulative reward. The discount factor γ determines whether the agent prioritizes immediate or future rewards, corresponding to the mileage guarantee U. m1tra Z m1tra (t) and immediate returns U m1ben The trade-off is achieved through γ in the quintuple: the larger γ is, the more the agent focuses on long-term rewards; the smaller γ is, the more it focuses on short-term gains.

5. An optimization system for considering the differentiated demand response of electric vehicles according to claim 4, characterized in that, Each Agile Contracted EV is deployed with an autonomous demand-response decision-making agent. At the beginning of each moment, the agent perceives the environmental state S. t Then the agent will state S t Input a deep Q-network and select action a based on its output state-action Q-value. t : During the charging process, calculate the charging demand of EVi for the next time period. for: In the formula: This indicates the charging requirements of the EVi when it first arrives at the charging station; For the charging needs of EVi during period t, P represents the target and initial state of charge of the EVi battery, respectively. i,t Calculating the charging power of EVi in time period t allows us to quantify the charging amount required by the agile contracted EVs to meet their travel needs in the next time period. This demand directly affects the mileage guarantee cost in objective function one. Based on the charging amount in the next time period, we can make reasonable decisions about whether to charge and the specific charging power, thereby determining action a. t The autonomous response gap of agile contracted EVs depends on the actual output of autonomous optimization, that is, the degree to which charging needs are met.

6. An optimization system for considering the differentiated demand response of electric vehicles according to claim 4, characterized in that, The confidence level of aggregator i's status information is calculated based on the aggregator's aggregated contract capacity, number of stable contracted EVs, and number of contracts: In the formula, κ i Confidence coefficient; The average number of stable EV contracts aggregated by aggregator i; σ i (t) represents the number of stable contracted EVs; C i (t) represents the aggregated contract capacity; Using the aggregator's incremental cost as state information, iterative optimization of state information is performed based on asymmetric communication weights, with a confidence level δ. i (t) is the underlying basis for asymmetric communication weight allocation, providing a quantitative standard for the dynamic adjustment of asymmetric weights, expressed as: In the formula, Y i r (t) and Y i r+1 (t) represents the iteration amount in the r-th and r+1-th iterations, taking into account the historical state information of aggregator i. and To account for the correction amounts in the r-th and r+1-th iterations of the historical state information of aggregator i and aggregator j; To account for the historical state information of aggregator j in the r-th iteration, α i Here, r represents the operating cost coefficient, and r represents the number of asymmetric consistency iterations. After the state information of the aggregators converges through iteration, the output power of each aggregator is determined based on the convergence value of the state information. The convergence condition and the output power of the aggregator are expressed as follows: In the formula, σ is the convergence threshold of the iteration; Y i * (t) represents the asymmetric consistency convergence value of the aggregator state information; Y i r (t) and Y i r+1 (t) represents the iteration amount in the rth and r+1th iterations, taking into account the historical state information of aggregator i; P i (t) represents the polymer output power; α i ,β i These are the operating cost coefficients; p sub To stabilize the economic compensation coefficient of contracted EVs, the contracted EVs perform aggregated demand response actions under the aggregated control of the aggregator, calculate the aggregated output cost according to objective function two, and feed it back to the power grid dispatch center.

7. An optimization system for considering the differentiated demand response of electric vehicles according to claim 4, characterized in that, In step 4, maximizing the cumulative reward function guides the agent to optimize the charging strategy. The reward function is defined as the aggregator's operating profit A in each time period. t =r t (S t ,a t The goal of the real-time pricing charging scheduling scheme is to maximize the total expected discount reward. In the formula: E[·] is the expectation function; Q(·) is the state-action value function; and the instantaneous reward r at the current moment is... t (S t ,a t The sum of expected rewards at all future moments is used to weight future rewards using a discount factor γ, where 0 < γ < 1, and the weight of rewards further into the future is smaller.

8. An optimization system for considering the differentiated demand response of electric vehicles according to claim 6, characterized in that, The incremental cost of aggregators is improved by introducing an adaptive weighting factor to update the position of the incremental cost of aggregators adaptively. The improved formula is as follows: Y i r (t)'=ωY i r (t) In the formula, Y i r (t)' represents the improved aggregator incremental cost; ω is the adaptive weighting factor. When the grid load fluctuates greatly or the charging and discharging demand of electric vehicles changes frequently, the adaptive weighting factor ω can be dynamically adjusted according to the real-time monitored market electricity price, wind and solar power forecast information, the number of existing electric vehicles at the charging station, and their state of charge.

9. An optimization system for considering the differentiated demand response of electric vehicles according to claim 7, characterized in that, The reward function is improved as follows: r t (S t ,are t )=ΜA t -Ν|E t a -THAT t EV | In the formula: The total charging load of the EV during time period t; t represents the amount of wind and solar power curtailed during time period t, and M is the aggregator's profit weighting factor. N is the penalty factor.

Images