V2G dynamic service rate generation method based on deep reinforcement learning

Abstract

The invention relates to a V2G dynamic service rate generation method based on deep reinforcement learning, and the method comprises the steps: dynamically optimizing a charging and discharging strategy through sensing the state of a power grid, the battery state of an electric vehicle and the demands of a user in real time, achieving the multi-target collaborative optimization, constructing a multi-source interaction environment comprising the power grid, a charging station, a charging pile, the electric vehicle and the user, and generating a V2G dynamic service rate. The method comprises the following steps of: defining a state vector, performing action space design and multi-target reward function design, performing implementation by adopting a depth deterministic policy gradient (DDPG) algorithm framework, performing extension design by adopting a multi-agent depth deterministic policy gradient (MADDPG) framework, and pushing a charging and discharging plan to a user after an optimal result is obtained. The problems of static service rate, serious battery loss, multi-target conflict and algorithm limitation existing in the V2G technology are solved, the electric vehicle charging and discharging strategy and service rate are optimized in real time, the power grid state is dynamically responded, the algorithm adaptability is improved, and multi-target collaborative optimization of power grid peak regulation, user income maximization and battery life prolonging is achieved.

Description

TECHNICAL FIELD

[0001] The application relates to a smart grid and electric vehicle (EV) cooperative control technology, in particular to a vehicle-to-grid (V2G) dynamic service rate generation method based on deep reinforcement learning (DRL). BACKGROUND

[0002] With the increasing penetration of electric vehicles, V2G technology can participate in two-way energy interaction of the power grid through the vehicle-mounted battery, but the existing related technology has the following problems:

[0003] (1) Static service rate: the traditional electricity price strategy cannot dynamically respond to power grid load fluctuations and user demand differences, resulting in low resource allocation efficiency.

[0004] (2) Serious battery wear: lack of optimization model for frequent charging and discharging, which accelerates battery capacity attenuation.

[0005] (3) Multi-objective conflict: existing methods focus on a single objective (such as user economy or grid stability), and lack systematic optimization of battery life.

[0006] (4) Algorithm limitations: rule-based control or traditional optimization algorithms (such as linear programming) are difficult to handle high-dimensional dynamic environments and lack real-time performance. SUMMARY

[0007] In view of the problems of static service rate, serious battery wear, multi-objective conflict and algorithm limitations in V2G technology, a V2G dynamic service rate generation method based on deep reinforcement learning is proposed. The method is used to optimize the charging and discharging strategy of electric vehicles and the service rate in real time, and realizes the multi-objective cooperative optimization of power grid peak shaving, user benefit maximization and battery life extension.

[0008] The technical scheme of the application is as follows:

[0009] A V2G dynamic service rate generation method based on deep reinforcement learning, by real-time sensing of power grid state, electric vehicle battery state and user demand, dynamically optimizing charging and discharging strategy, realizing multi-objective cooperative optimization, building a multi-source interaction environment including power grid, charging station, charging pile, electric vehicle and user, defining as state vector, designing action space and multi-objective reward function, using deep deterministic policy gradient DDPG algorithm framework for implementation, using multi-agent deep deterministic policy gradient MADDPG framework for extension design, obtaining the optimal result and pushing the charging and discharging plan to the user.

[0010] Further, the specific implementation process is as follows:

[0011] Step 1: System modeling and state space definition

[0012] A multi-source interaction environment is constructed, which comprises a power grid, a charging station, a charging pile, an electric vehicle, and a user, and is defined as a state vector S t :

[0013] S t = [p t , L t , G t , V dev , SOC t , SOH t , N max , T end , SOC target ]

[0014] wherein p t represents a real-time electricity price, which is obtained through a power market interface; L t represents a power grid load rate, i.e., a ratio of a current load to a maximum capacity; G t represents a new energy output; V dev represents a voltage overrun times, which is a number of times that a voltage exceeds a rated value by ±5% in a day; SOC t represents a remaining electric quantity of a vehicle, which is collected in real time by a battery management system; SOH t represents a battery health degree, which is calculated based on a cycle number and a capacity attenuation model; N max represents a maximum discharging number allowed by a user this time; T end represents an expected latest vehicle taking time of a user; SOC target represents an expected battery electric quantity when the user takes the vehicle;

[0015] Step 2: Action space design

[0016] The intelligent agent outputs a continuous action vector a t = [P, λ t ], wherein: the charging and discharging power P ∈ [-P discharge , P charge ], a positive value is charging, a negative value is discharging, and the power range is limited by the hardware of the charging pile; the dynamic service rate coefficient λ t , the service fee p service is calculated according to the following formula:

[0017] p service = λ t · (p t + η·max (0, load rate-90%)

[0018] wherein η = 0.2 is a load penalty coefficient, and the penalty is triggered when L t > 90%;

[0019] Step 3: Multi-objective reward function design

[0020] The reward function integrates user benefits, grid stability and battery life:

[0021] R t = a·R economy + b·R gird - g·R battery

[0022] User economic benefit R economy :

[0023] R economy = P discharge · p discharge - P charge · p charge - p service

[0024] Where p discharge is the discharge price, p charge is the charging price, and p service is the service fee;

[0025] Grid stability reward R gird :

[0026]

[0027] Where L peak and L valley are the daily load peak and valley values, and L base is the historical reference value peak-valley difference;

[0028] Battery loss penalty R battery :

[0029] R battery = 0.1·(△SOC) 2 + 0.05·N cycle

[0030] Where △SOC is the SOC change amplitude of a single charge-discharge cycle, and N cycle is the number of charge-discharge cycles;

[0031] Step 4: Deep reinforcement learning algorithm implementation

[0032] This step is implemented using the Deep Deterministic Policy Gradient (DDPG) algorithm framework, and the specific scheme is as follows:

[0033] Network architecture design:

[0034] Policy network, i.e. Actor network: This network is responsible for generating the optimal action based on the current environment state; its input is an 8-dimensional state vector S t; The network structure contains three fully connected hidden layers with 256, 128, and 64 neurons, respectively, and the ReLU function is used as the activation function for each hidden layer; the output layer generates a 2-dimensional action vector a t , which corresponds to the power allocation ratio P and the parameter λ t , respectively; to constrain the output range of the action, the output layer uses the hyperbolic tangent activation function;

[0035] The value network, i.e., the Critic network: This network is responsible for evaluating the value of a given state-action pair; its input layer simultaneously accepts an 8-dimensional state vector S t and a 2-dimensional action vector a t , which are concatenated to form a 10-dimensional input vector; the value network also contains three fully connected hidden layers with 256, 128, and 64 neurons, respectively, and the ReLU function is used as the activation function; the output layer generates a scalar value Q(S t , a t ), representing the expected cumulative reward of performing action a t in state S t ;

[0036] Training process:

[0037] Experience replay mechanism: The transition sample data generated during the interaction between the agent and the environment, i.e., the four-tuple (S t , a t , r t , S t+1 ) consisting of the current state S t , the performed action a t , the immediate reward r t , and the next state S t+1 after the transition, is stored in an experience pool; when training the network parameters, the priority is sampled from the experience pool based on the temporal difference error TD Error;

[0038] Target network update: To stabilize the training process, the algorithm maintains a target policy network and a target value network; the parameters θ - of the target network are updated slowly following the parameters θ of the online policy network and the online value network through software updates; the specific update rule is:

[0039] θ - ← 0.01θ + 0.99θ -

[0040] Exploration strategy: To encourage the agent to fully explore the environment in the early stages of training, Gaussian noise with a mean of zero is added to the action generated by the policy network; the standard deviation σ of this noise is initially set to 0.3; as the training process progresses, the value of σ gradually decreases linearly and eventually stabilizes at 0.1;

[0041] Step 5: Multi-user collaborative expansion

[0042] To adapt to multi-user scenarios (users 1 to N), this step adopts an extended design using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework, specifically including:

[0043] Centralized Critic Design: Each user (agent) is equipped with an independent policy network (Actor) to select actions based on their local state. However, a centralized value network (Critic) is introduced during training. This Critic network has access to the joint state information (global state) of all users and the joint actions taken by all users during training. Based on this global information, the centralized Critic outputs a global Q-value Q(S,a) to evaluate the joint actions a = (a1,a2,…,a) performed by all users in global state S. N The expected future cumulative total reward; this global Q value provides a more comprehensive value guidance signal for training the policy network of each user;

[0044] Attention Mechanism Integration: To improve the efficiency of centralized Critic networks in processing multi-user information and enable them to dynamically monitor the differences in the importance of different users' contributions to the global value, a multi-head attention mechanism is introduced into the hidden layers of the Critic network. The computation process of the attention mechanism is defined as follows:

[0045]

[0046] Where Q is the query vector, representing the representation vector of the target user or target feature for which the Critic network needs to calculate value; K is the key vector, representing the representation vector of all users or other feature elements, used to calculate the similarity between users and query vector Q; V is the value vector, representing the actual feature value vector corresponding to key vector K, and the basis for the final weighted summation; d k is the dimension of the key vector, representing the dimension of each vector in the key vector K;

[0047] use Scaling is performed to prevent the gradient of the Softmax function from vanishing due to an excessively large dot product result; QK T Calculate the dot product similarity between the query vector Q and all key vectors K; The scaled dot product similarity is normalized using Softmax to obtain a set of attention weights; the final output of Attention(Q,K,V) is the weighted sum of the value vectors V, and the weights are the attention weights calculated above.

[0048] Step 6: Dynamic Service Fee Generation and Strategy Output

[0049] Service rate calculation: According to λ t Dynamic adjustment, formula see step 2;

[0050] User interaction interface: Push the charging and discharging plan to the user, including the discharging power, time window, expected income and service cost, provide the battery health prediction report and SOH expected change curve.

[0051] Preferably, in step 3, the weight coefficient is determined by simulation experiment to determine the optimal proportion: α=0.6, β=0.3, γ=0.1.

[0052] The beneficial effects of the present application are:

[0053] By real-time perception of power grid state, vehicle state and user demand, the optimal charging and discharging strategy and service rate are generated, and the following goals are achieved:

[0054] (1) Dynamic response to power grid state: by adjusting the service rate in real time, the load distribution is optimized, and the peak-valley difference is reduced.

[0055] (2) Multi-objective collaborative optimization: balance user economy, power grid stability and battery life.

[0056] (3) Improve the adaptability of the algorithm: use DRL to handle complex dynamic environment, support multi-user scenario expansion. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The architecture diagram of the V2G dynamic service rate generation method of the present application;

[0058] Figure 2 The deep reinforcement learning training flowchart of the present application. DETAILED DESCRIPTION

[0059] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and gives detailed implementation mode and specific operation process, but the protection scope of the present application is not limited to the following examples.

[0060] The present application proposes a vehicle-to-grid (V2G) dynamic service rate generation method based on deep reinforcement learning (DRL), which dynamically optimizes the charging and discharging strategy and service rate by real-time perception of power grid state, electric vehicle (EV) battery state and user demand, realizes multi-objective collaborative optimization of power grid peak shaving, user income maximization and battery life extension. The technical solution combines the data-driven ability of DRL and the physical characteristics of power system, and is suitable for intelligent charging stations, virtual power plants and other scenarios.

[0061] Step 1: System modeling and state space definition

[0062] Construct a multi-source interactive environment that includes the power grid, charging stations, charging piles, electric vehicles, and users, defined as the state vector S. t :

[0063] S t =[p t ,L t G t V dev SCC t SOH t N max ,T end SOC target ]

[0064] Where, p t Represents real-time electricity prices, obtained through the electricity market interface; L t Represents the grid load factor, which is the ratio of current load to maximum capacity; G t Representing new energy sources, such as photovoltaics and wind power; V dev This represents the number of times the voltage exceeded the limit, counting the number of times the voltage exceeded ±5% of the rated value on a given day; SOC t Represents the vehicle's remaining battery power, collected in real time by the Battery Management System (BMS); SOH t Represents battery health, calculated based on a cycle count and capacity decay model; N max T represents the maximum number of discharges allowed by the user in this instance. end Represents the user's expected latest vehicle pickup time; SOC target This represents the expected battery level when the user picks up the vehicle.

[0065] Step 2: Motion Space Design

[0066] The agent outputs a continuous action vector a t =[P,λ t ], where: charging and discharging power P∈[-P discharge ,P charge Positive values ​​indicate charging, negative values ​​indicate discharging, and the power range is limited by the charging station hardware. Dynamic service fee coefficient λ t ∈[0.5,2], service fee p service The calculation formula is:

[0067] p service =λ t ·(p t +η·max(0, load factor-90%))

[0068] Where η = 0.2 is the load penalty coefficient, when L t A penalty is triggered when the percentage is greater than 90%.

[0069] Step 3: Multi-objective reward function design

[0070] The reward function integrates user benefits, grid stability, and battery life:

[0071] R t = α·R economy + β·R gird - γ·R battery

[0072] User economic benefit R economy :

[0073] R economy = P discharge ·p discharge - P charge ·p charge - p service

[0074] Where p discharge is the discharge price, p chargr is the charging price, and p srrvice is the service fee.

[0075] Grid stability reward R gird :

[0076]

[0077] Where L peak and L valley are the daily load peak and valley values, and L base is the historical benchmark peak-valley difference (30-day average).

[0078] Battery wear penalty R battery :

[0079] R battery = 0.1·(△SOC) 2 + 0.05·N cycle

[0080] Where △SOC is the SOC change amplitude of a single charge-discharge cycle, and N cycle is the number of charge-discharge cycles.

[0081] From the perspective of practical application and user focus, the weight coefficients are: α = 0.6, β = 0.3, and γ = 0.1. The optimal proportion is determined through simulation experiments.

[0082] Step 4: Deep reinforcement learning algorithm implementation

[0083] This step is implemented using the Deep Deterministic Policy Gradient (DDPG) algorithm framework, and the specific scheme is as follows:

[0084] (1) Network architecture design:

[0085] Policy Network (Actor Network): This network is responsible for generating the optimal action based on the current environment state. Its input is an 8-dimensional state vector S representing the current environment state. t The network structure consists of three fully connected hidden layers with 256, 128, and 64 neurons respectively. Each hidden layer uses the ReLU (Rectified Linear Unit) activation function. The output layer generates a 2D action vector 'a'. t These correspond to the power allocation ratio P and the parameter λ, respectively. t To constrain the output range of the action, the output layer uses the hyperbolic tangent (Tanh) activation function.

[0086] Value Network (Critic Network): This network is responsible for evaluating the value of a given state-action pair. Its input layer simultaneously accepts an 8-dimensional state vector S. t and 2D action vector a t The input vector is then concatenated to form a 10-dimensional input vector. Similar to the policy network, the value network also contains three fully connected hidden layers (with 256, 128, and 64 neurons respectively), and uses the ReLU function as the activation function. The output layer generates a scalar value Q(S). t a t ), indicating that in state S t Next, execute action a t The expected accumulated rewards.

[0087] (2) Training process:

[0088] Experience replay mechanism: The transition sample data generated during the interaction between the agent and the environment, i.e., the current state S t Execute action a t Receive instant rewards t and the next state S after the transition t+1 Quadruple (S t ,a t ,r t ,S t+1), which is stored into an Experience Replay Buffer. When training the network parameters, the latest samples are not directly used, but sampled from the Experience Replay Buffer based on the priority of the Temporal Difference Error (TD Error) to improve the data utilization efficiency and training stability.

[0089] Target Network Update: To stabilize the training process, the algorithm maintains a target policy network and a target value network. The parameters θ - Slowly update the parameters θ of the online policy network and online value network through software update. The specific update rule is:

[0090] θ - ← 0.01θ + 0.99θ -

[0091] Exploration Policy: To encourage the agent to fully explore the environment in the early stage of training, Gaussian noise with zero mean is added to the action generated by the policy network. The standard deviation σ of this noise is initially set to 0.3. As the training process progresses, the value of σ gradually linearly decays, and finally stabilizes at 0.1 to achieve the transition from full exploration to effective utilization.

[0092] Step 5: Multi-user Cooperative Expansion

[0093] To adapt to the multi-user scenario (user 1 to user N), this step uses the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework for extension design. The core idea of this framework is "centralized training, decentralized execution", which specifically includes:

[0094] (1) Centralized Critic Design: Each user (agent) is equipped with an independent policy network (Actor) to select actions based on its own local state. However, a centralized value network (Critic) is introduced during the training process. This Critic network can access the joint state information of all users (denoted as global state) and the actions taken by all users (denoted as joint action) during the training phase. Based on this global information, the centralized Critic outputs a global Q value (Q(S,a)), which is used to evaluate the expected future cumulative reward of all users executing joint action a = (a1, a2, …, a N ) in the global state S. This global Q value provides a more comprehensive value guidance signal for training the policy network of each user.

[0095] (2) Attention mechanism integration: In order to improve the efficiency of centralized Critic network in processing multi-user information, and enable it to dynamically focus on the importance difference of different users' contribution to global value, a multi-head attention mechanism is introduced in the hidden layer of the Critic network. The calculation process of the attention mechanism is defined as follows:

[0096]

[0097] Wherein, Q (Query query vector), represents the representation vector of the target user (or target feature) that the Critic network needs to calculate the value at present; K (Key key vector), represents the representation vector of all users (or other feature elements), and the similarity between the user and the query vector Q is calculated; V (Value value vector), represents the actual feature value vector corresponding to the key vector K, and the basis for the final weighted sum; d k (Key Dimension key vector dimension), represents the dimension of each vector in the key vector K.

[0098] Using Scaling is to prevent the gradient of the Softmax function from disappearing due to the large result of the dot product; QK T , calculate the dot product similarity of the query vector Q and all key vectors K; The scaled dot product similarity is normalized by Softmax operation to obtain a set of attention weights. These weights reflect the importance of each key (i.e. each user or feature element) under the current query; The final output of Attention (Q, K, V) is the weighted sum of the value vector V, and the weight is the attention weight calculated above. This enables the Critic network to adaptively focus on the most important user information for the current value evaluation, thereby more effectively capturing the complex dependency relationship between multiple users and dynamically allocating user weights to optimize global decision-making.

[0099] Step 6: Dynamic service rate generation and strategy output

[0100] (1) Service rate calculation: According to the dynamic adjustment of λ t , the formula is shown in step 2.

[0101] (2) User interaction interface: Push the charging and discharging plan to the user, including discharging power, time window, expected income and service cost, provide battery health prediction report and (SOH expected change curve).

[0102] The V2G dynamic service rate generation method based on deep reinforcement learning proposed in the present application has significant advantages in grid peak shaving capacity, user economy, battery life and algorithm efficiency compared with the prior art. The specific beneficial effects are as follows:

[0103] (1) Dynamic service rate optimization capability improves peak load efficiency

[0104] Technical features:

[0105] Dynamic service rate formula:

[0106] Service fee = λ t ·(p t +η·max(0, load rate-90%))

[0107] Where λ t is generated by DRL agent in real time, and η is the load penalty coefficient.

[0108] Reward function design:

[0109] When the grid load rate L t <90%, the reward function encourages to reduce the peak-valley difference; when L t ≥90%, the voltage out-of-limit event is punished.

[0110] Effect analysis:

[0111] Compared with the traditional static electricity price, the existing static electricity price strategy fixes the service rate, which cannot respond to load fluctuations. The present invention adjusts λ t in real time, reduces the discharge rate during the load low period to encourage charging, and increases the rate during the peak period to encourage discharging.

[0112] (2) User economic efficiency is significantly improved

[0113] Technical features:

[0114] Multi-objective reward function:

[0115] R economy =α·(P discharge ·p discharge -P charge ·p charge -p service )

[0116] The weight α=0.6, which prioritizes optimizing the user's net income.

[0117] Charge-discharge strategy optimization: Actor network generates power instruction P, which prioritizes charging during low electricity price and discharging during peak.

[0118] Effect analysis:

[0119] Compared with the traditional fault control, the existing method relies on fixed charging and discharging period, which is easy to miss the optimal electricity price window. The present invention makes real-time decisions through DRL to maximize the electricity price difference arbitrage space.

[0120] (3) battery life extension

[0121] Technical features:

[0122] Battery wear penalty term:

[0123] R battery = 0.1 · (△SOC) 2 + 0.05 · N cycle

[0124] Limit the SOC fluctuation (△SOC) and the number of charge-discharge cycles (N cycle ).

[0125] Action space constraints: the charge-discharge power P is limited by hardware to avoid overcharging / over-discharging.

[0126] Effect analysis:

[0127] Compared with the traditional unconstrained strategy, the existing method does not consider the battery health, which is easy to lead to the annual battery capacity decay rate exceeding 5%. The present application limits the SOC fluctuation (△SOC<30%) through the penalty term, and the experimental data shows that the annual capacity decay rate is reduced to 3.2%, and the life is extended by 20%.

[0128] The above-described embodiments only express one embodiment of the present application, which is described in detail and in detail, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of protection of the present application patent should be subject to the appended claims.

Claims

1.A method for generating a dynamic service rate of V2G based on deep reinforcement learning, characterized in that, By real-time perception of power grid state, electric vehicle battery state and user demand, dynamic optimization of charge and discharge strategy is realized, multi-objective collaborative optimization is realized, multi-source interaction environment including power grid, charging station, charging pile, electric vehicle and user is constructed, state vector is defined, action space design and multi-objective reward function design are carried out, DDPG algorithm framework is used for implementation, MADDPG framework is used for extension design, and the optimal result is pushed to the user after the charging and discharging plan is obtained. 2.The deep reinforcement learning based V2G dynamic service rate generation method of claim 1, wherein, The specific implementation process is as follows: Step 1: system modeling and state space definition A multi-source interaction environment including a power grid, a charging station, a charging pile, an electric vehicle, and a user is constructed, and is defined as a state vector S t : S t = [p t , L t , G t , V dev , SOC t , SOH t , N max , T end , SOC target ] wherein, p t represents real-time electricity price, obtained through the electricity market interface; L t represents grid load rate, i.e. the ratio of current load to maximum capacity; G t represents new energy output; V dev represents voltage overrun times, i.e. the number of times when the voltage exceeds ±5% of the rated value in the current day; SOC t represents the remaining capacity of the vehicle, collected in real time by the battery management system; SOH t represents the battery health, calculated based on the cycle number and capacity attenuation model; N max represents the maximum number of discharges allowed by the user this time T end representing the user's expected latest pickup time; SOC target representing the user's expected battery level at pickup; Step 2: action space design The agent outputs a continuous action vector a t =[P,λ t ], where: charging and discharging power P∈[-P discharge ,P charge Positive values ​​indicate charging, negative values ​​indicate discharging, and the power range is limited by the charging station hardware; dynamic service fee coefficient λ t Service fee p service The calculation formula is: p service = λ t · (p t + η · max(0, load factor - 90%) ) where η = 0.2 is a load penalty coefficient, which triggers a penalty when L t > 90%. Step 3: multi-objective reward function design The reward function optimizes user income, power grid stability and battery life: R t = a · R economy + β · R gird - γ · R battery User economy R economy : R economy = P discharge · p discharge - P charge · p charge - p service where p discharge is the discharge price, p charge is the charge price, p service is the service fee; Grid stability reward R gird : wherein L peak , L valley are the daily load peak and valley values, respectively, and L base is the historical reference peak-to-valley difference. Battery wear penalty R battery : R battery = 0.1 · (△SOC) 2 + 0.05 · N cycle wherein, the SOC change amplitude of single charge-discharge is ΔSOC, N cycle is the number of charge-discharge cycles; Step 4: deep reinforcement learning algorithm implementation This step uses DDPG algorithm framework for implementation, the specific scheme is as follows: Network architecture design: Policy network (Actor network): This network is responsible for generating the optimal action according to the current environment state; its input is an 8-dimensional state vector S representing the current environment state t ; The network structure contains three fully connected hidden layers, with 256, 128, and 64 neurons respectively; ReLU function is used as the activation function for each hidden layer; the output layer generates a 2-dimensional action vector a t , corresponding to the power allocation ratio P and the parameter λ respectively t ; In order to constrain the output range of the action, the output layer uses the hyperbolic tangent activation function; Critic network: This network is responsible for evaluating the value of a given state-action pair; its input layer takes in the 8-dimensional state vector S t and the 2-dimensional action vector a t , which are concatenated to form a 10-dimensional input vector; The value network also contains three fully connected hidden layers with 256, 128, and 64 neurons, respectively, and uses the ReLU function as the activation function; the output layer generates a scalar value Q(S t , a t ), representing the expected cumulative reward of performing action a t in state S t ; Training process: Experience replay mechanism: transition sample data generated in the process of interaction between the agent and the environment, i.e. the current state S t , the executed action a t , the obtained immediate reward r t , and the next state S t+1 after the transition, i.e. the quadruple (S t , a t , r t , S t+1 ) is stored in an experience pool; when training the network parameters, sampling is performed from the experience pool based on the priority of the time difference error TD Error; Target network update: To stabilize the training process, the algorithm maintains a target policy network and a target value network; the parameters θ - Slowly update the parameters θ of the online policy network and online value network through software update; the specific update rule is: θ - ← 0.01 θ + 0.99 θ - Exploration strategy: In order to make the agent fully explore the environment in the early stage of training, add Gaussian noise with mean of zero to the action generated by the policy network; The standard deviation σ of the noise is set to 0.3 initially; With the progress of the training process, the value of σ gradually linearly decays, and finally stabilizes at 0.1; Step 5: multi-user collaborative extension In order to adapt to the multi-user scene, that is, user 1 to user N, this step uses MADDPG framework for extension design, which includes: Centralized Critic design: each user, i.e., agent, is equipped with an independent policy network Actor for selecting actions according to its own local state; however, a centralized value network Critic is introduced in the training process; this Critic network can access the joint state information of all users, i.e., the global state, and the joint actions taken by all users, i.e., a = (a1, a2, …, an) during the training phase; based on this global information, the centralized Critic outputs a global Q value Q(S, a) for evaluating the expected future cumulative reward of all users performing joint actions a = (a1, a2, …, an) in the global state S; this global Q value provides a more comprehensive value guidance signal for training the policy network of each user; N ) Attention mechanism integration: In order to improve the efficiency of centralized Critic network in processing multi-user information, and make it can dynamically focus on the importance difference of different users to the global value contribution, multi-head attention mechanism is introduced in the hidden layer of Critic network; The calculation process of attention mechanism is defined as follows: Wherein Q is the Query query vector, representing the representation vector of the target user or the target feature which the Critic network currently needs to calculate the value; K is the Key key vector, representing the representation vector of all users or other feature elements, and the user calculates the similarity with the query vector Q; V is the Value value vector, representing the actual feature value vector corresponding to the key vector K, and the basis for the final weighted summation; d k is the key vector dimension, representing the dimension of each vector in the key vector K. Use Scaling is performed to prevent the dot product result from being too large, causing the gradient of the Softmax function to disappear; QK T , calculate the dot product similarity of the query vector Q and all key vectors K; Softmax normalization operation is performed on the scaled dot product similarity to obtain a set of attention weights; the final output of Attention(Q, K, V) is the weighted sum of the value vector V, and the weight is the attention weight calculated above; Step 6: dynamic service rate generation and strategy output Service rate calculation: according to λ t Dynamic adjustment, formula see step 2; User interaction interface: Push the charging and discharging plan to the user, including discharge power, time window, expected income and service cost, provide battery health prediction report and SOH expected change curve. 3.The deep reinforcement learning based V2G dynamic service rate generation method of claim 2, wherein, In step 3, the weight coefficients are determined by simulation experiment to obtain the optimal proportion: α=0.6, β=0.3, γ=0.1.

Images