Energy storage system energy management method based on adversarial imitation reinforcement learning

Abstract

The invention relates to the technical field of energy storage systems, and particularly discloses an energy storage system energy management method based on adversarial imitation reinforcement learning, and the method comprises the steps: constructing an adversarial imitation reinforcement network; after the construction of the adversarial imitation enhancement network is completed, performing imitation learning of the adversarial imitation enhancement network; after the imitation learning of the adversarial imitation enhancement network is completed, training the adversarial imitation enhancement network; and after the training of the adversarial imitation enhancement network is completed, managing the energy of the energy storage system by using an output result of the adversarial imitation enhancement network. According to the energy storage system energy management method based on adversarial imitation reinforcement learning, the energy management performance of the energy storage system can be remarkably improved, the service life of a battery is prolonged, and the economical efficiency and reliability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of energy storage system technology, and more specifically, to an energy management method for energy storage systems based on adversarial imitation reinforcement learning. Background Technology

[0002] Energy storage systems (ESS) play a crucial role in modern energy management, especially in the fields of renewable energy and electric vehicles. However, existing energy management methods for energy storage systems suffer from the following technical problems:

[0003] (1) Shortened battery life: Traditional energy management strategies often fail to fully consider the charge and discharge characteristics of batteries, leading to rapid capacity decay and shortened battery life. As the core component of an energy storage system, the battery's lifespan directly affects the system's economy and reliability. Frequent charge and discharge operations accelerate battery aging and increase replacement costs.

[0004] (2) Insufficient optimization performance: Existing rule-based methods struggle to guarantee optimal performance across different driving cycles, while optimization-based methods typically rely on past driving information and precise models, failing to adapt to real-time changing driving conditions. These methods cannot adjust energy allocation strategies in real-time when faced with complex driving scenarios, resulting in low energy utilization efficiency and an inability to achieve optimal energy management.

[0005] (3) Low training efficiency: Although deep reinforcement learning methods have strong self-learning capabilities, they suffer from the "cold start" problem in practical applications. This means that a large amount of trial and error training is required to obtain near-optimal performance, resulting in low training efficiency. A large amount of trial and error training not only increases training time but may also lead to unstable system performance in the initial stage, affecting the feasibility of practical applications.

[0006] (4) Limited generalization ability: Existing methods often exhibit poor generalization ability when faced with new and unseen driving cycles, and cannot effectively adapt to different driving scenarios. This limits the applicability of the system, making it difficult to cope with diverse driving conditions in practical applications, and reducing the system's practicality and flexibility. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an energy management method for energy storage systems based on adversarial imitation reinforcement learning, which can significantly improve the energy management performance of energy storage systems, extend battery life, and improve the economy and reliability of the system.

[0008] As a first aspect of the present invention, an energy management method for an energy storage system based on adversarial imitation reinforcement learning is provided, comprising the following steps:

[0009] Step S1: Construct an adversarial mimicry reinforcement network;

[0010] Step S2: After the adversarial imitation reinforcement network is constructed, the adversarial imitation reinforcement network performs imitation learning;

[0011] Step S3: After the adversarial imitation reinforcement network completes its imitation learning, the adversarial imitation reinforcement network is trained.

[0012] Step S4: After the adversarial imitation reinforcement network is trained, the energy of the energy storage system is managed using the output of the adversarial imitation reinforcement network.

[0013] Furthermore, step S1 also includes:

[0014] Step S11: Construct an energy storage system, wherein the energy storage system includes a battery model and a supercapacitor model;

[0015] (1) The battery model adopts the equivalent internal resistance model, and its open-circuit voltage, output power, power loss and state of charge are expressed as follows:

[0016] V ocv =V bat +I bat R bat

[0017] P bat =V bat I bat

[0018]

[0019] Among them, V ocv It is the open-circuit voltage of the battery, V. bat It is the battery's terminal voltage, I. bat It is the battery current, R bat It is the internal resistance of the battery, P bat It is the battery's output power, P loss,bat It's the power loss from the battery; SoC bat,k+1 It is the state of charge of the battery at time k+1, SoC bat,k C is the state of charge of the battery at time k. bat It is the battery's rated capacity, Δt is the sampling time, and I bat,k It is the battery current at time k;

[0020] (2) The supercapacitor model adopts a combination of ideal capacitance and internal resistance, and its voltage relationship, power loss and state of charge are expressed as follows:

[0021] V sc =VC -I sc R sc

[0022]

[0023] Among them, V sc It is the terminal voltage of the supercapacitor, V. C It is the voltage across an ideal capacitor, I sc It is the current of the supercapacitor, R sc It is the internal resistance of the supercapacitor, P loss This refers to the power loss of supercapacitors, and the SoC (System-on-a-Chip). sc,k+1 It is the state of charge of the supercapacitor at time k+1, SoC sc,k It is the state of charge of the supercapacitor at time k, I sc,k It is the current of the supercapacitor at time k, Δt is the sampling time, and C sc This is the rated capacitance of the supercapacitor, V. sc,max That is the maximum voltage of the supercapacitor;

[0024] Step S12: Define the state space and action space;

[0025] (1) The state space S includes the state of charge of the battery, the state of charge of the supercapacitor, and the load power P. load :

[0026] S = [SoC] sc SoC bat ,P load ]

[0027] Among them, SoC sc It refers to the state of charge of a supercapacitor, SoC bat It refers to the state of charge (P) of the battery. load This is the load power, which needs to be normalized using Min-Max:

[0028]

[0029] Among them, P min and P max These are the minimum and maximum values ​​of the load power, P. norm This is the normalized load power;

[0030] (2) The operating space includes the power distribution of the supercapacitor and the power distribution of the battery;

[0031] Power distribution of supercapacitor A:

[0032] A = [P] sc ]

[0033] Among them, Psc It is the output power of the supercapacitor;

[0034] Battery power distribution is achieved through load power P load Subtract the output power P of the supercapacitor sc To calculate:

[0035] P bat =P load -P sc

[0036] Among them, P bat It is the battery's output power;

[0037] Step S13: Design the reward function R, where the reward function R guides the agent's decision-making direction, and the optimization objective includes the battery capacity loss cost R. bat The total energy loss cost R of the energy storage system HESS :

[0038] R = -(R penalty +R HESS )

[0039] Where R is the reward function, R penalty It is a penalty for violating safety constraints, R HESS It is the total energy loss cost of the energy storage system;

[0040] (1) The capacity loss cost R of the battery bat , is represented as:

[0041]

[0042] Among them, C bat It is the unit cost of the battery, ΔSoC bat It is the loss of battery capacity;

[0043] (2) Total energy loss cost R of the energy storage system HESS Represented as:

[0044]

[0045] Among them, P loss,bat,k It is the power loss of the battery at time k, P loss,sc,k P is the power loss of the supercapacitor at time k. loss.converter,k This represents the power loss of the DC / DC converter at time k.

[0046] Step S14: Generate expert knowledge using dynamic programming, including the optimal power allocation mapping;

[0047] (1) Generate discretized optimal power allocation under typical driving cycles using dynamic programming:

[0048]

[0049] in, Let $k$ represent the optimal cumulative cost from time $k$ to the termination time. Let C represent the optimal cumulative cost from time k+1 to the final time. HESS,k P is the energy loss cost of the energy storage system at time k. sc,r It is the output power of the supercapacitor at time k;

[0050] (2) The discretized optimal power allocation is converted into a continuous form by cubic spline interpolation:

[0051]

[0052] Where ai are the interpolation coefficients; It is the optimal power allocation function for supercapacitors; SoC sc It refers to the state of charge of a supercapacitor, SoC sc,i is the supercapacitor charge state at discrete control point i; n is the number of discrete control points.

[0053] Furthermore, step S2 also includes:

[0054] (1) Improve the generative adversarial network to generate the adversarial imitation enhancement network;

[0055] The agent network and the discriminant network constitute the generative adversarial network, which is used to learn expert knowledge.

[0056]

[0057] in, It is the loss function for imitation learning; D(S,A) is the output of the discriminant network, representing the probability that the state-action pair (S,A) comes from expert knowledge; π agent It is the agent strategy, representing the probability distribution of an agent choosing action A given state S; π expert It is an expert strategy, representing the probability distribution of an expert choosing action A given state S; Indicating in the agent strategy π agent The expected value of the state-action pair (S,A) under the given conditions; In the expert strategy π expert The expected value of the state-action pair (S,A) under the given conditions;

[0058] (2) A smooth transition from imitation learning to self-exploration is achieved through dynamic imitation weights;

[0059] Dynamic imitation weights are used to balance imitation learning and self-exploration:

[0060]

[0061] Where λ(t) is the dynamic imitation weight at the current time step t; t is the current training step number; T0 is the maximum training step number;

[0062] When the output of the discriminant network stabilizes around 0.5, accelerate the decay of the dynamic imitation weights:

[0063] λ(t) = λ(t-1) × λ(t)

[0064] Where λ(t-1) is the dynamic imitation weight of the previous time step t-1.

[0065] Furthermore, step S3 also includes:

[0066] Step S31: Initialize the adversarial mimicry reinforcement network;

[0067] (1) Initialize the agent network πθ(A|S), the Critic network Qφ(S,A) and the discriminant network Dψ(S,A);

[0068] (2) Initialize the target agent network πθ′ and the target Critic network Qφ′;

[0069] (3) Initialize the dynamic imitation weight λ;

[0070] (4) Initialize the experience playback buffer D;

[0071] Step S32: Train the adversarial imitation reinforcement network;

[0072] (1) Select action: Based on the current state St, select action At through the agent network πθ(A|S);

[0073] (2) Perform the action: Perform the action At in the environment and obtain the next state St+1 and reward Rt;

[0074] (3) Store experience: Store the experience (St,At,Rt,St+1) into the experience replay buffer D;

[0075] (4) Sampling: Randomly sample a batch of experience {(Si,Ai,Ri,Si+1)} and expert experience from the experience playback buffer D. in, This indicates the state of the energy storage system at a certain time step i; This represents the optimal power allocation of the supercapacitor selected by the expert strategy at a certain time step i.

[0076] (5) Update the Critic network Qφ(S,A):

[0077]

[0078] in, The gradient loss of the Critic network; For experience replay buffer; S i This indicates the state of the energy storage system at time step i; S i+1 This indicates the state of the energy storage system at time step i+1; A i This represents the action selected by the agent at time step i; R i This indicates that at time step i, the agent executes action A. i The reward obtained later; γ represents the discount factor; A i+1 This represents the action selected by the agent at time step i+1; Q φ (S i A i ) indicates that in state S i Take action A i Expected returns;

[0079] Q φ′ (S i+1 A i+1 ) indicates that in state S i+1 Take action A i+1 Expected returns;

[0080] Indicates that in state S i+1 The maximum expected return that a sub-agent can obtain;

[0081] (6) Update the agent network πθ(A|S):

[0082]

[0083] in, π is the gradient loss of the agent network; θ (A|S i ) represents the policy of the proxy network; For strategy π θ (A|S i The gradient with respect to the parameter θ; Q represents the Q value. φ (S i A) Gradient of action A; λ is the dynamic imitation weight; D ψ (S i A) represents a state-action pair (S) iA) Probability derived from expert strategy; D represents probability ψ (S i A) The gradient of action A;

[0084] (7) Update the discriminant network Dψ(S,A):

[0085]

[0086] in, D is the gradient loss of the discriminant network; ψ (S i A i ) represents a state-action pair (S) i A i The probability derived from the agent policy; log D ψ (S i A i The state-action pairs (S) generated for the agent i A i The logarithmic probability of ). State-action pairs generated for experts The logarithmic probability;

[0087] (8) Update the dynamic imitation weight λ(t) based on the current training step t;

[0088] (9) Update the parameters of the target agent network πθ′ and the target Critic network Qφ′ using a soft update strategy:

[0089] θ′←τθ+(1-τ)θ′

[0090] φ′←τφ+(1-τ)φ′

[0091] Where θ′ represents the parameters of the target proxy network πθ′; φ′ represents the parameters of the target Critic network Qφ′; τ represents the soft update rate; θ represents the parameters of the current proxy network; and φ represents the parameters of the current Critic network.

[0092] The energy management method for energy storage systems based on adversarial imitation reinforcement learning provided by this invention has the following beneficial effects:

[0093] (1) Minimize battery capacity loss cost: By optimizing energy distribution, the number of charge and discharge cycles and the depth of charge and discharge can be reduced, thereby minimizing battery capacity loss cost and extending battery life;

[0094] (2) Improve optimization performance: Use expert knowledge to guide the exploration process of reinforcement learning agents to ensure that near-optimal energy management strategies can be achieved under different driving conditions, thereby improving the energy utilization efficiency of the system;

[0095] (3) Accelerating the training process: By utilizing adversarial imitation learning (GAIL) and deep reinforcement learning (DRL), expert knowledge is leveraged to accelerate the training process, ensuring that the system maintains high performance under real-time changing driving conditions. Expert knowledge guides the initial training of the agent, significantly reducing the "cold start" problem and accelerating the training process. This reduces the number of trial and error attempts required for training, improves training efficiency, and ensures that the system exhibits good performance in the initial stage.

[0096] (4) Enhanced generalization ability: By dynamically mimicking weights, the system can smoothly transition to the self-exploration stage, enhancing its adaptability to different driving cycles and ensuring that the system maintains good performance when facing new driving scenarios. This enables the system to adapt to different driving cycles, improves its generalization ability, and enhances its applicability in practical applications. Attached Figure Description

[0097] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0098] Figure 1 The flowchart illustrates an energy management method for an energy storage system based on adversarial imitation reinforcement learning, as provided by this invention.

[0099] Figure 2 The flowchart illustrates a specific implementation method for an energy management method for an energy storage system based on adversarial imitation reinforcement learning, as provided by this invention. Detailed Implementation

[0100] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an energy management method for an energy storage system based on adversarial imitation reinforcement learning proposed according to the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0101] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0102] This embodiment provides an energy management method for energy storage systems based on adversarial imitation reinforcement learning, such as... Figure 1 As shown, the energy management method for energy storage systems based on adversarial imitation reinforcement learning includes the following steps:

[0103] Step S1: Construct an adversarial mimicry reinforcement network;

[0104] Preferably, such as Figure 2 As shown, step S1 further includes:

[0105] Step S11: Construct an energy storage system, wherein the energy storage system includes a battery model and a supercapacitor model to achieve the complementary advantages of high energy density and high power density;

[0106] (1) The battery model adopts the equivalent internal resistance model, and its open-circuit voltage, output power, power loss and state of charge (SOC) are expressed as follows:

[0107] V ocv =V bat +I bat R bat

[0108] P bat =V bat I bat

[0109]

[0110] Among them, V ocv It is the open-circuit voltage of the battery, V. bat It is the battery's terminal voltage, I. bat It is the battery current (negative for charging, positive for discharging), R bat It is the internal resistance of the battery, P bat It is the battery's output power, P loss,bat It's the power loss from the battery; SoC bat,k+1 It is the state of charge of the battery at time k+1, SoCbat,k C is the state of charge of the battery at time k. bat It is the battery's rated capacity, Δt is the sampling time, and I bat,k It is the battery current at time k;

[0111] (2) The supercapacitor model adopts a combination of ideal capacitance and internal resistance, and its voltage relationship, power loss and state of charge (SOC) are expressed as follows:

[0112] V sc =V C -I sc R sc

[0113]

[0114]

[0115] Among them, V sc It is the terminal voltage of the supercapacitor, V. C It is the voltage across an ideal capacitor, I sc It is the current of the supercapacitor, R sc It is the internal resistance of the supercapacitor, P loss This refers to the power loss of supercapacitors, and the SoC (System-on-a-Chip). sc,k+1 It is the state of charge of the supercapacitor at time k+1, SoC sc,k It is the state of charge of the supercapacitor at time k, I sc,k It is the current of the supercapacitor at time k, Δt is the sampling time, and C sc This is the rated capacitance of the supercapacitor, V. sc,max That is the maximum voltage of the supercapacitor;

[0116] Step S12: To implement the energy management strategy, it is necessary to define the state space and action space;

[0117] (1) The state space S includes the state of charge of the battery, the state of charge of the supercapacitor, and the load power P. load :

[0118] S = [SoC] sc SoC bat ,P load ]

[0119] Among them, SoC sc It refers to the state of charge of a supercapacitor, SoC bat It refers to the state of charge (P) of the battery. load This is the load power, which needs to be normalized using Min-Max:

[0120]

[0121] Among them, P min and P max These are the minimum and maximum values ​​of the load power, P. norm This is the normalized load power;

[0122] (2) The operating space includes the power distribution of the supercapacitor and the power distribution of the battery;

[0123] Power distribution of supercapacitor A:

[0124] A = [P] sc ]

[0125] Among them, P sc It is the output power of the supercapacitor;

[0126] Battery power distribution is achieved through load power P load Subtract the output power P of the supercapacitor sc To calculate:

[0127] P bat =P load -P sc

[0128] Among them, P bat It is the battery's output power;

[0129] Step S13: Design the reward function R, where the reward function R guides the agent's decision-making direction, and the optimization objective includes the battery capacity loss cost R. bat The total energy loss cost R of the energy storage system HESS :

[0130] R = -(R penalty +R HESS )

[0131] Where R is the reward function, R penalty It is a penalty for violating safety constraints, R HESS It is the total energy loss cost of the energy storage system;

[0132] (1) The capacity loss cost R of the battery bat , is represented as:

[0133]

[0134] Among them, C bat It is the unit cost of the battery, ΔSoC bat It is the loss of battery capacity;

[0135] (2) Total energy loss cost R of the energy storage system HESS Represented as:

[0136]

[0137] Among them, P loss,bat,k It is the power loss of the battery at time k, P loss,sc,k P is the power loss of the supercapacitor at time k. loss.converter,k This represents the power loss of the DC / DC converter at time k.

[0138] Step S14: Generate expert knowledge, including the optimal power allocation mapping, using dynamic programming (DP);

[0139] (1) Generate discretized optimal power allocation under typical driving cycles using dynamic programming (DP):

[0140]

[0141] in, Let $k$ represent the optimal cumulative cost from time $k$ to the termination time. Let C represent the optimal cumulative cost from time k+1 to the final time. HESS,k P is the energy loss cost of the energy storage system at time k. sc,k It is the output power of the supercapacitor at time k;

[0142] (2) The discretized optimal power allocation is converted into a continuous form by cubic spline interpolation:

[0143]

[0144] Where ai are the interpolation coefficients, which are obtained by solving the interpolation conditions and the smoothness conditions; It is the optimal power allocation function for supercapacitors; SoC sc It refers to the state of charge of a supercapacitor, SoC sc,i is the supercapacitor charge state at discrete control point i; n is the number of discrete control points. i is an index variable used to iterate through all discrete control points during the summation process, with each i corresponding to a specific discrete control point.

[0145] Step S2: After the adversarial imitation reinforcement network is constructed, the adversarial imitation reinforcement network performs imitation learning;

[0146] It's important to note that the state space and action space are fundamental to the adversarial imitation reinforcement network learning framework. They define the states that the agent can perceive in the environment and the actions it can take. In other words, the definitions of the state space and action space provide the basis for decision-making in adversarial imitation reinforcement network learning. The agent selects actions in the action space by perceiving information in the state space to achieve the optimization objective. The expert knowledge (optimal power allocation mapping) generated through offline optimization is also based on these state and action definitions. Therefore, the definitions of the state space and action space are the cornerstone of the entire adversarial imitation reinforcement network learning framework, ensuring compatibility and consistency among all components.

[0147] It's important to note that the reward function is a core component of adversarial imitation reinforcement network learning, guiding the agent's decision-making. It defines the immediate reward the agent receives when taking an action at each time step. The agent's goal is to maximize the cumulative reward, which is equivalent to minimizing the total cost. The expert knowledge (optimal power allocation mapping) generated during offline optimization is based on the same optimization objective. Therefore, the reward function not only guides the online learning process but also ensures consistency between offline optimization and online learning.

[0148] Preferably, such as Figure 2 As shown, step S2 further includes:

[0149] (1) Improve the Generative Adversarial Network (GAN) to generate the adversarial imitation reinforcement network;

[0150] The agent network and the discriminator network constitute the generative adversarial network, which is used to learn expert knowledge.

[0151]

[0152] in, π is the loss function for imitation learning, specifically the loss function in Generative Adversarial Imitation Learning (GAIL). It measures the difference between the agent's behavior and the expert's behavior. By minimizing this loss function, the agent can learn behaviors similar to the expert. D(S,A) is the output of the discriminant network, representing the probability that the state-action pair (S,A) comes from expert knowledge. If D(S,A) is close to 1, it indicates that the state-action pair (S,A) is likely from the expert; if D(S,A) is close to 0, it indicates that the state-action pair (S,A) is likely from the agent. agent It is the agent strategy, representing the probability distribution of an agent choosing action A given state S; π expertIt is an expert strategy, which represents the probability distribution of an expert choosing action A given state S. Expert strategies are usually the optimal strategies generated through offline optimization (such as dynamic programming). Indicating in the agent strategy π agent The expected value of the state-action pair (S,A) under the given conditions; In the expert strategy π expert The expected value of the state-action pair (S,A) under the given conditions;

[0153] (2) A smooth transition from imitation learning to self-exploration is achieved through dynamic imitation weights;

[0154] Dynamic imitation weights are used to balance imitation learning and self-exploration:

[0155]

[0156] Where λ(t) is the dynamic imitation weight at the current time step t, which is a weight that changes over time and is used to adjust the balance between imitation learning and self-exploration. In the early stages of training, λ(t) is larger, and the agent relies more on expert knowledge for imitation learning; as training progresses, λ(t) gradually decreases, and the agent gradually increases the proportion of self-exploration; t is the current training step; T0 is the maximum training step.

[0157] When the output of the discriminant network stabilizes around 0.5, accelerate the decay of the dynamic imitation weights:

[0158] λ(t) = λ(t-1) × λ(t)

[0159] Where λ(t-1) is the dynamic imitation weight of the previous time step t-1.

[0160] Step S3: After the adversarial imitation reinforcement network completes its imitation learning, the adversarial imitation reinforcement network is trained.

[0161] Preferably, such as Figure 2 As shown, step S3 further includes:

[0162] Step S31: Initialize the adversarial mimicry reinforcement network;

[0163] (1) Initialize the agent network πθ(A|S), the Critic network Qφ(S,A) and the discriminant network Dψ(S,A);

[0164] (2) Initialize the target agent network πθ′ and the target Critic network Qφ′;

[0165] (3) Initialize the dynamic imitation weight λ;

[0166] (4) Initialize the experience playback buffer D;

[0167] Step S32: Train the adversarial imitation reinforcement network;

[0168] (1) Select action: Based on the current state St, select action At through the agent network πθ(A|S);

[0169] (2) Perform the action: Perform the action At in the environment and obtain the next state St+1 and reward Rt;

[0170] (3) Store experience: Store the experience (St,At,Rt,St+1) into the experience replay buffer D;

[0171] (4) Sampling: Randomly sample a batch of experience {(Si,Ai,Ri,Si+1)} and expert experience from the experience playback buffer D. in, The state refers to the state in the expert experience, which is a state sampled from the expert strategy. It represents the state of the energy storage system at a certain time step i. The state usually includes the state of charge of the battery, the state of charge of the supercapacitor, the load power, etc. The action refers to the action in the expert's experience. This action is sampled from the expert policy and represents the optimal power allocation of the supercapacitor selected by the expert policy at a certain time step i.

[0172] (5) Update the Critic network Qφ(S,A):

[0173]

[0174] in, The gradient loss of the Critic network; As an experience replay buffer, it stores experiences (S) generated during the interaction between the agent and the environment. i A i ,R i ,S i+1 );S i This indicates the state of the energy storage system at time step i; S i+1 This indicates the state of the energy storage system at time step i+1; A i This represents the action selected by the agent at time step i; R i This indicates that at time step i, the agent executes action A. i The reward obtained later; γ represents the discount factor; A i+1 This represents the action selected by the agent at time step i+1; Q φ (S i A i ) represents the Q-value of the Critic network, indicating the state S.i Take action A i Expected return; Q φ′ (S i+1 A i+1 Let be the Q-value of the target Critic network, representing the value in state S. i+1 Take action A i+1 Expected returns; Indicates that in state S i+1 The maximum expected return (Q value) that a downstream agent can obtain;

[0175] (6) Update the agent network πθ(A|S):

[0176]

[0177] in, π is the gradient loss of the agent network; θ (A|S i ) represents the policy of the agent network, indicating that in state S i Below is the probability distribution of the agent choosing action A; For strategy π θ (A|S i The gradient with respect to the parameter θ; The gradient of Q is represented by Q. φ (S i A) The gradient of action A; λ is the dynamic imitation weight, used to adjust the balance between imitation learning and self-exploration; D ψ (S i A) is the output of the discriminant network, representing the state-action pair (S). i A) Probability derived from expert strategy; To determine the gradient of the network, let probability D be represented. ψ (S i A) The gradient of action A;

[0178] (7) Update the discriminant network Dψ(S,A):

[0179]

[0180] in, D is the gradient loss of the discriminant network; ψ (S i A i The output of the discriminant network represents the state-action pair (S). i A i The probability derived from the proxy policy; logD ψ (S i A iThe state-action pairs (S) generated for the agent i A i The logarithmic probability of ) represents the judgment network's judgment on the state-action pair generated by the agent; State-action pairs generated for experts The logarithmic probability represents the judgment network's assessment of the expert-generated state-action pair;

[0181] (8) Update the dynamic imitation weight λ(t) based on the current training step t;

[0182] (9) Update the parameters of the target agent network πθ′ and the target Critic network Qφ′ using a soft update strategy:

[0183] θ′←τθ+(1-τ)θ′

[0184] φ′←τφ+(1-τ)φ′

[0185] Where θ′ represents the parameters of the target proxy network πθ′; φ′ represents the parameters of the target Critic network Qφ′; τ represents the soft update rate, a small constant between 0 and 1 used to control the update speed of the target network parameters; θ represents the parameters of the current proxy network; and φ represents the parameters of the current Critic network.

[0186] Step S4: After the adversarial imitation reinforcement network is trained, the output of the adversarial imitation reinforcement network is used to manage the energy of the energy storage system to ensure that the battery capacity loss cost R is minimized under different scenarios. bat The total energy loss cost R of the minimum and energy storage system HESS Minimum.

[0187] In this embodiment of the invention, (1) a highly efficient energy management strategy is achieved by combining Generative Adversarial Imitation Learning (GAIL) and Deep Reinforcement Learning (DRL). The exploration process of the reinforcement learning agent is guided by expert knowledge, accelerating the training process and improving optimization performance. (2) Dynamic imitation weights are designed to enable the system to smoothly transition from the expert-guided imitation learning stage to the self-exploration stage. This not only avoids over-reliance on expert knowledge but also improves the system's adaptability and generalization ability. (3) Discrete optimal control is converted into a continuous form through cubic spline interpolation, generating continuous expert knowledge. This allows the invention to learn the optimal energy allocation strategy in a continuous state space, improving control accuracy. (4) By optimizing the power allocation between the battery and supercapacitor, the battery capacity loss cost and total energy loss cost are minimized. This invention can achieve a near-optimal energy management strategy under different driving conditions, extending battery life and improving energy utilization efficiency.

[0188] In summary, the energy management method for energy storage systems based on Adversarial Imitation Reinforcement Learning (AIRL) provided by this invention aims to minimize battery capacity loss costs while optimizing energy allocation and extending battery life. By combining Generative Adversarial Imitation Learning (GAIL) and Deep Reinforcement Learning (DRL), expert knowledge is utilized to accelerate the training process, and dynamic imitation weights enable a smooth transition from imitation learning to self-exploration. This method can significantly improve the energy management performance of energy storage systems, extend battery life, and enhance the system's economy and reliability.

[0189] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An energy management method for energy storage systems based on adversarial imitation reinforcement learning, characterized in that, The energy management method for energy storage systems based on adversarial imitation reinforcement learning includes the following steps: Step S1: Construct an adversarial mimicry reinforcement network; Step S2: After the adversarial imitation reinforcement network is constructed, the adversarial imitation reinforcement network performs imitation learning; Step S3: After the adversarial imitation reinforcement network completes its imitation learning, the adversarial imitation reinforcement network is trained. Step S4: After the adversarial imitation reinforcement network is trained, the energy of the energy storage system is managed using the output of the adversarial imitation reinforcement network.

2. The energy management method for an energy storage system based on adversarial imitation reinforcement learning according to claim 1, characterized in that, Step S1 further includes: Step S11: Construct an energy storage system, wherein the energy storage system includes a battery model and a supercapacitor model; (1) The battery model adopts the equivalent internal resistance model, and its open-circuit voltage, output power, power loss and state of charge are expressed as follows: V ocv =V bat +I bat R bat P bat =V bat I bat Among them, V ocv It is the open-circuit voltage of the battery, V. bat It is the battery's terminal voltage, I. bat It is the battery current, R bat It is the internal resistance of the battery, P bat It is the battery's output power, P loss,bat It's the power loss from the battery; SoC bat,k+1 It is the state of charge of the battery at time k+1, SoC bat,k It is the state of charge of the battery at time k, C bat It is the battery's rated capacity, Δt is the sampling time, and I bat,r It is the battery current at time k; (2) The supercapacitor model adopts a combination of ideal capacitance and internal resistance, and its voltage relationship, power loss and state of charge are expressed as follows: V sc =V C -I sc R sc Among them, V sc It is the terminal voltage of the supercapacitor, V. C It is the voltage across an ideal capacitor, I sc It is the current of the supercapacitor, R sc It is the internal resistance of the supercapacitor, P loss This refers to the power loss of supercapacitors, and the SoC (System-on-a-Chip). sc,k+1 It is the state of charge of the supercapacitor at time k+1, SoC sc,k It is the state of charge of the supercapacitor at time k, I sc,k It is the current of the supercapacitor at time k, Δt is the sampling time, and C sc This is the rated capacitance of the supercapacitor, V. sc,max That is the maximum voltage of the supercapacitor; Step S12: Define the state space and action space; (1) The state space S includes the state of charge of the battery, the state of charge of the supercapacitor, and the load power P. load : S=[SoC sc ,SoC bat ,P load ] Among them, SoC sc It refers to the state of charge of a supercapacitor, SoC bat It refers to the state of charge (P) of the battery. load This is the load power, which needs to be normalized using Min-Max: Among them, P min and P max These are the minimum and maximum values ​​of the load power, P. norm It is the normalized load power; (2) The operating space includes the power distribution of the supercapacitor and the power distribution of the battery; Power distribution of supercapacitor A: A=[P sc ] Among them, P sc It is the output power of the supercapacitor; Battery power distribution is achieved through load power P load Subtract the output power P of the supercapacitor sc To calculate: P bat =P load -P sc Among them, P bat It is the battery's output power; Step S13: Design the reward function R, where the reward function R guides the agent's decision-making direction, and the optimization objective includes the battery capacity loss cost R. bat The total energy loss cost R of the energy storage system HESS : R=-(R penalty +R HESS ) Where R is the reward function, R penalty It is a penalty for violating safety constraints, R HESS It is the total energy loss cost of the energy storage system; (1) The capacity loss cost R of the battery bat , is represented as: Among them, C bat It is the unit cost of the battery, ΔSoC bat It is the loss of battery capacity; (2) Total energy loss cost R of the energy storage system HESS Represented as: Among them, P loss,bat,k It is the power loss of the battery at time k, P loss,sc,k P is the power loss of the supercapacitor at time k. loss.converter,k This represents the power loss of the DC / DC converter at time k. Step S14: Generate expert knowledge using dynamic programming, including the optimal power allocation mapping; (1) Generate discretized optimal power allocation under typical driving cycles using dynamic programming: in, Let $k$ represent the optimal cumulative cost from time $k$ to the termination time. Let C represent the optimal cumulative cost from time k+1 to the final time. HESS,K P is the energy loss cost of the energy storage system at time K. sc,k It is the output power of the supercapacitor at time k; (2) The discretized optimal power allocation is converted into a continuous form by cubic spline interpolation: Where ai are the interpolation coefficients; It is the optimal power allocation function for supercapacitors; SoC sc It refers to the state of charge of a supercapacitor, SoC sc,i is the supercapacitor charge state at discrete control point i; n is the number of discrete control points.

3. The energy management method for an energy storage system based on adversarial imitation reinforcement learning according to claim 2, characterized in that, Step S2 further includes: (1) Improve the generative adversarial network to generate the adversarial imitation enhancement network; The agent network and the discriminant network constitute the generative adversarial network, which is used to learn expert knowledge. in, It is the loss function for imitation learning; D(S,A) is the output of the discriminant network, representing the probability that the state-action pair (S,A) comes from expert knowledge; π agent It is the agent strategy, representing the probability distribution of an agent choosing action A given state S; π expert It is an expert strategy, representing the probability distribution of an expert choosing action A given state S; Indicating in the agent strategy π agent The expected value of the state-action pair (S,A) under the given conditions; In the expert strategy π expert The expected value of the state-action pair (S,A) under the given conditions; (2) A smooth transition from imitation learning to self-exploration is achieved through dynamic imitation weights; Dynamic imitation weights are used to balance imitation learning and self-exploration: Where λ(t) is the dynamic imitation weight at the current time step t; t is the current training step number; T0 is the maximum training step number; When the output of the discriminant network stabilizes around 0.5, accelerate the decay of the dynamic imitation weights: λ(t) = λ(t-1) × λ(t) Where λ(t-1) is the dynamic imitation weight of the previous time step t-1.

4. The energy management method for an energy storage system based on adversarial imitation reinforcement learning according to claim 3, characterized in that, Step S3 further includes: Step S31: Initialize the adversarial mimicry reinforcement network; (1) Initialize the agent network πθ(A|S), the Critic network Qφ(S,A) and the discriminant network Dψ(S,A); (2) Initialize the target agent network πθ′ and the target Critic network Qφ′; (3) Initialize the dynamic imitation weight λ; (4) Initialize the experience playback buffer D; Step S32: Train the adversarial imitation reinforcement network; (1) Select action: Based on the current state St, select action At through the agent network πθ(A|S); (2) Perform the action: Perform the action At in the environment and obtain the next state St+1 and reward Rt; (3) Store experience: Store the experience (St,At,Rt,St+1) into the experience replay buffer D; (4) Sampling: Randomly sample a batch of experience {(Si,Ai,Ri,Si+1)} and expert experience from the experience playback buffer D. in, This indicates the state of the energy storage system at a certain time step i; This represents the optimal power allocation of the supercapacitor selected by the expert strategy at a certain time step i. (5) Update the Critic network Qφ(S,A): in, The gradient loss of the Critic network; For experience replay buffer; S i This indicates the state of the energy storage system at time step i; S i+1 This indicates the state of the energy storage system at time step i+1; A i This represents the action selected by the agent at time step i; R i This indicates that at time step i, the agent executes action A. i The reward obtained later; γ represents the discount factor; A i+1 This represents the action selected by the agent at time step i+1; Q φ (S i A i ) indicates that in state S i Take action A i Expected return; Q φ′ (S i+1 A i+1 ) indicates that in state S i+1 Take action A i+1 Expected returns; Indicates that in state s i+1 The maximum expected return that a sub-agent can obtain; (6) Update the agent network πθ(A|S): in, π is the gradient loss of the agent network; θ (A|S i ) represents the policy of the proxy network; For strategy π θ (A|S i The gradient with respect to the parameter θ; Q represents the Q value. φ (S i A) Gradient of action A; λ is the dynamic imitation weight; D ψ (S i A) represents a state-action pair (S) i A) Probability derived from expert strategy; D represents probability ψ (S i A) The gradient of action A; (7) Update the discriminant network Dψ(S,A): in, D is the gradient loss of the discriminant network; ψ (S i A i ) represents a state-action pair (S) i A i The probability derived from the agent policy; log D ψ (S i A i The state-action pairs (S) generated for the agent i A i The logarithmic probability of ). State-action pairs generated for experts The logarithmic probability; (8) Update the dynamic imitation weight λ(t) based on the current training step t; (9) Update the parameters of the target agent network πθ′ and the target Critic network Qφ′ using a soft update strategy: θ′←τθ+(1-τ)θ′ φ′←τφ+(1-τ)φ′ Where θ′ represents the parameters of the target proxy network πθ′; φ′ represents the parameters of the target Critic network Qφ′; τ represents the soft update rate; θ represents the parameters of the current proxy network; and φ represents the parameters of the current Critic network.

Images