Battery charging and discharging optimization method, system and medium based on deep reinforcement learning

Abstract

The present invention discloses a battery charging and discharging optimization method, system and medium based on deep reinforcement learning, which forms a state set and a charging and discharging action strategy set based on historical data related to charging and discharging, and then trains a target network model. The main network of the target network model includes a prediction actor network and a prediction critic network, wherein the prediction actor network includes a current strategy prediction network and an auxiliary strategy prediction network. The current strategy prediction network is composed of a diffusion model, which reduces the sensitivity of the model and enhances the strategy generation ability and strategy exploration ability of the model. Based on the auxiliary strategy prediction and the output of the target actor network, comparative learning is performed to assist in improving the prediction ability of the model and solve the problem of over-estimation of the action strategy. The present invention can better cope with complex environmental changes, is more intelligent, can accurately give charging and discharging strategies, extend the service life of the battery, reduce the loss cost of the manufacturer, and has good practicality.

Description

Technical Field

[0001] The present invention belongs to the technical field of energy storage battery charging and discharging, and specifically relates to a battery charging and discharging optimization method, system and medium based on deep reinforcement learning. Background Art

[0002] Lithium batteries have the characteristics of high power density, high capacity, and high durability, and are favored by more fields. Each type of lithium battery has certain advantages and disadvantages and is suitable for different application scenarios. Energy storage batteries composed of lithium batteries have been widely used in industrial energy storage, data center backup power, emergency power supply and other scenarios, but the extensive charging and discharging mode will cause the battery to be overcharged and over-discharged for a long time, shorten the battery life cycle, and directly increase the economic cost of suppliers. Therefore, how to extend the service life of the battery and improve the efficiency of the energy storage battery has become a key issue for suppliers.

[0003] Commonly used energy storage battery charging and discharging strategy optimization methods can be divided into value estimation-based reinforcement learning methods, policy optimization-based reinforcement learning methods, and actor-critic-based reinforcement learning methods. The core idea of ​​the value estimation-based reinforcement learning method is to estimate the optimal Q value under all state actions and find the action strategy that maximizes the optimal Q value, that is, the optimal action strategy, but the action space must be discrete and cannot handle continuous action space. The policy optimization-based reinforcement learning method is divided into stochastic strategy and deterministic strategy. The former estimates the probability distribution mapping between state and action, and finally iteratively samples the action strategy from the probability distribution, while the latter takes the determined strategy performance function as the optimization target, obtains the optimal solution by gradient ascent of the target, and outputs the optimal action strategy. The actor-critic-based reinforcement learning method is a combination of the above two methods, and iteratively learns the optimal Q value and the optimal strategy at the same time. In summary, the actor-critic-based reinforcement learning method can better adapt to the uncertainty of the charging and discharging environment of energy storage batteries.

[0004] However, the actor-critic reinforcement learning method is highly sensitive to complex environments and is easily affected by environmental factors. In addition, the training process is unstable and difficult to converge, which does not conform to the actual task characteristics of energy storage batteries. Therefore, there is an urgent need for an energy storage battery charging and discharging strategy optimization solution that can adaptively learn from the environment and has high stability, and can interactively learn the optimal intelligent agent on a limited variety of time series data, thereby improving the accuracy of the action. Summary of the invention

[0005] The purpose of the present invention is to provide a battery charging and discharging optimization method, system and medium based on deep reinforcement learning, aiming to solve the above-mentioned problems.

[0006] The present invention is mainly achieved through the following technical solutions:

[0007] The battery charging and discharging optimization method based on deep reinforcement learning includes the following steps:

[0008] Step S1: Collect historical data related to the charging and discharging of energy storage batteries and obtain a state set S ={ s 1 , s 2 , …, s n} and charging and discharging action strategy set A ={ a 1 , a 2 , …, a n}, and form a training data set; s n = { E sn , E total , P sn , SOC , V sn},

[0009] in: E sn For energy storage batteries n The battery capacity collection at the moment;

[0010] E total is the total capacity of the energy storage battery;

[0011] P sn for n The battery power consumption at the moment;

[0012] SOC is the charge state set of the energy storage battery;

[0013] V sn for n The cost of battery loss at all times;

[0014] Step S2: constructing the target network model of the actor-critic, wherein the predicted actor network includes the current strategy prediction network and the auxiliary strategy prediction network, and the current strategy prediction network is composed of a diffusion model; then, the target network model is trained step by step based on the training data set;

[0015] Step S21: First, train the prediction actor network and the prediction critic network; the current strategy prediction network predicts the current state s t Action strategy under a t , the predictive critic network evaluates the action policy a t The policy gradient loss value is calculated based on the quality of the auxiliary policy prediction network and the target actor network; then, the contrastive learning loss value is calculated based on the output of the auxiliary policy prediction network and the target actor network; the loss value of the predicted actor network is the sum of the policy gradient loss value and the contrastive learning loss value;

[0016] Step S22: Then, the prediction critic network is trained; the value loss value is calculated by the output of the prediction critic network and the target critic network; the loss value of the prediction critic is the value loss value;

[0017] Step S23: updating the parameter weights of the current policy prediction network and the prediction critic network, and then soft-exchanging the parameter weights of the current policy prediction network and the prediction critic network with the target actor network and the target critic network respectively;

[0018] Step S24: repeat steps S21 to S23, iterating step by step until the algorithm converges;

[0019] Step S3: testing the trained target network model in a simulation environment and obtaining the optimal target network model;

[0020] Step S4: Input the current state into the optimal target network model to obtain the optimal battery charging and discharging action strategy.

[0021] In order to better implement the present invention, further, step S21 includes the following steps:

[0022] Step S211: The current state s t Input into the current strategy prediction network to get the action strategy in the current state a t ;

[0023] Step S212: Data group ( s t , a t ) is input into the prediction critic network to calculate the policy gradient loss value;

[0024] Step S213: Set the action strategy a t Input into the simulation environment to get the next state s t+1and reward for the decision-making process r t , get the array [ s t , a t , s t+1 , r t ] is stored in the resampled data pool;

[0025] Step S214: Set the action strategy a t Input into the auxiliary strategy prediction network to obtain the next moment predicted action strategy ; The next moment state s t+1 Input to the target actor's network to get the action strategy at the next moment a t+1 , and the contrastive learning loss value is calculated.

[0026] In order to better implement the present invention, further, in step S213, the reward of the decision-making process r t The calculation function is:

[0027] ,

[0028] ,

[0029] ,

[0030] in: α and β All are temperature coefficients, used to adjust rewards;

[0031] V st for t The cost of battery loss at all times;

[0032] r d Increased rewards for meeting customers' equipment power SOC requirements;

[0033] V total The production cost of the battery;

[0034] D is the discharge depth;

[0035] L ( D ) is the discharge depth D A function of the number of battery cycles under

[0036] E max is the maximum capacity of the battery;

[0037] P ev,t is the power consumption of the equipment at time t;

[0038] Δ t The time interval for changing the charging and discharging power of the electrical equipment;

[0039] SOC t for t The charge state at the moment;

[0040] for t The expected state of charge at the time;

[0041] ε is the set threshold.

[0042] In order to better implement the present invention, further, in step S22, the data group ( s t , a t ) is input into the prediction critic network to obtain Q t , the data set ( s t+1 , a t+1 ) is input into the target critic network to obtain Q t+1 , calculate the value loss L q for:

[0043] ,

[0044] ,

[0045] in: y t is the target value;

[0046] N is the batch size;

[0047] y t is the target value;

[0048] γ is the discount factor;

[0049] for the target critic network;

[0050] For predicting critic networks.

[0051] In order to better implement the present invention, further, the contrastive learning loss value L c for:

[0052] ,

[0053] Where: N is the batch size;

[0054] t is the time series index.

[0055] In order to better implement the present invention, further, the policy gradient loss value L g for:

[0056] ,

[0057] Where: N is the batch size;

[0058] To predict critic networks;

[0059] is the current policy prediction network;

[0060] The policy gradient loss value is maximized using the gradient ascent method.

[0061] The present invention is mainly achieved through the following technical solutions:

[0062] The battery charging and discharging optimization system based on deep reinforcement learning is based on the above-mentioned battery charging and discharging optimization method based on deep reinforcement learning, and includes a data processing module, a training module, a simulation test module and a real-time test module; the data processing module is used to collect historical data and obtain a state set S and charging and discharging action strategy set A， Then, a training data set is formed; the training module is used to build a target network model and use the training data set to train the target network model step by step; the simulation test module is used to test the trained target network model in a simulation environment and obtain the optimal target network model; the real-time test module is used to input the current state into the optimal target network model to obtain the best battery charging and discharging strategy.

[0063] In order to better realize the present invention, further, the target network model includes a predicted actor-home network, a predicted critic network, a target actor-home network and a target critic network, the predicted actor-home network includes a current strategy prediction network and an auxiliary strategy prediction network connected in sequence from front to back, the current strategy prediction network and the predicted critic network soft-exchange update parameters with the target actor-home network and the target critic network respectively; the current strategy prediction network has the same structure as the target actor-home network, and the predicted critic network has the same structure as the target critic network.

[0064] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned battery charging and discharging optimization method based on deep reinforcement learning.

[0065] The beneficial effects of the present invention are as follows:

[0066] (1) The battery charging and discharging strategy data of the present invention satisfies the Markov decision process, and forms a state set and a charging and discharging action strategy set based on historical data related to charging and discharging, thereby training the target network model. The present invention builds a target network model based on the actor-critic reinforcement learning framework, and the main network of the target network model includes a predicted actor network and a predicted critic network, wherein the predicted actor network includes a current strategy prediction network and an auxiliary strategy prediction network. The current strategy prediction network is composed of a diffusion model, which is used as an information extraction network, and the generation ability of the diffusion model is used to explore the optimal solution in the action strategy space. Moreover, the smoothness and stability of the diffusion model can improve the generalization of the model to the environment and reduce the sensitivity of the model, thereby enhancing the model's strategy generation ability and strategy exploration ability, and achieving the purpose of improving strategy optimization performance. Based on the output of the auxiliary strategy prediction and the target actor network, comparative learning is performed to assist in improving the prediction ability of the model and solve the problem of over-estimation of the action strategy.

[0067] (2) During operation, the present invention regards the entire target network model as an intelligent agent, and all influencing factors outside the intelligent agent are called the environment. When task scheduling comes, the intelligent agent gives the action strategy of charging and discharging, that is, the intelligent agent commands the scheduling system to control the environment to charge or discharge. The environment will give feedback to the intelligent agent, such as status and rewards. These rewards can be used to quantify the performance of the intelligent agent in the actual environment. The staff can analyze the shortcomings of the algorithm model in this quantitative way and make targeted improvements. Compared with the prior art, the present invention can better cope with the complex environmental changes in the charging and discharging of energy storage batteries, is more intelligent, can accurately give charging and discharging strategies, extend the service life of the battery, reduce the loss cost of the manufacturer, and has good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1It is a schematic diagram of the overall structure of the target network model of the present invention;

[0069] Figure 2 Schematic diagram of comparative learning between auxiliary policy prediction network and target actor network;

[0070] Figure 3 This is a flow chart of the battery charging and discharging optimization method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0071] Example 1

[0072] A battery charging and discharging optimization system based on deep reinforcement learning includes a data processing module, a training module, a simulation test module and a real-time test module; the data processing module is used to collect historical data and obtain a state set S and charging and discharging action strategy set A， Then, a training data set is formed; the training module is used to build a target network model and use the training data set to train the target network model step by step; the simulation test module is used to test the trained target network model in a simulation environment and obtain the optimal target network model; the real-time test module is used to input the current state into the optimal target network model to obtain the best battery charging and discharging strategy.

[0073] Preferably, if Figure 1 As shown in the figure, a target network model is built based on the DDPG algorithm to solve the instability in the training process and improve the convergence of the algorithm model. The target network model includes a predicted actor-home network, a target actor-home network, a predicted critic network and a target critic network, wherein the predicted actor-home network and the predicted critic network constitute the main network. The predicted critic network is used to guide the update of the predicted actor-home network and evaluate the pros and cons of the strategy generated by the predicted actor-home network, so as to achieve strategy optimization.

[0074] Preferably, the prediction actor network includes a current strategy prediction network and an auxiliary strategy prediction network connected in sequence from front to back. The current strategy prediction network and the prediction critic network are soft-swapped and updated with the target actor network and the target critic network respectively. Specifically, the current strategy prediction network is composed of a diffusion model, which is used as an information extraction network, and the generation ability of the diffusion model is used to explore the optimal solution in the action strategy space. Moreover, the smoothness and stability of the diffusion model can improve the generalization of the model to the environment and reduce the sensitivity of the model.

[0075] The auxiliary strategy prediction network obtains the next strategy based on the strategy predicted by the current strategy prediction network, and then compares and learns with the output of the target actor network to help improve the prediction ability of the model and solve the problem of over-estimation of action strategies. Specifically, the auxiliary strategy prediction network consists of a fully connected layer, a normalization layer, and an activation function layer, and is built using a residual structure.

[0076] Preferably, the target actor network has the same structure as the current strategy prediction network in the predicted actor network. The weight parameters of the network are not updated through training, but through soft switching. This fixed network only trains the other network, which can improve the convergence of the model and prevent gradient explosion or disappearance. The predicted critic network has the same structure as the target critic network, both of which are composed of a fully connected layer, a normalization layer, and an activation function layer, and are built using a residual structure.

[0077] It can be obtained from experiments that, compared with the prior art, the present invention can better cope with the complex environmental changes in which the energy storage battery is charged and discharged, is more intelligent, can accurately provide charging and discharging strategies, extend the service life of the battery, reduce the loss cost of the manufacturer, and has good practicality.

[0078] Embodiment 2:

[0079] A battery charging and discharging optimization method based on deep reinforcement learning is based on the actor-critic reinforcement learning framework, introducing an auxiliary policy prediction network of contrastive learning, and constraining the over-estimation problem of the current policy prediction network. Then, the smoothness and stability of the diffusion model are used to improve the generalization of the reinforcement learning algorithm to complex environments, and improve the quality of the policy output by the current policy prediction network, thereby enhancing the model's policy generation and policy exploration capabilities, and achieving the purpose of improving policy optimization performance. It includes the following steps:

[0080] Historical data of energy storage batteries are collected from multiple data sources such as energy storage battery management systems and dispatching systems, including time series data such as battery voltage, current, temperature, charge state, power, etc. Noise data such as outliers and missing values ​​are manually removed to ensure data integrity and accuracy.

[0081] According to the data characteristics and functions, the actor-critic reinforcement learning framework is built. First, the prediction actor network is built to predict the action strategy in the current state, and the prediction critic network is used to guide the learning. Then the prediction critic network is trained and optimized and updated using the temporal difference error method. Finally, the performance is tested in a simulation environment to select the best one.

[0082] The loss function of the model is divided into policy gradient loss function, value loss function and contrastive learning loss function;

[0083] After training the predicted actor network and the predicted critic network, the parameter weights of the two networks are updated. Finally, the current strategy prediction network and the predicted critic network soft-swap the parameter weights with the target actor network and the target critic network respectively. The entire training process is iterated multiple times until the algorithm converges.

[0084] Preferably, since the optimization problem of the charging and discharging strategy of the energy storage battery conforms to the characteristics of the Markov decision process, the problem is modeled as a Markov model with a continuous action space. Suppose the state set is S ={ s 1 , s 2 , …, s n}, the set of strategic actions that the agent can take is A ={ a 1 , a 2 , …, a n},in s n = { E sn , E total , P sn , SOC , V sn},in: E sn For energy storage batteries n The battery capacity collection at the moment;

[0085] E total is the total capacity of the energy storage battery;

[0086] P sn for n The battery power consumption at the moment;

[0087] SOC is the charge state set of the energy storage battery;

[0088] V sn for n The battery loss cost at all times.

[0089] Rewards for this decision-making process r t There are two main considerations: one is to reduce the loss cost of charging and discharging batteries, and the other is the SOC requirements of the customer's equipment power consumption. Therefore, the reward functionr for:

[0090] ,

[0091] ,

[0092] ,

[0093] in: α and β All are temperature coefficients, used to adjust rewards;

[0094] V st for t The cost of battery loss at all times;

[0095] r d Increased rewards for meeting customers' equipment power SOC requirements;

[0096] V total The production cost of the battery;

[0097] L ( D ) is the discharge depth D The battery cycle number function under

[0098] E max is the maximum capacity of the battery;

[0099] P ev,t is the power consumption of the equipment at time t;

[0100] Δ t The time interval for changing the charging and discharging power of the electrical equipment;

[0101] SOC t for t The charge state at the moment;

[0102] for t The expected state of charge at the time;

[0103] ε is the set threshold.

[0104] Preferably, if Figure 3 As shown, the present invention specifically comprises the following steps:

[0105] (1) First, initialize the parameters of the target network model, that is, initialize the parameters of the predicted actor network and the predicted critic network as well as the target actor network and the target critic network.

[0106] (2) Training the predictive actor network and the predictive critic network;

[0107] Get the current state interactively from the environment s t , and then input into the current strategy prediction network to get a t , and then the data set ( s t , a t ) is input into the prediction critic network to obtain Q t , calculate the policy gradient loss value L g :

[0108] ,

[0109] in the formula To predict the critic network, For the current policy prediction network, the loss value is optimized by using the gradient ascent method to maximize the loss.

[0110] like Figure 2 As shown, the action strategy a t Input into the simulation environment to explore the next state s t+1 and rewards r t , and obtain the data set [ s t , a t , s t+1 , r t ], stored in the resampled data pool, and the action strategy a t Input into the auxiliary strategy prediction network to obtain the next moment prediction action strategy , and then the status s t+1 Input to the target actor network to predict the action strategy a t+1 , and finally calculate the contrastive learning loss value L c :

[0111] ,

[0112] Where: N is the batch size; t is the time series index.

[0113] (3) Then, freeze the parameter weights of the target actor network and the target critic network, and only train the predicted critic network;

[0114] If the number of data in the resampled data pool is ≥ N, then sample from the resampled data pool and group the data ( s t , a t ) is input into the predictive critic network to obtain Q t , and then the data set ( s t+1 , a t+1 ) is input into the target critic network to obtain Q t+1 , and finally calculate the value loss L q :

[0115] ,

[0116] ,

[0117] The above formula uses the time difference method for estimation. y t is the target value, γ is the discount factor, which is used to control the size of the target value. For the target critic network, For predicting critic networks.

[0118] (4) Each time the main network is updated, a parameter weight update is required. The main network includes a prediction actor network and a prediction critic network. The current strategy prediction network and the prediction critic network soft-swap the parameter weights with the target actor network and the target critic network respectively.

[0119] (5) The entire training process is iterated multiple times until the algorithm converges. Determine whether the algorithm converges. If not, proceed to step (2), otherwise proceed to step (6).

[0120] (6) Perform a simulation environment test. If the target network model is optimal, perform a real environment test. Otherwise, proceed to step (2).

[0121] After testing the performance of the target network model in a simulation environment, the present invention selects the optimal target network model and puts it into the actual scene for operation. All model weight parameters are fixed and not updated. The simulation environment is a virtual environment for charging and discharging energy storage batteries. During operation, the entire target network model is used as an intelligent agent, and all influencing factors outside the intelligent agent are called the environment. When task scheduling comes, the intelligent agent gives the action strategy for charging and discharging, that is, the intelligent agent commands the scheduling system to control the environment for charging or discharging. The environment will give feedback to the intelligent agent, such as status and rewards. These rewards can be used to quantify the performance of the intelligent agent in the actual environment. The staff can analyze the deficiencies of the algorithm model in this quantitative way and make targeted improvements.

[0122] The present invention solves the problems of overestimation and low generalization by analyzing the task characteristics and based on the actor-critic reinforcement learning framework. Experiments show that compared with the prior art, the present invention can better cope with the complex environmental changes in which the energy storage battery is charged and discharged, is more intelligent, can accurately provide charging and discharging strategies, extend the service life of the battery, reduce the loss cost of the manufacturer, and has good practicality.

[0123] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A battery charging and discharging optimization method based on deep reinforcement learning, characterized in that: The following steps are involved: Step S1: Collect historical data related to the charging and discharging of energy storage batteries and obtain a state set S ={ s 1, s 2, ..., s n } and charging and discharging action strategy set , and form a training data set; s n = { E sn , E total , P sn , SOC , V sn }, in: E sn For energy storage batteries n The battery capacity collection at the moment; E total is the total capacity of the energy storage battery; P sn for n The battery power consumption at the moment; SOC is the charge state set of the energy storage battery; V sn for n The cost of battery loss at any time; Step S2: constructing the target network model of the actor-critic, wherein the predicted actor network includes the current strategy prediction network and the auxiliary strategy prediction network, and the current strategy prediction network is composed of a diffusion model; then, the target network model is trained step by step based on the training data set; Step S21: First, train the prediction actor network and the prediction critic network; the current strategy prediction network predicts the current state s t Action strategy under , the predictive critic network evaluates the action policy The policy gradient loss value is calculated based on the quality of the auxiliary policy prediction network and the target actor network; then, the contrastive learning loss value is calculated based on the output of the auxiliary policy prediction network and the target actor network; the loss value of the predicted actor network is the sum of the policy gradient loss value and the contrastive learning loss value; Step S22: Then, the prediction critic network is trained; the value loss value is calculated by the output of the prediction critic network and the target critic network; the loss value of the prediction critic is the value loss value; Step S23: updating the parameter weights of the current strategy prediction network and the prediction critic network, and then soft-exchanging the parameter weights of the current strategy prediction network and the prediction critic network with the target actor network and the target critic network respectively; Step S24: repeat steps S21 to S23, iterating step by step until the algorithm converges; Step S3: testing the trained target network model in a simulation environment and obtaining the optimal target network model; Step S4: input the current state into the optimal target network model to obtain the optimal battery charging and discharging action strategy; In step S22, the data group Input to the prediction critic network to get Q t , the data group Input to the target critic network to get Q t+1 , calculate the value loss L q for: ， ， in: y t is the target value; N is the batch size; y t is the target value; γ is the discount factor; for the target critic network; To predict critic networks; The contrastive learning loss value L c for: ， Where: N is the batch size; t is the time series index; The policy gradient loss value L g for: ， Where: N is the batch size; To predict critic networks; is the current policy prediction network; The policy gradient loss value is maximized using the gradient ascent method.

2. The battery charging and discharging optimization method based on deep reinforcement learning according to claim 1, characterized in that: The step S21 comprises the following steps: Step S211: The current state s t Input into the current strategy prediction network to get the action strategy in the current state ; Step S212: Group the data Input into the prediction critic network and calculate the policy gradient loss value; Step S213: Set the action strategy Input into the simulation environment to get the next state s t+1 and reward for the decision-making process r t , get the array Store in the resampled data pool; Step S214: Set the action strategy Input into the auxiliary strategy prediction network to obtain the next moment predicted action strategy ; The next moment state s t+1 Input to the target actor's network to get the action strategy at the next moment , and the contrastive learning loss value is calculated.

3. The battery charging and discharging optimization method based on deep reinforcement learning according to claim 2, characterized in that: In step S213, the reward of the decision-making process r t The calculation function is: ， ， ， in: α and β All are temperature coefficients, used to adjust rewards; V st for t The cost of battery loss at any time; r d Increased rewards for meeting customers' equipment power SOC requirements; V total The production cost of the battery; D is the discharge depth; L ( D ) is the discharge depth D A function of the number of battery cycles under E max is the maximum capacity of the battery; P ev,t is the power consumption of the equipment at time t; Δ t The time interval for changing the charging and discharging power of the electrical equipment; SOC t for t The charge state at the moment; for t The expected state of charge at the time; ε is the set threshold.

4. A battery charging and discharging optimization system based on deep reinforcement learning, which is based on the battery charging and discharging optimization method based on deep reinforcement learning according to any one of claims 1 to 3, characterized in that: It includes a data processing module, a training module, a simulation test module and a real-time test module; the data processing module is used to collect historical data and obtain a state set S and charging and discharging action strategy set A， Then, a training data set is formed; the training module is used to build a target network model and use the training data set to train the target network model step by step; the simulation test module is used to test the trained target network model in a simulation environment and obtain the optimal target network model; the real-time test module is used to input the current state into the optimal target network model to obtain the best battery charging and discharging strategy.

5. The battery charging and discharging optimization system based on deep reinforcement learning according to claim 4 is characterized in that: The target network model includes a predicted actor-family network, a predicted critic network, a target actor-family network and a target critic network. The predicted actor-family network includes a current strategy prediction network and an auxiliary strategy prediction network connected in sequence from front to back. The current strategy prediction network and the predicted critic network soft-swap and update parameters with the target actor-family network and the target critic network respectively. The current strategy prediction network has the same structure as the target actor-family network, and the predicted critic network has the same structure as the target critic network.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the battery charging and discharging optimization method based on deep reinforcement learning described in any one of claims 1 to 3 is implemented.

Images