Dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning

Abstract

The invention relates to the technical field of capacity balancing of a dynamic reconfigurable battery energy storage system, in particular to a dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning. The method comprises the following steps: acquiring the charge state of at least one battery monomer and / or at least one battery module of the dynamic reconfigurable battery network in the charging and discharging process; inputting the charge state into a pre-trained deep reinforcement learning model to obtain a control instruction of the dynamic reconfigurable battery network; and in response to the control instruction, reconstructing the dynamic reconfigurable battery network, so that the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets a preset capacity balancing condition. Therefore, the problems that in the related technology, the dynamic reconfigurable battery network is a highly complex nonlinear time-varying dynamic system, the complexity of the dynamic reconfigurable battery network is greatly increased along with the increase of the number of the single batteries, and consequently the dynamic reconfigurable battery network cannot be accurately controlled, and global optimization is difficult to achieve are solved.

Description

Technical Field

[0001] The present application relates to the technical field of capacity balancing of a dynamically reconfigurable battery energy storage system, and in particular to a method for capacity balancing of a dynamically reconfigurable battery network based on deep reinforcement learning. Background Art

[0002] It is impossible to achieve absolutely ideal battery production conditions, and accurate battery sorting is difficult to achieve. Therefore, even battery cells in the same module will exhibit different electrical characteristics, such as differences in internal impedance and aging behavior. The differences in battery cells will lead to unbalanced charging and discharging, which will lead to overcharging and over-discharging, increasing the risk of system failure. In addition, in a system with uneven battery capacity, the system's charging and discharging capabilities depend on the weakest battery cell, and there is an obvious "short board effect", which will greatly limit the energy utilization of the battery network.

[0003] In the related art, in order to overcome the problem of battery inconsistency, some balancing strategies, such as active balancing strategy and passive balancing strategy, can be adopted to solve the problem. However, whether it is an active balancing strategy or a passive balancing strategy, there will be problems such as low energy efficiency, increased cost or equipment volume. Therefore, it can be solved by a dynamically reconfigurable battery network, in which the capacity balancing of the dynamically reconfigurable battery network is generally based on the empirical method, that is, when charging, select several battery cells / modules with the smallest capacity for charging, or when discharging, select several battery cells / modules with the largest capacity for discharging.

[0004] However, in the related technology, the dynamically reconfigurable battery network is a highly complex nonlinear time-varying dynamic system. As the number of battery cells increases, its complexity will greatly increase, resulting in the inability to accurately control the dynamically reconfigurable battery network and difficulty in achieving global optimality, which urgently needs improvement. Summary of the invention

[0005] The present application provides a method for capacity balancing of a dynamically reconfigurable battery network based on deep reinforcement learning to solve the problem in the related art that the dynamically reconfigurable battery network is a highly complex nonlinear time-varying dynamic system. As the number of battery cells increases, its complexity will greatly increase, resulting in the inability to accurately control the dynamically reconfigurable battery network and difficulty in achieving global optimization.

[0006] The first aspect of the present application provides a method for capacity balancing of a dynamically reconfigurable battery network based on deep reinforcement learning, comprising the following steps: obtaining the state of charge of at least one battery cell and / or at least one battery module of a dynamically reconfigurable battery network during the charging and discharging process; inputting the state of charge into a pre-trained deep reinforcement learning model to obtain control instructions for the dynamically reconfigurable battery network; and reconstructing the dynamically reconfigurable battery network in response to the control instructions so that the capacity balancing capability of the reconstructed dynamically reconfigurable battery network meets preset capacity balancing conditions.

[0007] Optionally, in one embodiment of the present application, before inputting the state of charge into a pre-trained deep reinforcement learning model, it also includes: constructing an initial deep reinforcement learning model based on DQN (Deep Q-Network); obtaining the state quantity, action quantity and / or reward function of the initial deep reinforcement learning model; training the initial deep reinforcement learning model based on the state quantity, action quantity and / or reward function to obtain the pre-trained deep reinforcement learning model.

[0008] Optionally, in one embodiment of the present application, the expression of the profit function may be, but is not limited to,:

[0009]

[0010] Among them, σ is used to characterize the imbalance degree of single-cell SOC (State of Charge).

[0011] Optionally, in one embodiment of the present application, reconstructing the dynamic reconfigurable battery network in response to the control instruction includes: in response to the control instruction, verifying the reconstructed dynamic reconfigurable battery network based on the charge state to obtain a verification result of the reconstructed dynamic reconfigurable battery network; judging whether the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets a preset capacity balancing condition based on the verification result; if the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet the preset capacity balancing condition, retraining the deep reinforcement learning model until the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

[0012] The second aspect of the present application provides a dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning, including: an acquisition module, used to obtain the charge state of at least one battery cell and / or at least one battery module of the dynamic reconfigurable battery network during the charging and discharging process; a generation module, used to input the charge state into a pre-trained deep reinforcement learning model to obtain a control instruction of the dynamic reconfigurable battery network; a reconstruction module, used to reconstruct the dynamic reconfigurable battery network in response to the control instruction, so that the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

[0013] Optionally, in one embodiment of the present application, it also includes: a construction module, used to construct an initial deep reinforcement learning model based on DQN before inputting the charge state into a pre-trained deep reinforcement learning model; a setting module, used to obtain the state quantity, action quantity and / or benefit function of the initial deep reinforcement learning model; a training module, used to train the initial deep reinforcement learning model based on the state quantity, action quantity and / or benefit function to obtain the pre-trained deep reinforcement learning model.

[0014] Optionally, in one embodiment of the present application, the expression of the profit function may be, but is not limited to,:

[0015]

[0016] Among them, σ is used to characterize the imbalance degree of monomer SOC.

[0017] Optionally, in one embodiment of the present application, the reconstruction module includes: a verification unit, used to verify the reconstructed dynamic reconfigurable battery network based on the charge state in response to the control instruction, and obtain a verification result of the reconstructed dynamic reconfigurable battery network; a judgment unit, used to judge whether the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition based on the verification result; and a retraining unit, used to retrain the deep reinforcement learning model when the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet the preset capacity balancing condition, until the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning as described in the above embodiment.

[0019] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning.

[0020] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning.

[0021] The embodiment of the present application can input the charge state of at least one battery cell and / or at least one battery module of the dynamically reconfigurable battery network during the charging and discharging process into a pre-trained deep reinforcement learning model to obtain control instructions, and reconstruct the dynamically reconfigurable battery network based on the control instructions, so that the capacity balancing capability of the reconstructed dynamically reconfigurable battery network meets certain capacity balancing conditions, realizes the capacity balancing of the dynamically reconfigurable battery network, and reduces the complexity of the dynamically reconfigurable battery network. Thus, the problem in the related art that the dynamically reconfigurable battery network is a highly complex nonlinear time-varying dynamic system, and its complexity will greatly increase with the increase in the number of battery cells, resulting in the inability to accurately control the dynamically reconfigurable battery network and difficulty in achieving global optimization is solved.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning provided according to an embodiment of the present application;

[0025] Figure 2 A flowchart of the working principle of reinforcement learning provided according to an embodiment of the present application;

[0026] Figure 3 A block diagram of a deep learning network according to an embodiment of the present application;

[0027] Figure 4 A schematic block diagram of a network architecture of deep Q learning provided according to an embodiment of the present application;

[0028] Figure 5 A schematic block diagram of a profit function provided according to an embodiment of the present application;

[0029] Figure 6 A block diagram of a dynamically reconfigurable battery network based on an initial deep reinforcement learning model built in Simulink according to an embodiment of the present application;

[0030] Figure 7 A schematic block diagram of an initial deep reinforcement learning model training process provided according to an embodiment of the present application;

[0031] Figure 8 A block diagram of a simulation result after training provided according to an embodiment of the present application;

[0032] Fig. 9 A flowchart of the working principle of a dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning provided according to an embodiment of the present application;

[0033] Fig.10 A block diagram of a dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning provided according to an embodiment of the present application;

[0034] Fig.11 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0036] The following describes the method for capacity balancing of a dynamic reconfigurable battery network based on deep reinforcement learning in an embodiment of the present application with reference to the accompanying drawings. In view of the fact that the dynamic reconfigurable battery network mentioned in the above background technology is a highly complex nonlinear time-varying dynamic system, and its complexity will greatly increase with the increase in the number of battery cells, resulting in the problem that the dynamic reconfigurable battery network cannot be accurately controlled and it is difficult to achieve the global optimum, the present application provides a method for capacity balancing of a dynamic reconfigurable battery network based on deep reinforcement learning, in which the state of charge of at least one battery cell and / or at least one battery module of the acquired dynamic reconfigurable battery network during the charging and discharging process can be input into a pre-trained deep reinforcement learning model to obtain a control instruction, and the dynamic reconfigurable battery network is reconstructed based on the control instruction, so that the capacity balancing ability of the reconstructed dynamic reconfigurable battery network meets certain capacity balancing conditions, and the capacity balancing of the dynamic reconfigurable battery network is achieved, and the complexity of the dynamic reconfigurable battery network is reduced. Thus, the problem that the dynamic reconfigurable battery network is a highly complex nonlinear time-varying dynamic system in the related technology, and its complexity will greatly increase with the increase in the number of battery cells, resulting in the problem that the dynamic reconfigurable battery network cannot be accurately controlled and it is difficult to achieve the global optimum is solved.

[0037] Specifically, Figure 1 This is a flowchart of a dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning provided according to an embodiment of the present application.

[0038] like Figure 1 As shown, the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning includes the following steps:

[0039] In step S101, the state of charge of at least one battery cell and / or at least one battery module of a dynamically reconfigurable battery network during a charging and discharging process is obtained.

[0040] It can be understood that the embodiments of the present application can use sensors to detect the charge state of at least one battery cell in real time, and can also use sensors to detect the charge state of at least one battery module in real time. Sensors can also be used to detect the charge state of at least one battery cell and at least one battery module in real time. The acquisition of the charge state can be set by technicians in this field according to actual conditions, and this application does not impose specific restrictions.

[0041] In some embodiments, the embodiments of the present application can use sensors to detect the charge state of at least one battery cell in real time during the charging and discharging process of the dynamically reconfigurable battery network.

[0042] In some embodiments, the embodiments of the present application can use sensors to detect the charge state of at least one battery module in real time during the charging and discharging process of the dynamically reconfigurable battery network.

[0043] In some embodiments, the embodiments of the present application can use sensors to detect the charge state of at least one battery cell and at least one battery module in real time during the charging and discharging process of the dynamically reconfigurable battery network.

[0044] Optionally, in one embodiment of the present application, before inputting the state of charge into the pre-trained deep reinforcement learning model, it also includes: constructing an initial deep reinforcement learning model based on DQN; obtaining the state quantity, action quantity and / or benefit function of the initial deep reinforcement learning model; training the initial deep reinforcement learning model based on the state quantity, action quantity and / or benefit function to obtain the pre-trained deep reinforcement learning model. The expression of the benefit function can be, but is not limited to,:

[0045]

[0046] Among them, σ is used to characterize the imbalance degree of monomer SOC.

[0047] It can be understood that after obtaining the state of charge of the battery, the embodiment of the present application can input it into a pre-trained deep reinforcement learning model.

[0048] Among them, in the embodiments of the present application, deep reinforcement learning (DQN) is an artificial intelligence optimization algorithm that integrates reinforcement learning (RL) and deep learning (DL).

[0049] Among them, the reinforcement learning in the embodiment of the present application is a kind of machine learning (ML), and its principle diagram is as follows Figure 2 As shown in the figure, reinforcement learning consists of an environment and an agent. The agent obtains observations from the environment, generates corresponding actions based on the policy, and generates rewards after acting on the environment. Through training, the policy in the agent is improved.

[0050] In addition, the embodiment of the present application also provides a common solution algorithm for reinforcement learning, and its algorithm classification diagram is shown in Table 1. In general, the algorithm for solving reinforcement learning in the embodiment of the present application is divided into two types: actor and critic. Among them, the main idea of ​​the actor algorithm is strategy search, that is, the input is the observation, and the output is the probability of each action. According to the discrete and continuous situations, it is divided into two categories: stochastic algorithm and deterministic algorithm; and the main idea of ​​the critic algorithm is to make decisions based on the value function, that is, the input is the observation and all possible actions, and the output is the benefit value of each action taken under the corresponding observation, and the strategy is determined according to the benefit; in addition, the embodiment of the present application can also combine the actor algorithm and the critic algorithm, and derive many actor-critic algorithms, such as PPO (Proximal Policy Optimization), DDPG (Deep Deterministic Policy Gradient), SARSA (State-Action-Reward-State-Action), Policy Gradient, TRPO (Trust Region Policy Gradient), etc. Optimization, trust region strategy optimization), SAC (Soft Actor-Critic, reinforcement learning algorithm based on maximum entropy), TD3 (Twin Delayed Deep Deterministic Policy Gradient, double delayed deep deterministic policy gradient), etc. Among them, Table 1 is a classification table of reinforcement learning solution algorithms provided according to an embodiment of the present application.

[0051] Table 1

[0052]

[0053] Deep learning can essentially be seen as building a functional relationship between input and output. A deep neural network can be composed of an input layer, an output layer, and a hidden layer, such as Figure 3 As shown in the figure, the hidden layer consists of a fully connected layer and an activation function. Each neuron has two adjustable parameters: weight and bias. The activation function is used to increase nonlinear characteristics. Commonly used functions include relu and tanh. Since deep learning networks are usually composed of a large number of neural networks, they can fit complex nonlinear functions more efficiently than single-layer neural networks.

[0054] DQN is a type of deep reinforcement learning that combines Q-learning in reinforcement learning with deep neural networks. The role of deep learning is to construct a function to calculate the Q-table, taking the state quantity and all action quantities extracted from the environment as the input of the function, and the Q value corresponding to each action in each state as the output of the function, such as Figure 4 As shown in the figure, the agent makes decisions in real time based on the function. Of course, the parameters of the neural network in the initial agent are not trained, so the decision made may not be the best. Through the training algorithm in the agent, the parameters in the neural network are optimized in continuous training. After training, the agent can generate a reasonable Q-table to make decisions.

[0055] Furthermore, the embodiments of the present application can construct an initial deep reinforcement learning model based on DQN, and define the state quantity, action quantity and benefit function of the initial deep reinforcement learning model, which directly determines the quality of the initial deep reinforcement learning model training. For example, the embodiments of the present application can take a 1×2 reconfigurable battery network as an example to analyze how to use the initial deep reinforcement learning model for control in a dynamic reconfigurable battery network.

[0056] It can be understood that the purpose of the control of the embodiment of the present application is to achieve the balance of the charge state of each monomer. Therefore, the state quantity of the embodiment of the present application can be selected as the charge state of each monomer. Further, the reconfigurable battery network of the embodiment of the present application controls the on and off of each monomer through a switch, so the action quantity is selected as the switch state, 1 represents on, and 0 represents off; for the selection of the benefit function, since the goal is to make the charge state of the monomer as balanced as possible, the smaller the balance, the greater the benefit. Therefore, the expression of the benefit function determined in the embodiment of the present application can be, but is not limited to, as follows:

[0057]

[0058] Among them, σ is used to characterize the imbalance degree of monomer SOC, σ = std(SOC1, OC2).

[0059] Further, in the embodiment of the present application, the meaning of the benefit function can be that when the standard deviation of SOC is less than 0.001, it is considered that the two monomers have reached equilibrium, and the benefit is relatively large at this time; otherwise, the benefit function is negative, and the greater the degree of imbalance, the smaller the benefit. The benefit function is plotted as a curve, such as Figure 5 shown.

[0060] In step S102, the state of charge is input into a pre-trained deep reinforcement learning model to obtain control instructions for a dynamically reconfigurable battery network.

[0061] As a possible implementation method, the embodiment of the present application can input the state of charge into a pre-trained deep reinforcement learning model, formulate a balancing control strategy, and output control instructions for a dynamically reconfigurable battery network.

[0062] In step S103, in response to the control instruction, the dynamically reconfigurable battery network is reconfigured so that the capacity balancing capability of the reconfigured dynamically reconfigurable battery network meets a preset capacity balancing condition.

[0063] It can be understood by those skilled in the art that the dynamically reconfigurable battery network in the embodiment of the present application can reconfigure the network according to the control instruction to achieve capacity balancing, so that the capacity balancing capability of the reconfigured dynamically reconfigurable battery network meets certain capacity balancing conditions. Among them, the certain capacity balancing conditions can be set by those skilled in the art according to actual conditions, and this application does not impose specific restrictions.

[0064] Optionally, in one embodiment of the present application, in response to a control instruction, a dynamically reconfigurable battery network is reconstructed, including: in response to the control instruction, verifying the reconstructed dynamically reconfigurable battery network based on the state of charge to obtain a verification result of the reconstructed dynamically reconfigurable battery network; based on the verification result, determining whether the capacity balancing capability of the reconstructed dynamically reconfigurable battery network meets a preset capacity balancing condition; if the capacity balancing capability of the reconstructed dynamically reconfigurable battery network does not meet the preset capacity balancing condition, retraining the deep reinforcement learning model until the capacity balancing capability of the reconstructed dynamically reconfigurable battery network meets the preset capacity balancing condition.

[0065] It is understandable that the embodiments of the present application can be simulated and verified by Matlab or Simulink. For example, the embodiments of the present application can build a dynamically reconfigurable battery network based on the initial deep reinforcement learning model in Simulink, which can be as follows: Figure 6 As shown. Figure 6 It can be seen that the dynamic reconfigurable battery network constructed in the embodiment of the present application is divided into a main circuit part and a control part. Among them, the main circuit part of the embodiment of the present application is a reconfigurable battery network composed of two battery cells in parallel, and the main body of the control part is a DQN intelligent agent, which takes the defined state quantity and benefit function as input and the action quantity as output. During the simulation operation, the intelligent agent can make decisions in real time according to the state of the environment.

[0066] In addition, before using the intelligent agent for control, the embodiment of the present application should firstly train the intelligent agent. The training process is as follows: Figure 7 The sign of the end of a training cycle is that the terminal voltage of a certain monomer reaches the discharge cut-off voltage. Figure 7It can be seen that with the increase in the number of training times, the revenue of each training cycle generally shows an upward trend, the average revenue curve also rises steadily, and the revenue in the first few cycles increases rapidly, and then tends to saturation, indicating that the initial deep reinforcement learning model has been continuously improved during the training process.

[0067] Furthermore, the embodiment of the present application can use the trained deep reinforcement learning model to control the dynamic reconfigurable battery network, and the simulation results are as follows: Figure 8 As shown. Figure 8 It can be seen that in the embodiment of the present application, although the initial SOCs of the two monomers are different (the initial SOCs are 1 and 0.8 respectively), after being controlled by DQN, the SOCs of the monomers reach equilibrium at 166s, and remain highly balanced until the end of discharge, verifying the capacity balancing capability of DQN for the dynamically reconfigurable battery network, so that the capacity balancing capability of the dynamically reconfigurable battery network meets certain capacity balancing conditions. Among them, certain capacity balancing conditions can be set by technicians in this field according to actual conditions, and this application does not make specific restrictions.

[0068] As a possible implementation method, the embodiment of the present application can verify the reconstructed dynamic reconfigurable battery network, and based on the verification result, determine whether the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets certain capacity balancing conditions, and when the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet certain capacity balancing conditions, retrain the deep reinforcement learning model until the certain capacity balancing conditions are met.

[0069] The working principle of the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning proposed in the embodiment of the present application is introduced in detail below in conjunction with a specific embodiment.

[0070] in, Fig. 9 The present invention is a flowchart of the working principle of a dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning provided according to an embodiment of the present application.

[0071] Step S901: obtaining the state of charge of at least one battery cell and / or at least one battery module of a dynamically reconfigurable battery network during a charging and discharging process.

[0072] Step S902: Based on DQN, construct an initial deep reinforcement learning model.

[0073] Step S903: Train the initial deep reinforcement learning model to obtain a pre-trained deep reinforcement learning model.

[0074] Step S904: Input the state of charge into a pre-trained deep reinforcement learning model to generate control instructions to reconstruct the dynamically reconfigurable battery network.

[0075] According to the method for capacity balancing of a dynamically reconfigurable battery network based on deep reinforcement learning proposed in the embodiment of the present application, the state of charge of at least one battery cell and / or at least one battery module of the dynamically reconfigurable battery network during the charging and discharging process can be input into a pre-trained deep reinforcement learning model to obtain control instructions, and the dynamically reconfigurable battery network can be reconstructed based on the control instructions, so that the capacity balancing capability of the reconstructed dynamically reconfigurable battery network meets certain capacity balancing conditions, and the capacity balancing of the dynamically reconfigurable battery network is achieved, and the complexity of the dynamically reconfigurable battery network is reduced. Thus, the problem in the related art that the dynamically reconfigurable battery network is a highly complex nonlinear time-varying dynamic system, and its complexity will greatly increase with the increase in the number of battery cells, resulting in the inability to accurately control the dynamically reconfigurable battery network and difficulty in achieving global optimization is solved.

[0076] Next, a dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0077] Fig.10 A block diagram of a dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning provided according to an embodiment of the present application.

[0078] like Fig.10 As shown, the dynamic reconfigurable battery network capacity balancing device 10 based on deep reinforcement learning includes: an acquisition module 100, a generation module 200 and a reconstruction module 300.

[0079] The acquisition module 100 is used to acquire the state of charge of at least one battery cell and / or at least one battery module of the dynamic reconfigurable battery network during the charging and discharging process;

[0080] A generation module 200 for inputting the state of charge into a pre-trained deep reinforcement learning model to obtain control instructions for a dynamically reconfigurable battery network;

[0081] The reconfiguration module 300 is used to reconfigure the dynamic reconfigurable battery network in response to the control instruction, so that the capacity balancing capability of the reconfigured dynamic reconfigurable battery network meets the preset capacity balancing condition.

[0082] Optionally, in one embodiment of the present application, it also includes: a construction module, a setting module and a training module.

[0083] Among them, the construction module is used to build an initial deep reinforcement learning model based on DQN before inputting the charge state into the pre-trained deep reinforcement learning model.

[0084] A setup module for obtaining the state, action, and / or reward function of an initial deep reinforcement learning model.

[0085] The training module is used to train the initial deep reinforcement learning model based on the state quantity, action quantity and / or benefit function to obtain a pre-trained deep reinforcement learning model.

[0086] Optionally, in one embodiment of the present application, the expression of the profit function may be, but is not limited to,:

[0087]

[0088] Among them, σ is used to characterize the imbalance degree of monomer SOC.

[0089] Optionally, in one embodiment of the present application, the reconstruction module 300 includes: a verification unit, a judgment unit and a retraining unit.

[0090] The verification unit is used to verify the reconstructed dynamic reconfigurable battery network based on the state of charge in response to the control instruction, and obtain the verification result of the reconstructed dynamic reconfigurable battery network.

[0091] The judging unit is used to judge whether the capacity balancing capability of the reconfigured dynamic reconfigurable battery network meets the preset capacity balancing condition according to the verification result.

[0092] A retraining unit is used to retrain the deep reinforcement learning model when the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet the preset capacity balancing condition, until the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

[0093] It should be noted that the aforementioned explanation of the embodiment of the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning is also applicable to the dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning in this embodiment, and will not be repeated here.

[0094] According to the dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning proposed in the embodiment of the present application, the state of charge of at least one battery cell and / or at least one battery module of the dynamic reconfigurable battery network during the charging and discharging process can be input into a pre-trained deep reinforcement learning model to obtain control instructions, and the dynamic reconfigurable battery network can be reconstructed based on the control instructions, so that the capacity balancing ability of the reconstructed dynamic reconfigurable battery network meets certain capacity balancing conditions, and the capacity balancing of the dynamic reconfigurable battery network is achieved, and the complexity of the dynamic reconfigurable battery network is reduced. In this way, the problem in the related art that the dynamic reconfigurable battery network is a highly complex nonlinear time-varying dynamic system, and its complexity will greatly increase with the increase in the number of battery cells, resulting in the inability to accurately control the dynamic reconfigurable battery network and difficulty in achieving global optimization is solved.

[0095] Fig.11 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0096] A memory 1101 , a processor 1102 , and a computer program stored in the memory 1101 and executable on the processor 1102 .

[0097] When the processor 1102 executes the program, the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning provided in the above embodiment is implemented.

[0098] Furthermore, the electronic device further comprises:

[0099] The communication interface 1103 is used for communication between the memory 1101 and the processor 1102 .

[0100] The memory 1101 is used to store computer programs that can be executed on the processor 1102 .

[0101] The memory 1101 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0102] If the memory 1101, the processor 1102 and the communication interface 1103 are implemented independently, the communication interface 1103, the memory 1101 and the processor 1102 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 1101, the processor 1102 and the communication interface 1103 are integrated on a chip, the memory 1101, the processor 1102 and the communication interface 1103 can communicate with each other through an internal interface.

[0104] The processor 1102 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0105] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning.

[0106] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0111] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0112] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning, characterized in that: The following steps are involved: Obtaining a state of charge of at least one battery cell and / or at least one battery module of a dynamically reconfigurable battery network during a charging and discharging process; Inputting the state of charge into a pre-trained deep reinforcement learning model to obtain control instructions for the dynamically reconfigurable battery network; In response to the control instruction, the dynamically reconfigurable battery network is reconfigured so that a capacity balancing capability of the reconfigured dynamically reconfigurable battery network meets a preset capacity balancing condition.

2. The method for dynamically reconfigurable battery network capacity balancing based on deep reinforcement learning according to claim 1 is characterized in that: Before inputting the state of charge into a pre-trained deep reinforcement learning model, the method further includes: Based on the deep Q-network DQN, build the initial deep reinforcement learning model; Obtaining the state quantity, action quantity and / or benefit function of the initial deep reinforcement learning model; The initial deep reinforcement learning model is trained based on the state quantity, action quantity and / or benefit function to obtain the pre-trained deep reinforcement learning model.

3. The method for dynamically reconfigurable battery network capacity balancing based on deep reinforcement learning according to claim 2 is characterized in that: The expression of the profit function is: Among them, σ is used to characterize the imbalance degree of the single cell state of charge SOC.

4. The method for dynamically reconfigurable battery network capacity balancing based on deep reinforcement learning according to claim 1 is characterized in that: The step of reconfiguring the dynamically reconfigurable battery network in response to the control instruction includes: In response to the control instruction, verifying the reconstructed dynamic reconfigurable battery network based on the state of charge to obtain a verification result of the reconstructed dynamic reconfigurable battery network; According to the verification result, determining whether the capacity balancing capability of the reconfigured dynamic reconfigurable battery network meets a preset capacity balancing condition; If the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet the preset capacity balancing condition, the deep reinforcement learning model is retrained until the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

5. A dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning, characterized in that: include: An acquisition module, used to acquire the charge state of at least one battery cell and / or at least one battery module of the dynamic reconfigurable battery network during the charging and discharging process; A generation module, configured to input the state of charge into a pre-trained deep reinforcement learning model to obtain control instructions for the dynamically reconfigurable battery network; The reconfiguration module is used to reconfigure the dynamically reconfigurable battery network in response to the control instruction so that the capacity balancing capability of the reconfigured dynamically reconfigurable battery network meets a preset capacity balancing condition.

6. The dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning according to claim 5 is characterized in that: Also includes: A construction module for constructing an initial deep reinforcement learning model based on DQN before inputting the state of charge into a pre-trained deep reinforcement learning model; A setting module, used to obtain the state quantity, action quantity and / or benefit function of the initial deep reinforcement learning model; A training module is used to train the initial deep reinforcement learning model based on the state quantity, action quantity and / or benefit function to obtain the pre-trained deep reinforcement learning model.

7. The dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning according to claim 6 is characterized in that: The expression of the profit function is: Among them, σ is used to characterize the imbalance degree of monomer SOC.

8. The dynamic reconfigurable battery network capacity balancing device based on deep reinforcement learning according to claim 5 is characterized in that: The reconstruction module comprises: A verification unit, configured to verify the reconstructed dynamic reconfigurable battery network based on the state of charge in response to the control instruction, and obtain a verification result of the reconstructed dynamic reconfigurable battery network; A judging unit, configured to judge whether the capacity balancing capability of the reconfigured dynamic reconfigurable battery network meets a preset capacity balancing condition according to the verification result; A retraining unit is used to retrain the deep reinforcement learning model when the capacity balancing capability of the reconstructed dynamic reconfigurable battery network does not meet the preset capacity balancing condition, until the capacity balancing capability of the reconstructed dynamic reconfigurable battery network meets the preset capacity balancing condition.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning as described in any one of claims 1 to 4.

11. A computer program product, characterized in that It includes a computer program, which, when executed, is used to implement the dynamic reconfigurable battery network capacity balancing method based on deep reinforcement learning as described in any one of claims 1 to 4.

Images