Reconfigurable battery topological structure and deep reinforcement learning control method

Abstract

The invention discloses a reconfigurable battery topological structure and a deep reinforcement learning control method, and relates to the technical field of battery energy storage. Comprising n batteries and a switch array and is used for realizing battery isolation, series connection and parallel connection; when the battery Bi is connected in series, the first switch Si, 1 and the third switch Si, 3 are disconnected, and the second switch Si, 2 is closed; when the batteries Bi are connected in parallel, the second switches Si and 2 are switched off, and the first switches Si and 1 and the third switches Si and 3 are switched on; when the battery Bi is isolated, the first Si, 1 and the third Si, 3 of the switch are switched off, the second Si, 2 of the switch is switched on, the first front switch Si-1, 1 and the second front switch Si-1, 2 of the previous battery are switched off, and the third front switch Si-1, 3 of the previous battery is switched on. According to the method, the state value and the action advantages are calculated respectively, the value of each state is estimated more accurately, the problem of excessive estimation of the Q-value by the DQN is relieved, the stability of the algorithm is improved, and the convergence speed of the model is increased.

Description

Technical Field

[0001] The present invention relates to the field of battery energy storage technology, and more specifically to a reconfigurable battery topology structure and a deep reinforcement learning control method. Background Art

[0002] Battery energy storage systems have the advantages of fast power response, dense energy storage, and flexible deployment. They are one of the fastest-growing, most widely used, and more mature energy storage technologies in the energy storage industry. In order to respond to complex user needs and obtain higher voltage and capacity, the system usually integrates a large number of single cells into a battery pack in a fixed series-parallel manner. However, due to the uneven manufacturing process of battery manufacturers, the voltage, capacity, and internal parameters of single cells cannot be completely unified, which can easily cause battery performance imbalance during operation and cause the "short board effect" of the system, greatly reducing the overall performance, safety, and life of the system. Therefore, researchers have proposed a variety of solutions to avoid the "short board effect" of the system to extend battery life, improve battery performance, and the reliability of energy storage systems. One is to design an energy transfer battery balancing method, which is mainly divided into two categories: passive balancing and active balancing. Among them, active balancing is a common balancing method. It constructs additional energy transmission channels through capacitors, inductors, transformers, power electronic converters, and other devices to realize the transfer of battery energy between different batteries, but this method will result in energy waste and increased system costs. Another is to design a balancing method for reconfigurable battery networks, which is recognized as the most promising solution to overcome these defects. In the reconfigurable battery topology composed of switches and batteries, the battery connection method is dynamically adjusted according to factors such as battery status and environment to achieve battery balancing.

[0003] In order to overcome the differences between batteries, the use of reconfigurable battery packs to achieve consistent battery performance and improve system performance is a research hotspot in the current energy storage industry. A reconfigurable battery pack means that each single cell is connected to a set of high-speed power electronic switches to form a dynamically reconfigurable battery network. In the network, the topology of the battery cells will be reconfigured according to the state and load of each battery. However, the dynamic reconfiguration of the battery pack has a path combination optimization problem. Summary of the invention

[0004] In view of this, the present invention provides a reconfigurable battery topology structure and a deep reinforcement learning control method.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A reconfigurable battery topology structure includes n batteries and a switch array for realizing battery isolation, series connection and parallel connection;

[0007] When battery B iWhen connected in series, the first switch S i,1 and the third S i,3 Disconnect, the second switch S i,2 Closed; when battery B i When connected in parallel, the second switch S i,2 Open, the first switch S i,1 and the third S i,3 Closed; when battery B i When isolating, switch first S i,1 and the third S i,3 Disconnect, switch second S i,2 Closed, and the first front switch S of the previous battery i-1,1 and the second antecedent S i-1,2 Disconnect, the third front switch S i-1,3 closure.

[0008] A reconfigurable battery topology deep reinforcement learning control method comprises the following steps:

[0009] Obtaining the SOC value of each single battery in the topology structure to determine the SOC difference in the battery pack;

[0010] In the Dueling DQN network structure improved based on the deep reinforcement learning DQN algorithm, the SOC of each battery is used as the input of the network and the Q-value of each action is output.

[0011] Optionally, the Dueling DQN network structure combines the separated state value and advantage value branches according to the following formula to estimate the Q-value:

[0012] Q(s,a;θ,η,β)=V(s,θ,β)+A(s,a;θ,η);

[0013] Among them, V represents state value; A represents advantage value; s represents state; a represents action; θ is the first hidden layer parameter; η and β are value layer and advantage layer parameters respectively.

[0014] Optionally, the Q-value is also centralized, and the advantage value A is expressed as the advantage value of a certain action minus the average of all advantage values ​​in the state. The updated calculation method is as follows, where |A| is the behavior dimension;

[0015]

[0016] Among them, V represents state value; A represents advantage value; s represents state; a represents action; θ is the first hidden layer parameter; η and β are value layer and advantage layer parameters respectively.

[0017] Optionally, the Dueling DQN network structure training process includes initializing three variables: experience pool, agent, and environment.

[0018] Optionally, the agent defines three space variables, including state space, behavior space, and optimization goal.

[0019] Optionally, the state space is the set of all SOC combinations exhibited by all batteries in the battery pack, and the state at time t is defined as follows:

[0020]

[0021] Represented as the SOC value of the i-th battery at time t.

[0022] Optionally, the behavior space is a single battery. The behavior is the connection state of the battery, that is, 0 and 1, which are mapped to the state of the switch according to the preset. Then the behavior space is the number of all combinations of different connection states of each battery. The behavior a at time t t It is a 0-1 matrix consisting of switch states.

[0023] Optionally, the optimization goal is to determine the equilibrium effect, the reward r at time t t Definition, where SOC mean is the average SOC of the battery pack; according to the definition of reward as follows, r t The closer it is to 0, the smaller the SOC difference between batteries is, and the better the balancing effect is:

[0024]

[0025] It can be seen from the above technical solution that compared with the prior art, the present invention provides a reconfigurable battery topology structure and a deep reinforcement learning control method, which utilizes its model-free characteristics to significantly reduce the complexity of building experience or expert models, and proposes a Dueling DQN method to achieve SOC balancing of reconfigurable battery packs. The algorithm calculates the state value and action advantage respectively, estimates the value of each state more accurately, and alleviates the problem of DQN's over-estimation of the Q-value, so as to improve the stability of the algorithm and accelerate the convergence of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0027] Figure 1A schematic diagram of the topological structure of a reconfigurable battery pack network of the present invention;

[0028] Figure 2a A schematic diagram of a serial connection mode of batteries of the present invention in a reconfigurable battery pack network topology structure;

[0029] Figure 2b A schematic diagram of a parallel connection mode of batteries of the present invention in a reconfigurable battery pack network topology structure;

[0030] Figure 2c A schematic diagram of an isolated access method of a battery in a reconfigurable battery pack network topology structure of the present invention;

[0031] Figure 3 Schematic diagram of the Dueling DQN network structure of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Compared with fixed-connected battery packs, reconfigurable battery packs have the following advantages: (1) Protecting battery safety. When a battery experiences thermal runaway, overcharge, over-discharge, or failure, the reconfigurable battery pack can change the circuit connection method and isolate the abnormal battery. (2) Improving the performance of the battery energy storage system. Due to battery differences during battery operation, the battery pack ends its operation prematurely, reducing the available capacity of the battery pack. Therefore, the use of a reconfigurable battery pack can achieve battery balancing, greatly maximizing the available capacity, extending the battery pack's operating time, improving system performance, and extending battery life.

[0034] In order to fully utilize the advantages of reconfigurable battery packs, the existing switch bypass type reconfigurable network topology flexibility is often low. The reason for proposing the switch bypass type is to illustrate that the three-switch array type of the present invention has the advantages of being more flexible, etc., and can only realize the two operations of battery access and isolation. Therefore, the present invention adopts a reconfigurable topology structure with high flexibility that can realize battery isolation, series connection and parallel connection. The structure is as follows Figure 1 As shown, the battery pack consists of n batteries and a switch array, and the connection status of the battery is determined by three switch states.

[0035] The embodiment of the present invention discloses a reconfigurable battery topology structure, including n batteries and a switch array, for realizing battery isolation, series connection and parallel connection;

[0036] like Figure 2a As shown, when battery B i When connected in series, the first switch S i,1 and the third S i,3 Disconnect, the second switch S i,2 Closed; such as Figure 2b As shown, when battery B i When connected in parallel, the second switch S i,2 Open, the first switch S i,1 and the third S i,3 Closed; such as Figure 2c As shown, when battery B i When isolating, switch first S i,1 and the third S i,3 Disconnect, switch second S i,2 Closed, and the first front switch S of the previous battery i-1,1 and the second antecedent S i-1,2 Disconnect, the third front switch S i-1,3 closure.

[0037] In order to achieve battery SOC balancing, the present invention does not consider all battery connection conditions, but only considers two behaviors: battery series connection and battery parallel connection. The battery connection state is represented by 0 for series connection and 1 for parallel connection. The switch state is represented by 0 for disconnection and 1 for closing. i The corresponding relationship between the connection status and the switch status is shown in Table 1.

[0038] Table 1 Battery B i The corresponding relationship between the connection status and the switch status

[0039]

[0040] A reconfigurable battery topology deep reinforcement learning control method comprises the following steps:

[0041] Obtaining the SOC value of each single battery in the topology structure to determine the SOC difference in the battery pack;

[0042] The Q-learning algorithm is combined with a deep neural network to obtain the Dueling DQN network structure, which takes the SOC of each battery as the input of the network and outputs the Q-value of each action.

[0043] exist Figure 1 In the reconfigurable battery network topology shown in the figure, SOC balancing is achieved. In a battery pack with n batteries and SOC differences between the batteries, the SOC state of the battery pack is represented by ST = {SOC1, SOC2, ..., SOC i ,…,SOC n},SOC iThe SOC of the ith battery is represented by the SOC of the battery pack. The SOC difference of the battery pack is quantified by formula (1). Then the SOC balancing of the battery pack means that the extreme difference ΔSOC is getting smaller and smaller during operation.

[0044] ΔSOC=max(ST)-min(ST)(1)

[0045] The core idea of ​​the Dueling DQN network structure is to combine the Q-learning algorithm with a deep neural network. Specifically, DQN uses a deep neural network to approximate the Q function, that is, the state-action value function. The Q function is used to estimate the expected reward that can be obtained by performing a certain action in a certain state. Due to the presence of noise or errors, the DQN algorithm will cause over-estimation of the Q-value during training, resulting in algorithm instability and slow convergence speed, so that the reconfigurable battery pack cannot quickly find the optimal behavior. Therefore, the present invention proposes an improved Dueling DQN algorithm based on DQN as a control strategy for the reconfigurable battery pack. The algorithm also uses a behavior network and a target network. The behavior network is used as a policy network to select behaviors according to the environment. The target network is used to update network parameters, and the training process is the same as that of DQN. Compared with the DQN algorithm, the difference lies in the output of the network structure. Dueling DQN introduces two branches, state value and action advantage, to calculate the Q-value, takes the SOC of each battery as the input of the network, and outputs the Q-value of each action. The network structure is as follows Figure 3 shown.

[0046] exist Figure 3 In the above method, the two separated branches of state value and action advantage are combined according to formula (2) to estimate the Q-value. The advantage of this method is that even if the action advantage is overestimated, the state value can still provide a relatively accurate value estimate, thereby reducing the impact of overestimation on the overall Q-value.

[0047] Q(s,a;θ,η,β)=V(s,θ,β)+A(s,a;θ,η)(2)

[0048] In formula (2), s is the state; a is the action; θ is the first hidden layer parameter; η and β are the value layer and advantage layer parameters respectively. If the neural network trains V to a fixed value of 0, then the A value is equal to the Q-value, which is equivalent to the DQN network. Therefore, it is necessary to centralize the Q-value and express the advantage value A as the advantage value of a certain action minus the average of all advantage values ​​in the state. The updated calculation method is shown in formula (3), where |A| is the behavior dimension.

[0049]

[0050] Dueling DQN is trained according to Algorithm 1. First, the three variables of experience pool, agent and environment are initialized. In the code, M and T represent the total number of episodes and the step size of each episode, respectively. Line 7 introduces the ∈-greedy algorithm. By randomly selecting actions with a certain probability, it helps the agent explore the unknown environment on the one hand, and prevents it from falling into the local optimum on the other hand, and finds the global optimal strategy. On line 8, the agent's behavior network will execute formula (3), select the behavior with the largest Q-value, and then execute the behavior in the reconfigurable battery pack environment, and save the reward and the state at the next moment to the experience pool. On line 12, the agent updates the network parameters, selects a batch of data (S, A, R, S_) from the experience pool to train the behavior network, and uses the Q-value calculated by the behavior network according to (S, A) as the predicted value y, expressed as Q(S, A; ξ), where ξ is the parameter of the neural network. The true label y i The R and target networks need to be constructed according to formula (4), and the network parameters are updated according to formulas (5)-(7), where α is the learning rate. When the network is updated, the behavior network is updated every step, while the target network is updated every C steps. The training process is the same as the DQN algorithm, as shown in Table 2.

[0051] Table 2

[0052]

[0053]

[0054] y i =R+γmax(Q(S - ,A;ξ - ))(4)

[0055] L(ξ)=y i -Q(S,A;ξ)) 2 (5)

[0056] ▽ ξ L(ξ)=(R+γmax(Q(S - ,A;ξ - ))-Q(S,A;ξ))▽ ξ Q(S,A;ξ)(6)

[0057] ξ=ξ+α▽ ξ L(ξ)(7)

[0058] Agent defines three space variables, including state space, behavior space and optimization goal.

[0059] State space: The agent selects the best behavior according to the environmental state. In order to achieve battery SOC balance, the present invention uses the SOC of all batteries in the reconfigurable battery pack as the state. The state at time t is defined as formula (8)

[0060]

[0061] Behavior space: The behavior of a single battery is the connection state of the battery, that is, 0 and 1, and then mapped to the state of the switch according to Table 1. The behavior space is the number of all combinations of different connection states of each battery. Therefore, the behavior a at time t is t It is a 0-1 matrix consisting of switch states.

[0062] Optimization goal: The smaller the ΔSOC between batteries, the better the balancing performance. In order to minimize the difference, the present invention sets the reward r at time t. t Defined as formula (9), where SOC mean is the average SOC of the battery pack. According to the definition of the reward, r t The closer it is to 0, the smaller the SOC difference between batteries is, and the better the balancing effect is.

[0063]

[0064] The total reward of an Episode, i.e. the return, is defined as formula (10), where T is the running time of an Episode.

[0065]

[0066] It should be noted that one of the major advantages of deep reinforcement learning models is that there is no need to prepare a data set. Instead, the agent repeatedly executes multiple episodes in a pre-designed environment. An episode refers to the process from the start to the end of a task. The agent continuously interacts with the environment to learn in an episode. In this process, each action taken by the agent may change its state, and the reward is calculated by the state change before and after the action is taken. By collecting the state, behavior, and reward information during the interaction process and feeding it back to the agent, it can find the optimal strategy. The goal of this strategy is always to maximize the total reward of an episode, that is, to maximize the return.

[0067] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0068] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reconfigurable battery topology, characterized in that: It includes n batteries and a switch array for isolating, connecting in series and in parallel the batteries; When battery B i When connected in series, the first switch S i,1 and the third S i,3 Disconnect, the second switch S i,2 Closed; when battery B i When connected in parallel, the second switch S i,2 Open, the first switch S i,1 and the third S i,3 Closed; when battery B i When isolating, switch first S i,1 and the third S i,3 Disconnect, switch second S i,2 Closed, and the first front switch S of the previous battery i-1,1 and the second antecedent S i-1,2 Disconnect, the third front switch S i-1,3 closure.

2. A deep reinforcement learning control method for reconfigurable battery topology, characterized in that: The following steps are involved: Obtaining the SOC value of each single battery in the topology structure to determine the SOC difference in the battery pack; In the Dueling DQN network structure improved based on the deep reinforcement learning DQN algorithm, the SOC of each battery is used as the input of the network and the Q-value of each action is output.

3. A reconfigurable battery topology deep reinforcement learning control method according to claim 2, characterized in that: In the Dueling DQN network structure, the two branches of separated state value and advantage value are combined according to the following formula to estimate the Q-value: Q(s,a;θ,η,β)=V(s,θ,β)+A(s,a;θ,η); Among them, V represents state value; A represents advantage value; s represents state; a represents action; θ is the first hidden layer parameter; η and β are value layer and advantage layer parameters respectively.

4. A reconfigurable battery topology deep reinforcement learning control method according to claim 2, characterized in that: It also includes centralizing the Q-value, expressing the advantage value A as the advantage value of a certain action minus the average of all advantage values ​​in that state. The updated calculation method is as follows, where |A| is the behavior dimension; Among them, V represents state value; A represents advantage value; s represents state; a represents action; θ is the first hidden layer parameter; η and β are value layer and advantage layer parameters respectively.

5. A reconfigurable battery topology deep reinforcement learning control method according to claim 2, characterized in that: The Dueling DQN network structure training process includes initializing three variables: experience pool, agent, and environment.

6. A reconfigurable battery topology deep reinforcement learning control method according to claim 5, characterized in that: Agent defines three space variables, including state space, behavior space and optimization goal.

7. A reconfigurable battery topology deep reinforcement learning control method according to claim 6, characterized in that: The state space is the set of all SOC combinations exhibited by all batteries in the battery pack, and the state at time t is defined as follows: Represented as the SOC value of the i-th battery at time t.

8. A reconfigurable battery topology deep reinforcement learning control method according to claim 6, characterized in that: The behavior space is that the behavior of a single battery is the connection state of the battery, that is, 0 and 1, which are mapped to the state of the switch according to the preset. Then the behavior space is the number of all combinations of different connection states of each battery. The behavior a at time t t It is a 0-1 matrix consisting of switch states.

9. A reconfigurable battery topology deep reinforcement learning control method according to claim 6, characterized in that: The optimization goal is to determine the equilibrium effect, the reward r at time t t Definition, where SOC mean is the average SOC of the battery pack; according to the definition of reward as follows, r t The closer it is to 0, the smaller the SOC difference between batteries is, and the better the balancing effect is:

Images