A battery equalization method and device based on reinforcement learning

Abstract

The application provides a battery equalization method based on reinforcement learning, comprising the following steps: establishing a physical field model of a battery control board as a simulation environment for battery equalization, wherein the physical field model comprises system parameters to be identified; collecting experimental data of the battery control board, and identifying and determining the system parameters in the physical field model based on the experimental data; and based on a reinforcement learning algorithm, training a neural network through interaction with the simulation environment in combination with a greedy rule and a preset constraint condition, and realizing battery equalization control through an equalization control signal output by the neural network.

Description

Technical Field

[0001] This invention relates to the field of battery equalization, and more particularly to a battery equalization method and apparatus based on reinforcement learning. Background Technology

[0002] With the increasing popularity of electric vehicles, the capacity and lifespan of power batteries are receiving more and more attention. Among the many methods to improve battery capacity and lifespan, battery balancing ensures that the voltage deviation of individual battery cells or the battery pack remains within a expected range, thereby guaranteeing that each individual battery cell remains in the same state during normal use and preventing overcharging and over-discharging. More specifically, the battery balancing controller eliminates cell inconsistencies through discharge, thereby improving the overall capacity and lifespan of the battery.

[0003] However, due to the relatively small discharge current and the ever-increasing capacity of power batteries, balancing efficiency has become a major challenge in achieving effective balancing. Existing power battery controllers often employ a greedy algorithm to determine the switching of the balancing channel. A greedy algorithm always makes the choice that seems best at the moment, without considering the overall optimal solution; the algorithm obtains a locally optimal solution in some sense. However, battery balancing problems typically require consideration of hardware constraints. When considering hardware constraints, the greedy algorithm suffers from a long balancing time, which cannot meet the ever-increasing demands of usage.

[0004] In order to overcome the above-mentioned defects of the existing technology, there is an urgent need in the field for a battery balancing method that can significantly improve balancing efficiency while ensuring the accuracy of battery balancing, and is suitable for widespread application. Summary of the Invention

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0006] To overcome the aforementioned deficiencies in the existing technology, this invention provides a battery balancing method based on reinforcement learning, comprising: establishing a physical field model of a battery control board as a simulation environment for battery balancing, the physical field model including system parameters to be identified; collecting experimental data of the battery control board, identifying and determining the system parameters in the physical field model based on the experimental data; and training a neural network based on a reinforcement learning algorithm, combined with greedy rules and preset constraints, through interaction with the simulation environment, and realizing battery balancing control through the balancing control signal output by the neural network.

[0007] In one embodiment, preferably, the reinforcement learning algorithm-based training of the neural network through interaction with the simulation environment, combined with greedy rules and preset constraints, may include: inputting state data from the simulation environment into the neural network, the state data including temperature sampling signal values ​​and cell balancing requirements; outputting action data from the simulation environment based on the greedy rules and preset constraints, the action data including multiplexer signals for discharging in the battery control board; the neural network outputting voting signals to the simulation environment based on the state data and action data, the voting signals including control signals for multiplexers for discharging in the battery control board; and negatively taking the total time scalar value required for the battery to reach balancing as the cumulative reward in the reinforcement learning algorithm and feeding it back to the neural network, repeating the above steps to train the neural network to maximize the cumulative reward, and performing motor balancing control based on the final output balancing control signal.

[0008] In one embodiment, preferably, the reinforcement learning algorithm includes the PPO2 algorithm, and the neural network includes a policy network and a value network. The reinforcement learning algorithm, combined with greedy rules and preset constraints, trains the neural network through interaction with the simulation environment. This may further include: the policy network receiving state data from the simulation environment and returning the current control variable, recording the result data of the interaction with the simulation environment in sample playback; the value network undergoing supervised learning to estimate the reward value in the current state, where the reward value is the negative of the time scalar value required for battery equalization control in the current state; the policy network determining an update gradient based on an advantage estimator and an entropy incentive term, where the advantage estimator is used to estimate the advantage of the current state by subtracting the reward value estimate from the actual reward of the current control variable, and the entropy incentive term is used to encourage the exploration of different strategies; and updating the weights of the policy network using gradient descent, feeding back the updated control variable and state data to the policy network, repeating the above steps to gradually reduce the entropy incentive term, thereby causing the policy network to converge to a deterministic policy.

[0009] In one embodiment, preferably, the reinforcement learning algorithm-based training of the neural network through interaction with the simulation environment, combined with greedy rules and preset constraints, may further include: performing an AND operation on the control signal of the deterministic policy finally output by the policy network and the multiplexer signal obtained based on the greedy rule to determine the voting signal. The AND operation includes: in response to the greedy rule requiring a certain switch to be closed, regardless of the control signal output by the policy network, the switch is closed in the voting signal; and in response to the greedy rule requiring a certain switch to be opened, the control value of the switch in the control signal output by the policy network is used as the control value of the switch in the final equalization control signal.

[0010] In one embodiment, preferably, the greedy rule may include the calculation of the remaining power of the battery cell, and the preset constraint may include the temperature constraint of the PCB thermal model in the battery control board.

[0011] In one embodiment, preferably, the battery is composed of multiple sets of cells connected in series, each cell corresponding to a discharge channel controlled by a switch. The preset constraint may also include a hardware constraint to ensure that the switches of adjacent discharge channels cannot be turned on simultaneously.

[0012] In one embodiment, preferably, establishing a physical field model of the battery control board as a simulation environment for battery equalization may include: modeling the battery control board as a differential algebraic equation model, abstracting the PCB board as multiple blocks of units, assuming that the parameters of the multiple units are isotropic; and adding a constant heat source at a predetermined location to simulate the heating of electronic components on the battery control board other than the discharge resistor.

[0013] In one embodiment, preferably, the physical field model of the battery control board used as a simulation environment for battery equalization may further include: using Simulink / Simscape to build the physical field model, wherein the outer Simulink solver of the model adopts a fixed-step discrete solver, and the inner Simscape solver of the model adopts an inverse Euler solver.

[0014] In one embodiment, preferably, the identification and determination of the system parameters in the physical field model based on the experimental data may include: fitting the experimental data using a white-box calibration combined with a search algorithm to determine the values ​​of the system parameters.

[0015] In one embodiment, preferably, the system parameters may include the thermal capacity coefficient, thermal conductivity coefficient, thermal convection coefficient, the length of the battery control board in each direction, external heat generation, mass density, and resistivity heat generation. The experimental data may include experimental data of the battery control board in a constant temperature chamber. The method of fitting the experimental data using white-box calibration combined with a search algorithm to determine the values ​​of the system parameters may include: except for specific parameters, other system parameters are written into the physical field model through manual measurement. The specific parameters are a portion of the system parameters including the thermal capacity coefficient, thermal conductivity coefficient, thermal convection coefficient, external heat generation, mass density, and the length of the battery control board in a specific direction; and for the specific parameters, the approximate value range of the parameters is first given by material properties, and then the parameter value closest to the experimental data is searched in the parameter space using a genetic algorithm. The specific direction is the direction in which the heat distribution on the battery control board is uncertain.

[0016] In one embodiment, preferably, the experimental data also includes experimental data of the battery control board on a real vehicle. The identification and determination of the system parameters in the physical field model based on the experimental data may further include: establishing a model of the battery control board housing on the real vehicle, selecting the parameters of the housing model, and determining the system parameters together with the experimental data through white-box calibration and search algorithm.

[0017] In one embodiment, preferably, the battery is composed of a group of cells connected in series, the group of cells being divided into b modules with the same structure, each module having a corresponding cell. There are several discharge channels, including a, b, and All are positive integers. When a discharge channel is opened, the corresponding cell discharges through the resistor connected in parallel within that discharge channel. Temperature sensors are provided on both the front and back sides of the battery control board in the center of each module.

[0018] In one embodiment, preferably, the state data includes a (b·c)-dimensional temperature vector and an a-dimensional equalization demand vector. The (b·c)-dimensional temperature vector includes b numerical values ​​of c groups of temperature sampling signal values ​​from the temperature sensors in the module. Each equalization demand vector corresponds to the equalization demand value of a battery cell, and the unit of the equalization demand value is Ah. The action data includes a-dimensional switch signal Boolean values, representing the switching of the discharge channel corresponding one-to-one with a group of battery cells. The voting signal includes... The switch control signals satisfy the hardware constraint that ensures the switches of adjacent discharge channels cannot be opened simultaneously, where a, b, c, and All are positive integers.

[0019] Another aspect of the present invention provides a reinforcement learning-based battery equalization device, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the reinforcement learning-based battery equalization method as described in any of the preceding claims.

[0020] The present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the reinforcement learning-based battery balancing method as described in any of the preceding claims. Attached Figure Description

[0021] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0022] Figure 1 This is a schematic diagram of a battery balancing circuit for a single discharge channel according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the unfolded equalization channel of a battery control board according to an embodiment of the present invention.

[0024] Figure 3 This is a schematic flowchart illustrating a battery balancing method based on reinforcement learning according to an embodiment of one aspect of the present invention.

[0025] Figure 4 , 5 These are simplified model diagrams of the battery control board on two directional planes.

[0026] Figure 6 , 7 These are simulation diagrams of the thermal field model of a single module area and the entire battery control board in Simulink.

[0027] Figure 8 This is a graph illustrating experimental data collection according to an embodiment of the present invention;

[0028] Figures 9-11 This is a parameter calibration result diagram of the NTC temperature curves of the experimental plate in three module regions in a constant temperature chamber, according to an embodiment of the present invention.

[0029] Figures 12-14 This is a parameter calibration result diagram of the NTC temperature curves of three module areas of the battery control board on a real vehicle, according to an embodiment of the present invention.

[0030] Figure 15 This is a schematic diagram illustrating the principle of a battery balancing system according to an embodiment of the present invention;

[0031] Figure 16 This is a block diagram illustrating a neural network strategy training method according to an embodiment of the present invention;

[0032] Figure 17 This is a comparison diagram of the opening and closing of the equalization channel during the equalization process in an embodiment of the present invention and in the prior art;

[0033] Figure 18 This is a comparison diagram of the change in electricity demand over time during the balancing process between an embodiment of the present invention and the prior art;

[0034] Figure 19 This is a comparison graph of temperature change over time during the equilibration process of an embodiment of the present invention and the prior art;

[0035] Figure 20 This is a schematic diagram illustrating the change over time in the total number of channels opened during the balancing process of an embodiment of the present invention and the prior art;

[0036] Figure 21 This is a schematic diagram of the average temperature of the thermal field during the equilibrium process, comparing an embodiment of the present invention with that of the prior art; and

[0037] Figure 22 This is a schematic diagram of the device structure of a battery equalization device based on reinforcement learning, according to another embodiment of the present invention.

[0038] For clarity, a brief explanation of the reference numerals in the accompanying drawings is provided below:

[0039] 101 battery cell

[0040] 102a and 102b resistors

[0041] 103 Switch

[0042] 1501 Neural Network

[0043] 1502 Simulation Environment

[0044] 1601 Value Network

[0045] 1602 Policy Network

[0046] 1603 Advantage Estimator

[0047] 1604 Entropy Incentive Term

[0048] 1605 Greed Rule Detailed Implementation

[0049] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0052] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0053] Battery balancing refers to maintaining the voltage deviation of individual cells or the battery pack within a expected range through appropriate discharge, thereby ensuring that each individual cell remains in the same state during normal use and avoiding overcharging or over-discharging.

[0054] Figure 1 This is a schematic diagram of a battery balancing circuit for a single discharge channel according to an embodiment of the present invention.

[0055] like Figure 1As shown, the arrow indicates a balanced discharge channel circuit. Each channel corresponds to one battery cell. When a channel is open, the battery cell discharges through two parallel resistors in the circuit. For example, in... Figure 1 In the embodiment shown, switch 103 is turned on in the equalization channel indicated by the arrow, and cell 101 discharges through resistors 102a and 102b.

[0056] Figure 2 This is a schematic diagram illustrating the unfolded equalization channel of the battery control board according to an embodiment of the present invention.

[0057] Please refer to Figure 2 In this embodiment, the power battery controlled by the experimental battery control board consists of 64 groups of cells connected in series. These 64 groups of cells can be divided into four isomorphic modules, namely module one to module four in the figure. Therefore, each control module has 16 equalization discharge channels. Figure 2 As shown, the numbers in the small boxes represent the indices of the resistor columns, which can be combined with... Figure 1 The equalization channel N uses resistor array N-1 and resistor array N. Figure 2 The NTC in this design is a thermistor used as a temperature sensor. Preferably, as part of the engineering redundancy design, an NTC can be placed at the same position in the length and width directions on both the front and back of the control board.

[0058] This article uses the battery control board structure of the above embodiment as an example to describe the battery balancing method provided by the present invention.

[0059] Figure 3 This is a schematic flowchart illustrating a battery balancing method based on reinforcement learning according to an embodiment of one aspect of the present invention.

[0060] Please refer to Figure 3 The battery equalization method 300 based on reinforcement learning provided by this invention may include:

[0061] Step 301: Establish a physical field model of the battery control board as a simulation environment for battery equalization. The physical field model includes the system parameters to be identified.

[0062] Step 302: Collect experimental data from the battery control board, and identify and determine the system parameters in the physical field model based on the experimental data; and

[0063] Step 303: Based on the reinforcement learning algorithm, combined with the greedy rule and preset constraints, the neural network is trained through interaction with the simulation environment, and the battery equalization control is achieved through the equalization control signal output by the neural network.

[0064] Each step will be explained in detail below.

[0065] Figure 4 ,5 This is a simplified schematic diagram of the battery control board on two different directional planes.

[0066] Please refer to Figure 4 The dashed box area represents the resistor column and equalization channel corresponding to Module 1. Taking Module 1 as an example, each module area is divided into 9 units, as shown by the vertical lines along the y-axis in the diagram. This can be combined with... Figure 5 The small boxes indicate the resistor columns arranged on the front and back of the battery control board for heat dissipation. The numbers in the boxes still indicate the resistor column indices. The NTC401 is located in the middle of the battery control board, which is consistent with the actual situation.

[0067] Preferably, to simplify the model, it is assumed that the nine units are nearly identical, and that the heat dissipation of the components along the y-direction is minimal.

[0068] The next step is to perform step 301: establish a physical field model of the battery control board as a simulation environment for battery equalization.

[0069] More specifically, the battery control board can be modeled as a differential algebraic equation (DAE) model. Similarly, the PCB board can be abstracted into multiple units, assuming that the parameters of these multiple units are isotropic, thereby reducing the computation time and complexity.

[0070] Simultaneously, a constant heat source can be added at predetermined locations to simulate the heating of electronic components on the battery control board, excluding the discharge resistor. For example, in addition to the cells within the module area, intermediate heat conduction units in the x-direction can be considered, and a constant heat source can be added in the y-direction of each region to approximate the heating of other electronic components besides the discharge resistor. Since the experiment is conducted in a constant temperature chamber, the entire thermal field can be assumed to be in an ideal temperature source with infinite heat capacity.

[0071] Preferably, in order to speed up the simulation, in one embodiment, the physical field model is built using Simulink / Simscape, wherein the outer Simulink solver of the model adopts a fixed-step discrete solver, and the inner Simscape solver of the model adopts an inverse Euler solver.

[0072] Figure 6 , 7 These are simulation diagrams of the thermal field models of a single module area and the entire battery control board in Simulink.

[0073] For example, the simulation step size for modeling used to identify system parameters can be set to 100 seconds per step, or this step size parameter can be adjusted according to actual needs. The model can generate code using the Simulink embedded coder and compile it into a .dll dynamic link library, which can then be used to call the thermal field system in other programming environments (such as Python).

[0074] It should be noted that the use of Simulink / Simscape to build the physics model is merely an illustrative example. The Simulink / Simscape physics model is a simulation environment with good generalization and can be used for reinforcement learning interaction. Therefore, it is a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Other model forms, such as mechanistic models, data-driven models, or surrogate models, as well as data augmentation models, i.e., initializing mechanistic models in a data-driven manner, can all be applied to the battery balancing method of the present invention and should also be included within the scope of protection of the present invention.

[0075] Experiments showed that, in the Python environment, the simulation of the thermal field system for 3500 seconds (36 sampling points, each sampling point with 40-dimensional signals) took an average of 0.60 seconds (10 times), which effectively improved the computational efficiency.

[0076] Step 301 mentions that the physical field model includes system parameters to be identified, which can be combined with... Figure 4 , 5 In this embodiment, these parameters may include, for example,: thermal capacity coefficient (J / kg*K), thermal conductivity coefficient (W / m*K), thermal convection coefficient (W / m^2*K), length in the x-direction (m), length in the y-direction (m), length in the z-direction (m), external heat generation (W), mass density (kg / m^3), and resistivity heat generation (W). Considering the relatively small heterogeneity of each unit, it can be assumed that each unit has the same parameters, such as the same thermal capacity, thermal conductivity, or density.

[0077] After constructing the structure of the digital model, the model parameters need to be calibrated to align with the real system. That is, step 302 is executed: the experimental data of the battery control board is collected, and the system parameters in the physical field model are identified and determined based on the experimental data.

[0078] In one embodiment, an equalization channel switching experiment can be first conducted in a constant temperature chamber to monitor the temperature of the NTC and collect data.

[0079] Figure 8 This is a graph illustrating experimental data collection according to an embodiment of the present invention.

[0080] like Figure 8 As shown, in this embodiment, region three of the battery control experimental board is unloaded, so only the remaining region is operational. The horizontal axis in the figure represents time, and the vertical axis represents the temperature values ​​of each NTC on the experimental board. Figure 8The figure shows the temperature changes of the NTC in the three regions during the process of channel 16 in region 2 being opened for 2168 seconds and then closed. It can be seen that opening the channel in region 2 causes temperature changes in other regions, indicating a coupling relationship between temperature changes in multiple regions.

[0081] To reduce the search space of the search algorithm, in a preferred embodiment, white-box calibration combined with the search algorithm can be used to fit the experimental data to determine the values ​​of the system parameters.

[0082] For example, apart from specific parameters, other system parameters are written into the physical field model through manual measurement. The specific parameters are some system parameters that include the thermal capacity coefficient, the thermal conductivity coefficient, the thermal convection coefficient, the external heat generation, the mass density, and the length of the battery control board in a specific direction.

[0083] These other system parameters can be easily measured, for example, they can be combined with Figure 4 , 5 The x-direction (long side) and z-direction (thickness) of the PCB board are fixed, so the parameters of the x and z directions can be manually measured and written.

[0084] For this specific parameter, the approximate range of the parameter value is first given by the material properties, and then the parameter value that is closest to the experimental data is searched in the parameter space by a genetic algorithm. This specific direction is the direction in which the heat distribution on the battery control board is uncertain.

[0085] In this embodiment, these specific parameters can be six parameters: heat capacity, thermal conductivity, heat convection coefficient, length in the y-direction, external heat generation, and mass density. Similarly, they can be combined... Figure 4 , 5 The y-direction (short side) of the PCB board contains other components and heat dissipation areas, which have uncertainties. Therefore, the length of the y-direction is used as a specific parameter so that this part of the uncertainty is automatically corrected by the length variable of the y-direction.

[0086] Figures 9-11 This is a parameter calibration result diagram of the NTC temperature curves of three module regions in a constant temperature chamber, illustrating an experimental plate according to an embodiment of the present invention.

[0087] In the figure, the light-colored curve represents the experimental data of temperature changes, and the dark-colored curve represents the simulation results of the thermal model parameter calibration mentioned above.

[0088] Furthermore, the utility function of the genetic algorithm can be the multidimensional mean squared error (MSE). The population searches for the minimum MSE, where MSE is the average of all sample points in a single experiment, expressed mathematically as:

[0089]

[0090] Where y represents experimental data. The data represents the simulation data, where n represents the number of samples and c represents the number of sensors.

[0091] In this embodiment, the obtained local extremum point of MSE is 0.3866, with dimensions in °C. 2 This indicates that a good simulation effect has been achieved.

[0092] The above are the identification results of the experimental board in the constant temperature chamber. In another embodiment, modeling and parameter calibration can also be performed on a real vehicle. For example, a model of the battery control board casing on a real vehicle can be built, the parameters of the casing model can be selected, and the system parameters can be determined together with the experimental data through white-box calibration and search algorithm fitting.

[0093] Figures 12-14 This is a parameter calibration result diagram of the NTC temperature curves of three module areas of the battery control board on a real vehicle, illustrated according to an embodiment of the present invention.

[0094] Similarly, in the figure, the light-colored curve represents the experimental data of temperature changes, and the dark-colored curve represents the simulation results of the thermal model parameter calibration mentioned above. During identification, to verify the generalization ability of the mechanistic model, this embodiment only used the first 10% of the data (1401 points) for parameter training, and tested the model using the complete trajectory. The training MSE was 7.87, and the test MSE was 10.38. The test results indicate that the mechanistic model can well reflect the actual thermal field dynamics and can serve as a simulation environment for the next step of strategy training.

[0095] After determining the system parameters, step 303 is executed: based on the reinforcement learning algorithm, combined with the greedy rule and preset constraints, the neural network is trained through interaction with the simulation environment, and the battery equalization control is achieved through the equalization control signal output by the neural network.

[0096] Figure 15 This is a schematic diagram illustrating the principle of a battery balancing system according to an embodiment of the present invention.

[0097] Please refer to Figure 15 In a preferred embodiment, step 303 may further specifically include: inputting the state data from the simulation environment into the neural network 1501, wherein the state data includes temperature sampling signal values ​​and cell balancing demand values. Here, the state data refers to… Figure 15 m kMore specifically, in the battery control board embodiment described above, the measured values ​​of physical quantities can be a 40-dimensional temperature vector and a 64-dimensional equilibrium demand vector. The 40-dimensional temperature vector, easily understood, consists of the temperature signal values ​​of 10 temperature sampling points in each of the four regions on the battery control board mentioned above; these are the NTC temperature sampling signals, and these signals are continuous values. The 64-dimensional equilibrium demand vector represents the equilibrium demand value for each of the 64 battery cells, also a continuous value, measured in Ah.

[0098] It should be noted that in this embodiment, the neural network 1501 can be a simple multi-layer fully connected layer. The type of neural network described herein is merely illustrative and not intended to limit the scope of protection of this invention. Changing the neural network structure also falls under the category of neural network controllers. For example, subnetwork N1 can be used to encode the power that needs to be balanced in all regions, subnetwork N2 can be used to uniformly encode the temperature of each region, and subnetwork N3 can be used to encode the NTC of all regions. The three networks fuse information at the feature channel level to obtain the implicit expression of the system at the current moment. Finally, the input policy network and value network output the policy and value at the current moment, respectively. Similar methods can be applied to the battery balancing method provided by this invention and should also be included within the scope of protection of this invention.

[0099] At the same time, simulation environment 1502 also receives the above-mentioned status data, that is Figure 15 The process is shown in circle 1. Next, in the simulation environment 1502, action data is output based on the greedy rule and the preset constraints. This action data includes the multiplexer signals for discharging in the battery control board. That is, the process shown in circle 2, a 0,k This refers to action data.

[0100] A greedy algorithm is a method of solving a problem that always makes the best choice at the moment, without considering the overall optimal solution. The algorithm arrives at a locally optimal solution in some sense. In the battery balancing problem, a greedy algorithm can be used to calculate the remaining charge of each cell, thereby determining the on / off state of each balancing channel to achieve the balancing requirement.

[0101] In one embodiment, such as Figure 15 As shown, the preset constraints in the battery balancing method provided by this invention can include temperature constraints and hardware constraints. The temperature constraint refers to the temperature constraint of the PCB thermal model in the battery control board, while the hardware constraint can be combined with reference to... Figure 1 Hardware constraints can ensure that the switches of adjacent discharge channels cannot be turned on at the same time, which simplifies the calculation while satisfying the physical hardware constraints.

[0102] This can be combined with the above text Figure 4, 5 The simplified model of the battery control board has 16 channel switches in each area, resulting in a total of 64 channel switch signal values. These switch signals can be Boolean values. However, since adjacent discharge channels cannot be switched on simultaneously, each module area can only have a maximum of eight channels open, hence the action data here is a 32-dimensional vector.

[0103] Next, you can continue to refer to... Figure 15 In this embodiment, the neural network 1501 outputs a voting signal to the simulation environment 1502 based on the state data and the action data. This voting signal includes the control signal of the multiplexer in the battery control board used for discharge, which is the process shown by the arrow in circle 3. Here, a... k (voting) indicates a voting signal.

[0104] Unlike solutions that simply use neural networks or greedy rules, the battery balancing method provided by this invention, in this embodiment, forms a voting signal a by performing an AND operation on the outputs of both. k (voting). This process will be explained in detail again when introducing specific reinforcement learning algorithms below.

[0105] This invention employs a reinforcement learning algorithm. The total time required for the battery to reach equilibrium is negatively represented as the cumulative reward in this algorithm. Simply put, negativening the scalar time yields a non-positive real number. The training objective of reinforcement learning is to maximize the expected reward for any state, which is equivalent to maximizing the cumulative reward, i.e., obtaining the maximum value of this non-positive real number. (Continue to the next section...) Figure 15 The battery balancing method provided by this invention, after the neural network 1501 outputs a voting signal, feeds the accumulated reward back to the neural network, repeats the above steps to train the neural network to maximize the accumulated reward, and then uses the final output balancing control signal x. k Perform motor balancing control, which is the process shown in circle 4.

[0106] In a preferred embodiment, the reinforcement learning algorithm in the battery balancing method provided by the present invention can be the PPO2 algorithm. PPO2 is a reinforcement learning algorithm proposed by OpenAI, which adds a ratio to describe the difference between the new and old policies. By limiting the update step size of the policy through hyperparameters, a more ideal training effect can be obtained.

[0107] Figure 16 This is a block diagram illustrating a neural network strategy training method according to an embodiment of the present invention.

[0108] Please refer to Figure 16The neural network includes a policy network 1602 and a value network 1601. The training process of the neural network may further include: the policy network 1602 receiving state data of the simulation environment and returning the current control variable, and recording the result data of the interaction with the simulation environment to the sample playback; the value network 1601 undergoing supervised learning to estimate the reward value in the current state, which is the negative of the time scalar value required for battery equalization control in the current state; the policy network 1602 determining the update gradient based on the advantage estimator 1603 and the entropy incentive term 1604, whereby the advantage estimator 1603 is used to estimate the advantage of the current state by subtracting the reward value estimate of the value network 1601 from the actual reward of the current control variable, and the entropy incentive term 1604 is used to encourage the exploration of different policies; and updating the weights of the policy network using gradient descent, and feeding back the updated control variable and state data to the policy network 1602, repeating the above steps to gradually reduce the entropy incentive term 1604, thereby causing the policy network 1602 to converge to a deterministic policy.

[0109] In simple terms, the single-step reward in reinforcement learning can be the negative of the time of each step of control. Such a reward is dense and monotonic, and the neural network can easily obtain the gradient information of the weights from this reward, thereby improving the efficiency of the algorithm.

[0110] It should be noted that the PPO2 algorithm used here is merely a preferred example, intended to more clearly illustrate the operation steps of the battery balancing algorithm provided by this invention, and not to limit the scope of protection of this invention. In fact, other policy network-based reinforcement learning algorithms, such as DDPG, TD3, SAC, TRPO, PPO, etc., can also be used. Alternatively, other value network-based reinforcement learning algorithms, such as a series of variants of DQN, can also be used. These algorithms differ only in the parameter tuning methods of the neural network and still fall within the scope of neural network controllers. Different reward designs can also be used during training; for example, slight modifications can be combined, such as if a reward is found... If temperature or hardware limitations are violated, a pre-defined penalty is directly added to the reward at the current step, preventing the learned policy from generating more actions that would violate the limitations. Furthermore, because the reward or penalty at a future step will affect the current step through the V or Q value, this design prevents the learned policy from tending to generate actions that will violate the limitations in the future. Similar algorithms can be applied to the battery balancing method provided in this invention and should also be included within the scope of protection of this invention.

[0111] Furthermore, in a preferred embodiment, please combine Figure 16The process of training the neural network may also include: performing an AND operation between the control signal of the deterministic policy finally output by the policy network 1602 and the multiplexer signal obtained based on the greedy rule 1605 to determine the voting signal or the control signal.

[0112] More specifically, the AND operation may include: in response to the greedy rule 1605 requiring a switch to be turned off, the switch is turned off in the voting signal regardless of the control signal output by the policy network 1602; and in response to the greedy rule 1605 requiring a switch to be turned on, the control value of the switch in the control signal output by the policy network 1602 is used as the control value of the switch in the final equalization control signal.

[0113] For example, to balance power consumption according to actual needs, the greedy rule 1605 gives the channel switches. Taking module region one as an example, the 16-dimensional switch vector is [0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1]. Since adjacent channels cannot be opened at the same time, the channel indices that can be opened are [1,3,5,7,9,11,13,15]. The output signal of the neural network for region one is [1,1,1,0,1,0,1,1]. Performing an AND operation and a mask operation with the channel index, the final actual channel index is [1,3,5,9,13,15], which is converted into a 16-dimensional Boolean value of [0,1,0,1,0,1,0,0,0,1,0,0,0,1,0,1].

[0114] Finally, the equalization control signal obtained after training the model is handed over to the actuator for execution, thereby realizing battery equalization control.

[0115] The effectiveness of the battery balancing method provided by this invention will be verified below.

[0116] The following data were used as the initial conditions for the imbalance in the experiment. The values ​​represent the amount of charge that each cell needs to be balanced, in Ah. Each module has 16 cells:

[0117] [[0.245,0.21,0,0.77,0.455,0.245,0.42,0.84,0.07,0.455,0.245,0.175,0.77,0.175,0.07,1.155],

[0118] [0.175,0,0.665,1.155,0.63,0.35,0,0.77,0.14,1.05,0.245,0.56,0,0.42,0.7,0],

[0119] [0.28,0,0.315,0.245,0.49,0.455,0.105,0.455,0.35,0.665,0,0.21,0.175,0.035,0.07,0.21],

[0120] [0.105,0.455,0.42,0.42,1.015,0.63,0,0,0.385,0.805,0.56,0.315,0.21,0.7,0.42,0.595]]

[0121] Experiments based on the above data demonstrate that the balancing time of the traditional greedy strategy controller is 36,840 seconds, while the balancing time of the greedy neural network controller provided by this invention is 28,080 seconds, saving 31.20% of the time.

[0122] It should be noted that, in an optional embodiment, the greedy part of the greedy rule combined with the neural network controller in the battery balancing method provided by this invention can be discarded, and the neural network selects the best action from all possible actions. This embodiment only increases the action space and belongs to the same category as the greedy-neural network control method. For example, the output of the neural network can be directly used as a mask to apply to the balancing demand vector at the current moment, thereby obtaining the open channel vector. Then, based on whether the current temperature violates the temperature limit, and whether opening a channel violates the hardware limit for continuous channels, etc., Make adjustments to become the final action output at the current moment.

[0123] The performance comparison will be detailed below with reference to the attached figures.

[0124] Figure 17 This is a comparison diagram of the opening and closing of equalization channels during the equalization process, between an embodiment of the present invention and existing technologies. The left column shows the strategy of a traditional greedy strategy controller under a specific initial condition, while the right column shows the strategy of the greedy-neural network equalization control method provided by the present invention under the same initial condition. The four rows represent the four module areas of the PCB board. The horizontal axis in each diagram represents time, and the vertical axis represents the channel index, corresponding to 16 channels (0-15) within each module. Light colors in the diagram represent channels open (Boolean value of 1), and dark colors represent channels closed (value of 0). Ultimately, all channels of both strategies will be closed, indicating that equalization is complete. Figure 17 A comparison of the left and right columns shows that the battery balancing method provided by this invention completes the balancing work in less time, achieving higher balancing efficiency.

[0125] Figure 18This is a comparison graph showing the change in electricity demand over time during the balancing process, between an embodiment of the present invention and existing technologies. The image order and the meaning of the vertical axis are as follows: Figure 17 The horizontal axis represents the balanced demand for electricity, and the shades of color represent different demand values.

[0126] Figure 19 This is a comparison chart of temperature changes over time during the equilibrium process of an embodiment of the present invention and the prior art. The image order and the meaning of the horizontal axis are shown. Figure 18 Similarly, the vertical axis represents the temperature values ​​of the nine units and the NTC temperature sensor in each module area, and the color intensity represents the temperature level.

[0127] Figure 20 This diagram illustrates the change over time in the total number of channels opened during the equalization process, comparing an embodiment of the present invention with existing technologies. The horizontal axis represents time, and the vertical axis represents the number of channels opened. The two types of light-colored lines represent the original data of the battery equalization process in the prior art, which relies solely on greedy rules, and the equalization process provided by the present invention, which combines reinforcement learning. The dark-colored lines represent the curves after smoothing the two types of original data, facilitating observation of the changing trends.

[0128] Figure 21 This is a schematic diagram of the average temperature of the thermal field during the equilibrium process, based on an embodiment of the present invention and existing technologies. Similarly, the horizontal axis represents time, and the vertical axis represents the average temperature. The diagram includes both the original data from the existing technology and the method of the present invention, as well as smoothed curves for easy observation of trends.

[0129] from Figures 17-21 As can be seen, the battery equalization method provided by this invention can complete the equalization task with the same initial requirements much faster. Specifically, the battery equalization method provided by this invention uses 936 steps and takes a total of 28,080 seconds, while the traditional method in the prior art uses 1,228 steps and takes a total of 36,840 seconds. Therefore, the battery equalization method based on reinforcement learning provided by this invention effectively improves the equalization efficiency.

[0130] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0131] According to another aspect of the invention, an embodiment of a reinforcement learning-based battery equalization device 2200 is also provided herein.

[0132] like Figure 22As shown, the reinforcement learning-based battery equalization device 2200 provided in this embodiment may include a memory 2201 and a processor 2202 coupled to the memory 2201. The processor 2202 may be configured to implement any of the reinforcement learning-based battery equalization methods described above.

[0133] According to another aspect of the invention, an embodiment of a computer storage medium is also provided herein. This computer storage medium stores a computer program. When executed by a processor, the computer program can implement the steps of any of the reinforcement learning-based battery balancing methods described above.

[0134] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this hardware-software interchangeability, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as departing from the scope of the invention. The processors described herein can be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. As an example, the processors, any portion thereof, or any combination thereof presented in this disclosure can be implemented using microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuitry, and other suitable processing components configured to perform the various functions described throughout this disclosure. The functionality of the processor, any part of the processor, or any combination of processors presented in this disclosure can be implemented by software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[0135] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0136] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium.

[0137] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then those coaxial cables, fiber optic cables, twisted pairs, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0138] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A battery equalization method based on reinforcement learning, comprising: A physical field model of the battery control board is established as a simulation environment for battery equalization. The physical field model includes the system parameters to be identified. The experimental data of the battery control board is collected, and the system parameters in the physical field model are identified and determined based on the experimental data. as well as Based on reinforcement learning algorithms, and combined with greedy rules and preset constraints, a neural network is trained through interaction with the simulation environment. Battery equalization control is achieved through the equalization control signal output by the neural network, including: The method of training a neural network based on a reinforcement learning algorithm, combined with greedy rules and preset constraints, through interaction with the simulation environment, includes: The state data in the simulation environment is input into the neural network, and the state data includes temperature sampling signal values ​​and cell balancing demand values. In the simulation environment, action data is output based on the greedy rule and the preset constraints. The action data includes the multiplexer signal for discharging in the battery control board. The neural network outputs a voting signal to the simulation environment based on the state data and the action data. The voting signal includes a control signal from a multiplexer in the battery control board used for discharging. The total time scalar value required for the battery to reach equilibrium is negatively taken as the cumulative reward in the reinforcement learning algorithm and fed back to the neural network. The above steps are repeated to train the neural network to maximize the cumulative reward. Motor equilibrium control is then performed based on the final output equilibrium control signal.

2. The battery balancing method as described in claim 1, characterized in that, The reinforcement learning algorithm includes the PPO2 algorithm, the neural network includes a policy network and a value network, and the step of training the neural network based on the reinforcement learning algorithm, combined with greedy rules and preset constraints, through interaction with the simulation environment, further includes: The policy network receives the state data of the simulation environment and returns the current control variable, and records the result data of the interaction with the simulation environment to the sample playback; The value network supervised learning estimates the reward value in the current state, which is the negative of the time scalar value required for battery equalization control in the current state; The policy network determines the update gradient based on an advantage estimator and an entropy incentive term. The advantage estimator is used to estimate the advantage of the current state by subtracting the reward value of the value network from the actual reward of the current control variable. The entropy incentive term is used to encourage the exploration of different policies. The weights of the policy network are updated using gradient descent, and the updated control variables and state data are fed back to the policy network. This process is repeated to gradually reduce the entropy incentive term, thereby causing the policy network to converge to a deterministic policy.

3. The battery balancing method as described in claim 2, characterized in that, The method of training a neural network based on a reinforcement learning algorithm, combined with greedy rules and preset constraints, through interaction with the simulation environment, further includes: A bitwise AND operation is performed on the control signal of the deterministic policy finally output by the policy network and the multiplexer signal obtained based on the greedy rule to determine the voting signal. The bitwise AND operation includes: In response to the greedy rule requiring a switch to be turned off, regardless of the control signal output by the policy network, the switch is turned off in the voting signal; and In response to the greedy rule requiring a certain switch to be turned on, the control value of that switch in the control signal output by the policy network is used as the control value of that switch in the final equalization control signal.

4. The battery balancing method as described in claim 1, characterized in that, The greedy rule includes the calculation of the remaining power of the battery cell, and the preset constraints include the temperature constraints of the PCB thermal model in the battery control board.

5. The battery balancing method as described in claim 4, characterized in that, The battery is composed of multiple cells connected in series, and each cell corresponds to a discharge channel controlled by a switch. The preset constraints also include a hardware constraint that ensures that the switches of adjacent discharge channels cannot be turned on at the same time.

6. The battery balancing method as described in claim 1, characterized in that, The establishment of a physical field model for the battery control board as a simulation environment for battery equalization includes: The battery control board is modeled as a differential algebraic equation model, and the PCB board is abstracted as multiple modular units, assuming that the parameters of these multiple units are isotropic; and A constant heat source is added at a predetermined location to simulate the heating of electronic components on the battery control board, excluding the discharge resistor.

7. The battery balancing method as described in claim 6, characterized in that, The establishment of a physical field model for the battery control board as a simulation environment for battery equalization also includes: The physical field model was built using Simulink / Simscape, where the outer Simulink solver of the model used a fixed-step discrete solver, and the inner Simscape solver of the model used an inverse Euler solver.

8. The battery balancing method as described in claim 1, characterized in that, The process of identifying and determining the system parameters in the physical field model based on the experimental data includes: The experimental data were fitted using a combination of white-box calibration and a search algorithm to determine the values ​​of the system parameters.

9. The battery balancing method as described in claim 8, characterized in that, The system parameters include thermal capacity coefficient, thermal conductivity coefficient, thermal convection coefficient, the length of the battery control board in all directions, external heat generation, mass density, and resistivity heat generation. The experimental data includes experimental data of the battery control board in a constant temperature chamber. The method of fitting the experimental data using white-box calibration combined with a search algorithm to determine the values ​​of the system parameters includes: Except for specific parameters, other system parameters are entered into the physical field model through manual measurement. The specific parameters are a subset of the system parameters, including the heat capacity coefficient, the thermal conductivity coefficient, the thermal convection coefficient, the external heat generation, the mass density, and the length of the battery control board in a specific direction. For the specific parameter, the approximate range of the parameter value is first given by the material properties, and then the parameter value that is closest to the experimental data is searched in the parameter space by a genetic algorithm. The specific direction is the direction in which the heat distribution on the battery control board is uncertain.

10. The battery balancing method as described in claim 9, characterized in that, The experimental data also includes experimental data of the battery control board on a real vehicle, and the process of identifying and determining the system parameters in the physical field model based on the experimental data further includes: A model of the battery control board housing on the actual vehicle is established. The parameters of the housing model are selected and determined together with the system parameters by fitting experimental data through white-box calibration and search algorithm.

11. The battery balancing method as described in claim 1, characterized in that, The battery is composed of a group of cells connected in series, and the group of cells is divided into b modules with the same structure. Each module has a corresponding cell. There are several discharge channels, including a, b, and All are positive integers. When a discharge channel is opened, the corresponding cell discharges through the resistor connected in parallel within the discharge channel. Temperature sensors are provided on both the front and back sides of the battery control board at the center of each module.

12. The battery balancing method as described in claim 11, characterized in that, The status data includes The 3D temperature vector and the 3D equilibrium demand vector, the The dimensional temperature vector includes b numerical values ​​from the temperature sensors in the module and c sets of temperature sampling signal values. Each equalization demand vector corresponds to an equalization demand value for a battery cell, and the unit of the equalization demand value is Ah. The action data includes Boolean values ​​of a-dimensional switching signals, representing the switching of discharge channels corresponding one-to-one with a groups of cells. The voting signals include The switch control signals satisfy the hardware constraint that ensures the switches of adjacent discharge channels cannot be opened simultaneously, where a, b, c, and All are positive integers.

13. A battery equalization device based on reinforcement learning, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the reinforcement learning-based battery balancing method as described in any one of claims 1 to 12.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the battery balancing method based on reinforcement learning as described in any one of claims 1 to 12.

Images