Power battery model optimization algorithm based on reinforcement learning
By using a reinforcement learning-based power battery model optimization algorithm, and leveraging a multi-agent system and physical constraint projection, the multi-objective dynamic optimization problem of power batteries under complex operating conditions was solved, achieving stable and reliable battery management and improving battery performance and lifespan.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA AUTOMOTIVE ENG RES INST
- Filing Date
- 2026-01-15
- Publication Date
- 2026-04-28
AI Technical Summary
Existing power battery management and optimization strategies are difficult to adapt to dynamically changing operating conditions and battery aging states. They lack adaptive adjustment capabilities, cannot dynamically balance multiple objectives, and their optimization perspective is usually short-term, lacking the ability to optimize long-term cumulative effects, and are subject to safety constraints.
A power battery model optimization algorithm based on reinforcement learning is adopted. By constructing a high-precision simulation environment and a multi-agent system, cooperative and independent reward functions are designed, and physical constraint projection is combined to achieve multi-objective dynamic optimization of the power battery.
It achieves stable and reliable multi-objective optimization under complex operating conditions, ensuring that the strategy has both multi-objective balancing ability and physical feasibility, improving battery operating performance and lifespan utilization, and possessing adaptive characteristics and real-time decision-making capabilities.
Smart Images
Figure CN121936047A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of new energy vehicle technology, specifically to a power battery model optimization algorithm based on reinforcement learning. Background Technology
[0002] As a core component of electric vehicles, the performance of the power battery directly affects the vehicle's range, lifespan, and safety. In practical applications, optimizing the battery system requires balancing multiple objectives such as lifespan, safety, and efficiency. Furthermore, the operating environment is subject to uncertainties such as fluctuations in operating conditions and temperature changes, making it a typical multi-objective, strongly constrained dynamic decision-making problem.
[0003] Currently, the management and optimization strategies for power batteries mainly rely on two traditional technologies: rule-based methods and model predictive control (MPC). (1) Rule-based methods, such as setting fixed charging and discharging current thresholds and temperature-triggered cooling, are simple in logic and reliable in execution, and were widely used in early battery management systems. However, these methods are essentially static and experience-based strategies, which are difficult to adapt to dynamically changing operating conditions and battery aging states, and cannot actively make dynamic trade-offs among multiple objectives, thus limiting their optimization potential. (2) Model predictive control establishes a mathematical model of the battery and solves an optimization problem in a finite time domain in each control cycle. It can handle multiple constraints and predict the short-term state in the future to a certain extent, and its performance is better than fixed rules. However, its effectiveness depends heavily on the prediction accuracy of the model, and the battery, especially its aging dynamics, is a highly nonlinear and time-varying system that is difficult to describe accurately with a simple model.
[0004] More importantly, both rule-based strategies and MPC typically have a short-term optimization perspective and lack the ability to directly optimize long-term cumulative effects (such as cyclic aging and calendar aging). When faced with multiple conflicting long-term and short-term goals such as lifetime, safety, and efficiency, these methods often compromise through pre-set weights, lacking adaptive adjustment capabilities. Furthermore, under strict safety constraints (such as the absolute boundaries of voltage, temperature, and SOC), the feasible domain of their optimization actions is limited, and they may fall into suboptimal solutions or fail to guarantee real-time safety under complex boundary conditions.
[0005] Therefore, there is an urgent need to propose a new power battery optimization algorithm to overcome the shortcomings of existing technologies. Summary of the Invention
[0006] The present invention aims to provide a power battery model optimization algorithm based on reinforcement learning, which can realize multi-objective dynamic optimization of power batteries under complex working conditions and achieve stable and reliable optimization results.
[0007] The basic solution provided by this invention is: a power battery model optimization algorithm based on reinforcement learning, comprising the following steps: S1. Construct a high-precision simulation environment for the power battery system, wherein the simulation environment integrates an equivalent circuit model, a thermodynamic model, and a battery health state degradation model; S2. Based on the simulation environment, establish a first agent and a second agent corresponding to the battery power allocation decision and the battery thermal management decision, respectively, and define an independent observation space and a continuous action space for each agent. S3. Design a reward function that integrates cooperative and independent objectives. The cooperative objectives include power tracking accuracy and state of charge stability, while the independent objectives include minimizing energy loss, suppressing thermal stress, and minimizing capacity decay rate. S4. At each decision moment, obtain the current state of the simulation environment and input it into the first agent and the second agent respectively, and output the original action instructions by their respective policy networks. S5. Through the constraint projection module, the original action command is mapped to a physical action domain that meets the preset safe range of state of charge, safe range of temperature and charge / discharge rate limits, so as to obtain the corrected executable action. S6. Use the executable action to drive the simulation environment to the next state, and calculate the reward obtained by each agent according to the reward function; S7. Using the empirical data consisting of the current state, the executable action, the reward, and the next state, the policy network and value network parameters of the first agent and the second agent are updated in parallel using a reinforcement learning algorithm. S8. Repeat S4 to S7 until the cumulative reward of the agent converges, and obtain the trained power allocation strategy model and thermal management strategy model.
[0008] The working principle and advantages of this invention are as follows: This invention presents a reinforcement learning-based optimization algorithm for power battery models. This algorithm enables multi-objective dynamic optimization of power batteries under complex operating conditions, achieving stable and reliable optimization results. It ensures that the optimized strategy possesses both multi-objective balancing capabilities and physical feasibility. The key points are: First, reliable multi-objective optimization is achieved through the division of labor and collaborative control among multiple agents. This scheme specifically decomposes the power battery optimization task into two sub-tasks: power allocation and thermal management, which are executed by independent agents respectively. Through a collaborative-independent reward mechanism, a dynamic balance is achieved among multiple objectives, including power point tracking, SOC stability, energy efficiency improvement, and lifespan extension. This allows for automatic adjustment of the strategy focus under different operating conditions, simultaneously improving convergence speed and control accuracy.
[0009] Secondly, the implementation of reinforcement learning is ensured through physical constraint embedding and a constraint projection safety layer. In reinforcement learning, rewards and penalties are often used to constrain behavior, but this is a "soft constraint," and the agent may still risk exceeding the limits in pursuit of higher scores. This solution introduces physical constraints such as the SOC safety range, temperature threshold, rate limit, and SOH decay curve during policy training and execution. Furthermore, it corrects actions before execution using least-squares projection, ensuring the physical feasibility and safe operation of control commands. This avoids the risks of overcharging, over-discharging, and thermal runaway, enabling data-driven reinforcement learning to be safely applied to battery systems with high reliability requirements.
[0010] Third, the reliability and adaptability of the optimization strategy are improved through high-precision simulation environment modeling. This solution establishes a high-precision simulation environment covering SOC dynamic equations, heat conduction models, and degradation models based on Ah throughput and Arrhenius equations. This environment can support the strategy to maintain efficient energy conversion and low capacity decay under multiple operating conditions, achieving simultaneous improvement in operating performance and lifetime utilization.
[0011] Fourth, this scheme (S4-S8) can form a complete perception-decision-learning closed loop. The agent continuously interacts with the environment, using empirical data to update the network in parallel (S7), constantly evolving the policy. This data-driven approach enables the final trained policy model (S8) to adapt to various complex and dynamic conditions covered by the training data, possessing strong generalization ability and adaptive characteristics. More importantly, the converged policy network is a lightweight neural network that can be directly deployed in the embedded hardware of the vehicle's BMS, achieving millisecond-level online real-time decision-making. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of the algorithm flow of an embodiment of a power battery model optimization algorithm based on reinforcement learning according to the present invention. Detailed Implementation
[0013] The following detailed explanation illustrates the specific implementation methods: The basic implementation examples are as follows: Figure 1 The following is an example of a power battery model optimization algorithm based on reinforcement learning, comprising the following steps: S1. Construct a high-precision simulation environment for the power battery system, wherein the simulation environment integrates an equivalent circuit model, a thermodynamic model, and a battery health state degradation model.
[0014] Specifically, the equivalent circuit model uses a first-order RC model to describe the voltage-current transient characteristics of the battery. This includes: (1) State update: The state of charge (SOC) of the battery is updated according to the ampere-hour integration method: ; in, For the battery in time step The charging and discharging current (discharging is positive, charging is negative). The sampling period is This is the rated capacity.
[0015] (2) Calculation of terminal voltage: ; Among them, OCV ( ) is the open-circuit voltage-SOC relationship function, specifically referring to the open-circuit voltage corresponding to SOC. This represents the battery's internal resistance.
[0016] The thermodynamic model is a single-degree-of-freedom thermal model, whose equation of state describes the change of battery temperature over time. Its inputs include the heat power generated by the battery's internal resistance loss and the cooling power provided by the cooling system; its equations are as follows: ; in, For battery heat capacity; Battery temperature; The ambient temperature; Let t be the cooling power of the cooling system at time t; , where is the Joule heat generation power; h is the convective heat transfer coefficient.
[0017] The battery state-of-health (SOH) degradation model is used to quantify the real-time and cumulative effects of different control strategies on battery life. Its output serves as part of the reward function to guide strategy optimization. The core of this model lies in combining cyclic aging effects with acceleration effects to calculate the real-time rate of state-of-health decay.
[0018] Specifically, the battery health state degradation model includes a cycle degradation component based on ampere-hour throughput and a temperature acceleration factor based on the Arrhenius equation. The temperature acceleration factor is used to weight the cycle degradation component according to the battery temperature to calculate the total capacity degradation rate; its model is specifically expressed as follows: (1) Cyclic degradation component based on ampere-hour throughput: This component characterizes the instantaneous capacity degradation rate of the battery due to charge and discharge current when temperature is not considered. Its model is based on the ampere-hour throughput theory: ; in, This refers to the battery's rated cycle life. This is the rated capacity. This formula indicates that the charging and discharging current... The larger the absolute value, the faster the instantaneous degradation rate of the battery's health status.
[0019] (2) Temperature acceleration factor based on the Arrhenius equation: This factor, based on the Arrhenius equation, is used to quantify the accelerating effect of battery temperature on the aging rate. ; in, Pre-exponential factors; The activation energy, which reflects the ease with which battery materials undergo aging reactions, is given by R, a gas constant. This refers to the absolute temperature of the battery.
[0020] (3) Real-time health status decay rate: temperature acceleration factor As weights applied to the basic cyclic decay component, a real-time decay rate model that comprehensively considers current and temperature can be obtained: ; This instantaneous rate is the penalty term in the reward function. The basis for the calculation.
[0021] (4) Cumulative capacity decay: By integrating the above real-time decay rate over time, the total capacity decay from the initial moment to the current moment can be obtained, which is used to evaluate the long-term lifetime performance of the strategy offline. ; In discrete-time step simulations, each step... Changes in health status within the body Calculated by the following formula: .
[0022] S1.1, Set operational constraints. Specifically, this includes SOC constraints: For example, setting , This is to protect the battery from overcharging and over-discharging. Temperature constraints: For example, setting , To prevent lithium plating during low-temperature charging and the risk of thermal runaway during high-temperature charging. Rate constraint conditions: Not exceeding the instantaneous battery rate limit.
[0023] S2. Based on the simulation environment, establish a first intelligent agent corresponding to the battery power allocation decision and the battery thermal management decision, respectively. Second intelligent agent Each agent is given an independent observation space and a continuous action space.
[0024] Specifically, the observation space of the first intelligent agent includes at least the current state of charge, battery temperature, load power demand, and historical charge / discharge current, and its action space is the charge / discharge current command; the observation space of the second intelligent agent includes at least the current battery temperature, state of charge, and historical cooling power, and its action space is the cooling system power command.
[0025] Furthermore, action : ; This represents the minimum value of the charging and discharging current. This represents the maximum value of the charging and discharging current.
[0026] action : ; 0 represents the minimum power of the cooling system (i.e., cooling is not activated); This represents the maximum power output allowed by the cooling system (upper limit of cooling capacity).
[0027] S3. Design a reward function that integrates cooperative and independent objectives. The cooperative objectives include power tracking accuracy and state of charge stability, while the independent objectives include minimizing energy loss, suppressing thermal stress, and minimizing capacity decay rate.
[0028] The reward function combines collaborative rewards and independent rewards through a weighted sum. The total reward for the first intelligent agent is: ; The total reward for the second agent is: ; in, For the aforementioned collaboration reward, This is a dynamically adjustable collaboration weight coefficient.
[0029] ;in, These are the power tracking weighting coefficients; Power required by the load; This represents the actual output power of the battery. For SOC tracking weight coefficients; Let t be the state of charge of the battery at time t; This is the SOC reference value.
[0030] and The independent rewards for the first agent and the second agent are as follows: ;in, This is the battery loss weighting coefficient; Let t be the internal resistance of the battery. Let t be the charging / discharging current at time t; The safe current threshold; For current safety weighting coefficient; for Only when the current exceeds the safe threshold ( A penalty is only incurred when the limit is exceeded; otherwise, this part is 0.
[0031] ;in, This is the SOH attenuation weighting coefficient; The change in battery health status at time t (the degree of degradation of SOH at the current time). The larger the value, the faster the battery ages. This is the cooling power weighting coefficient; This refers to the power of the cooling system.
[0032] S4. At each decision moment, obtain the current state of the simulation environment and input it into the first agent and the second agent respectively, and output the original action instructions by their respective policy networks.
[0033] S5. Through the constraint projection module, the original action command is mapped to a physical action domain that meets the preset safe range of state of charge, safe range of temperature and charge / discharge rate limits, so as to obtain the corrected executable action.
[0034] The constrained projection module achieves action correction by solving a constrained optimization problem. Its objective function is to minimize the difference between the corrected action and the original action command. The constraints are the physical feasible region defined by the state of charge safety range, temperature safety range and charge / discharge rate limit.
[0035] Right now: ; in, The original output action of the intelligent agent. This is the revised action. This is the physically feasible region.
[0036] S6. Use the executable action to drive the simulation environment to the next state, and calculate the reward obtained by each agent according to the reward function.
[0037] Each agent maintains a policy network, two value networks, and a corresponding target value network, and encourages policy exploration through entropy regularization.
[0038] Initialize battery simulation environment parameters Initialize the policy network of the two agents. and the corresponding value network Initialize the experience replay buffer At each time step : Get the current state Construct the observation vectors for the two agents respectively: ; The two agents sample actions according to the policy network: ; Combine actions into a combined action. After constraint projection correction: ; Environmental correction actions Return to the next state Collaboration rewards Two independent awards and the termination mark ; Calculate the total reward, deposit ,Will deposit .
[0039] S7. Using the empirical data consisting of the current state, the executable action, the reward, and the next state, the policy network and value network parameters of the first agent and the second agent are updated in parallel using a reinforcement learning algorithm.
[0040] The reinforcement learning algorithm is a soft actor-critic algorithm. When updating the agent's network parameters, batch data sampled from the experience replay buffer is used. The value network parameters are updated by minimizing the temporal difference error of the value network, and the policy network parameters are updated by maximizing the weighted sum of the value function estimate and the policy entropy.
[0041] The specific update process is as follows: S7.1: Batch data sampling.
[0042] From the experience replay buffer N empirical data points are randomly sampled to form a batch training dataset.
[0043] S7.2: Calculation of the target Q value.
[0044] For each sample j, calculate the target Q value. : ; in, .
[0045] S7.3: Value network parameter update.
[0046] Update the value network parameters by minimizing the temporal difference error. : ; in, The value network learning rate.
[0047] S7.4: Policy network parameter update.
[0048] The policy network parameters are updated by maximizing the weighted sum of the value function estimate and the policy entropy. : ; in, is the learning rate of the policy network.
[0049] S7.5: Adaptive adjustment of temperature parameters.
[0050] Temperature parameters It can be adaptively adjusted in the following ways: ; in, The learning rate for the temperature parameter, Let be the target entropy.
[0051] S7.6: Target network soft update.
[0052] The target value network parameters are slowly tracked through online network parameters using a soft update method. ; where τ is the soft update coefficient.
[0053] The above update steps apply to the first intelligent agent. Second intelligent agent Each is executed independently.
[0054] in, Identifying intelligent agents — , Represents the first intelligent agent. Represents the second agent; j is the sample index. , indicating the first in the batch of data One sample; k is the value network sequence number — This represents two value networks maintained by each agent; the empirical data is: for each agent Experience replay buffer Data tuples stored in .
[0055] S8. Repeat S4 to S7 until the cumulative reward of the agent converges, and obtain the trained power allocation strategy model and thermal management strategy model.
[0056] During training, the performance metrics of the current strategy under test conditions are periodically evaluated. These metrics include average energy efficiency, state of charge (SOC) fluctuation range (standard deviation of SOC fluctuation), battery temperature over-limit time ratio, and simulated capacity decay. The weighting ratio of cooperative rewards to independent rewards in the reward function is dynamically adjusted based on the evaluation results. For example, after every N rounds of training (e.g., N=50), the current strategy is run on a reserved set of standard test conditions (such as WLTC and UDDS loops), and the performance metrics are calculated. If the battery temperature over-limit time ratio is found to be too high, the weight of the high-temperature penalty term in the thermal management independent reward is appropriately increased; if the SOC fluctuation range is too large, the weight of the SOC penalty term in the cooperative reward is increased.
[0057] The trained power allocation strategy model and thermal management strategy model will be compiled into executable code and deployed to the battery management system (BMS) of the electric vehicle for online real-time generation of control commands.
[0058] During online operation, the BMS collects the observations defined by S4 every second and inputs them into the deployed policy network. The network outputs normalized actions, which are then processed by the constraint projection logic of S8 (at this time, it is a simple feedforward calculation or table lookup) to generate the final safe current command and cooling power command, which are sent to the motor controller and thermal management system for execution, respectively.
[0059] This embodiment provides a power battery model optimization algorithm based on reinforcement learning, which can realize multi-objective dynamic optimization of power batteries under complex working conditions and achieve stable and reliable optimization results, ensuring that the optimized strategy has both multi-objective balancing ability and physical feasibility.
[0060] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.
Claims
1. A power battery model optimization algorithm based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct a high-precision simulation environment for the power battery system, wherein the simulation environment integrates an equivalent circuit model, a thermodynamic model, and a battery health state degradation model; S2. Based on the simulation environment, establish a first agent and a second agent corresponding to the battery power allocation decision and the battery thermal management decision, respectively, and define an independent observation space and a continuous action space for each agent. S3. Design a reward function that integrates cooperative and independent objectives. The cooperative objectives include power tracking accuracy and state of charge stability, while the independent objectives include minimizing energy loss, suppressing thermal stress, and minimizing capacity decay rate. S4. At each decision moment, obtain the current state of the simulation environment and input it into the first agent and the second agent respectively, and output the original action instructions by their respective policy networks. S5. Through the constraint projection module, the original action command is mapped to a physical action domain that meets the preset safe range of state of charge, safe range of temperature and charge / discharge rate limit, so as to obtain the corrected executable action. S6. Use the executable action to drive the simulation environment to the next state, and calculate the reward obtained by each agent according to the reward function; S7. Using the empirical data consisting of the current state, the executable action, the reward, and the next state, the policy network and value network parameters of the first agent and the second agent are updated in parallel using a reinforcement learning algorithm. S8. Repeat S4 to S7 until the cumulative reward of the agent converges, and obtain the trained power allocation strategy model and thermal management strategy model.
2. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The reward function combines collaborative rewards and independent rewards through a weighted sum. The total reward for the first intelligent agent is: ; The total reward for the second agent is: ; in, For the collaboration reward, and These are the independent rewards for the first and second intelligent agents, respectively. This is a dynamically adjustable collaboration weight coefficient.
3. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The constrained projection module achieves action correction by solving a constrained optimization problem. Its objective function is to minimize the difference between the corrected action and the original action command. The constraints are the physical feasible region defined by the state of charge safety range, temperature safety range and charge / discharge rate limit.
4. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The battery health state degradation model includes a cycle decay component based on ampere-hour throughput and a temperature acceleration factor based on the Arrhenius equation. The temperature acceleration factor is used to weight the cycle decay component according to the battery temperature to calculate the total capacity decay rate.
5. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm is the soft actor-critic algorithm, in which each agent maintains a policy network, two value networks, and a corresponding target value network, and encourages policy exploration through entropy regularization.
6. The power battery model optimization algorithm based on reinforcement learning according to claim 5, characterized in that, When updating the network parameters of the agent, batch data sampled from the experience replay buffer is used to update the value network parameters by minimizing the temporal difference error of the value network, and the policy network parameters are updated by maximizing the weighted sum of the value function estimate and the policy entropy.
7. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, During training, the performance metrics of the current strategy under test conditions are periodically evaluated. These performance metrics include average energy efficiency, state of charge fluctuation range, battery temperature over-limit time ratio, and simulated capacity decay. The weight ratio of cooperative rewards and independent rewards in the reward function is dynamically adjusted based on the evaluation results.
8. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The observation space of the first intelligent agent includes at least the current state of charge, battery temperature, load power demand, and historical charge / discharge current, and its action space is the charge / discharge current command; the observation space of the second intelligent agent includes at least the current battery temperature, state of charge, and historical cooling power, and its action space is the cooling system power command.
9. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The thermodynamic model is a single-degree-of-freedom thermal model, whose state equation describes the change of battery temperature over time. Its inputs include the heat power generated by the internal resistance loss of the battery and the cooling power provided by the cooling system.
10. The power battery model optimization algorithm based on reinforcement learning according to claim 1, characterized in that, The trained power allocation strategy model and thermal management strategy model will be compiled into executable code and deployed to the battery management system of the electric vehicle for online real-time generation of control commands.