A method and system for enhancing the sample efficiency of lithium battery charging strategies based on model short-branch inference

Abstract

This invention pertains to lithium battery charging technology and proposes a method and system for enhancing the sample efficiency of lithium battery charging strategies based on short-branch model deduction. The method includes the following steps: conducting charge-discharge tests on the battery based on the actions taken by a reinforcement learning agent, and sampling charging state transition data; training a battery proxy model using the sampled charging state transition data; performing short-branch deduction using the battery proxy model based on the actual charging state trajectory, obtaining the deduced branch trajectory data of the charging state transition; combining the actual charging state trajectory data with the deduced branch trajectory data to train the agent, repeating the above steps until the reinforcement learning training converges, and obtaining a reinforcement learning agent capable of executing the optimal charging strategy. This invention constructs a battery proxy model and utilizes the model's branch deduction, effectively improving the sample efficiency of charging strategy optimization and alleviating the error accumulation problem caused by the recursive use of the model in traditional optimization methods.

Description

Technical Field

[0001] This invention pertains to lithium battery charging technology, specifically relating to a method and system for enhancing the sample efficiency of lithium battery charging strategies based on model short-branch deduction. Background Technology

[0002] With the increasing popularity of electric vehicles, lithium-ion batteries have become the mainstream choice due to their high energy density, long cycle life, and low maintenance costs. However, the disadvantage of electric vehicles in charging speed, especially compared to traditional gasoline vehicles, seriously affects the user experience and limits further market expansion. Although increasing the charging current can speed up charging, high current can trigger a series of adverse electrochemical reactions, leading to battery degradation, decreased charging efficiency, and even potential safety accidents. Therefore, how to improve charging speed while ensuring battery safety and durability has become an urgent problem to be solved.

[0003] Recent research has made some progress in optimizing fast charging strategies. Traditional methods, mostly based on pre-defined rules, are simple and easy to implement, but they cannot fully account for the dynamic changes inside the battery, making it difficult to meet the requirements of efficient charging. Optimization methods based on battery reaction mechanisms, such as equivalent circuit models and electrochemical models, can more accurately simulate battery behavior, improving charging efficiency and extending battery life. However, these methods often rely on high-precision models, which limits their reliability and efficiency in practical applications.

[0004] In recent years, reinforcement learning methods have attracted widespread attention as an emerging optimization tool. Through autonomous learning and optimization, reinforcement learning can improve the performance of charging strategies without relying on battery models. However, its low sample efficiency in practical applications leads to high experimental costs. To address this issue, some model-based reinforcement learning methods introduce environment models to improve sample efficiency; however, the recursive use of these models can lead to error accumulation, thus affecting the optimization accuracy of the strategy. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a method and system for enhancing the sample efficiency of lithium battery charging strategies based on model short branch inference. It adopts a charging strategy optimization framework that combines reinforcement learning and model branch inference, which can directly optimize strategies on real batteries, avoiding the complex and tedious modeling process. At the same time, it effectively improves sample efficiency by making limited use of model inference data and alleviates the negative impact of model error accumulation.

[0006] On one hand, embodiments of the present invention provide a method for enhancing the sample efficiency of lithium battery charging strategies based on model short-branch inference, comprising the following steps:

[0007] S1. Based on the actions taken by the reinforcement learning agent, charge and discharge tests are performed on the battery, and charging state transition data is sampled.

[0008] The process of sampling charging state transition data is as follows: the initial charging state parameters are obtained and transmitted to the agent; the agent takes corresponding actions based on the current battery state, and at the same time obtains the corresponding state of the battery at the next moment; the action is to adjust the charging current of the battery.

[0009] S2. Train the battery proxy model using the sampled charging state transition data; use each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model to train the model, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery.

[0010] S3. Based on the actual charging state trajectory, a short branch inference is performed using a battery proxy model. That is, starting from a certain charging state in the actual sampling trajectory, an action different from the next action of the actual charging state trajectory is taken, and the next charging state of the battery is obtained by using the prediction of the battery proxy model. In other words, the sampling process of the battery is simulated to obtain the branch trajectory data of the inferred charging state transition.

[0011] S4. Combine the real charging state trajectory data with the inferred branch trajectory data to train the agent; and repeat the above steps S1-S3 until the reinforcement learning training converges to obtain a reinforcement learning agent that can execute the optimal charging strategy.

[0012] Preferably, the agent includes a policy network and a value network. The value network is used to calculate the Q value. The goal of the policy network is to find a strategy that maximizes the weighted sum of the expected cumulative reward and the policy entropy based on the Q value calculated by the value network, the current policy, the charging current, and the charging state of the battery.

[0013] On the other hand, embodiments of the present invention also provide a lithium battery charging strategy sample efficiency enhancement system based on model short branch inference, which is implemented based on the above-mentioned lithium battery charging strategy sample efficiency enhancement method, and the system includes the following modules:

[0014] The sampling module performs charge and discharge tests on the battery based on the actions taken by the reinforcement learning agent, and samples the charging state transition data. The process of sampling the charging state transition data is as follows: the initial charging state parameters are obtained and transmitted to the agent; the agent takes corresponding actions according to the current battery state, and at the same time obtains the corresponding state of the battery at the next moment; the action is to adjust the charging current of the battery.

[0015] The model training module uses the sampled charging state transition data to train the battery proxy model; it uses each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model for model training, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery.

[0016] The extrapolation module, based on the actual charging state trajectory, uses a battery proxy model to perform short-branch extrapolation. That is, taking a certain charging state in the actual sampling trajectory as the starting point, it takes an action different from the next action of the actual charging state trajectory, and uses the prediction of the battery proxy model to obtain the next charging state of the battery. In other words, it simulates the sampling process of the battery and obtains the extrapolated branch trajectory data of the charging state transition.

[0017] The agent training module combines real charging state trajectory data with inferred branch trajectory data to train the agent; and repeats the sampling of charging state transition data, training and inference of the battery agent model until the reinforcement learning training converges, thereby obtaining a reinforcement learning agent that can execute the optimal charging strategy.

[0018] Compared with the prior art, the beneficial effects achieved by the present invention include:

[0019] This invention proposes a lithium battery charging strategy optimization framework based on reinforcement learning and model branch inference. The model learns the battery's charging state transitions through sampled data from the agent and generates additional samples using short-branch inference, thereby accelerating the optimization process of the charging strategy. This invention significantly outperforms model-free reinforcement learning methods in terms of sample efficiency, finding solutions with high reward values ​​in fewer training rounds. Limited use of model inference data aids agent training and effectively solves the problem of decreased convergence accuracy caused by error accumulation in traditional model-based methods. Attached Figure Description

[0020] Figure 1 This is a flowchart of the lithium battery charging strategy sample efficiency enhancement method based on model short branch inference in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the overall framework of the lithium battery charging strategy sample efficiency enhancement system based on model short branch inference in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the process of using a model for prediction in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of model branch deduction in an embodiment of the present invention;

[0024] Figure 5The graphs show the changes in the main parameters of the strategy optimization method and the model-free strategy optimization method adopted in the embodiments of the present invention during the training process. Among them, (a) is the graph of the change in the Return value, (b) is the graph of the change in the charging time, (c) is the graph of the change in the violation of the negative electrode potential constraint, and (d) is the graph of the change in the violation of the temperature constraint.

[0025] Figure 6 The graphs show the optimal charging strategy found by the strategy optimization method and the model-free strategy optimization method adopted in the embodiments of the present invention during the initial training period (first 100 training rounds) and the end of training period (1000 training rounds). Among them, (a) is the curve of charging current change, (b) is the curve of SOC change, (c) is the curve of negative electrode potential change, and (d) is the curve of temperature change. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the embodiments and accompanying drawings. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The directional terms such as left, center, right, top, and bottom in the embodiments of the present invention are only relative concepts or referenced to the normal use state of the product, and should not be considered as limiting.

[0027] An embodiment of the present invention provides a method for enhancing the sample efficiency of lithium battery charging strategies based on model short-branch inference, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0028] Step 1: Based on the actions taken by the reinforcement learning agent, perform charge and discharge tests on the battery and sample the charging state transition data.

[0029] In this embodiment, the battery is discharged until the state of charge (SOC) is 0. Then, the battery is left to stand at room temperature (25°C) until the battery temperature matches the ambient temperature before subsequent experiments are conducted.

[0030] The sampling process for charging state transition data is as follows: the initial charging state parameters are obtained and transmitted to the agent, which includes a policy network and a value network; the agent takes corresponding actions based on the current battery state (e.g., adjusting the battery charging current and setting the charging current range to 1C to 4C), and at the same time obtains the corresponding state of the battery at the next moment.

[0031] The acquired charging state transition data will be stored in the real trajectory experience pool, which is constructed using a queue structure to ensure that the sampled data completes the update process.

[0032] In an agent, the value network is used to calculate the Q-value; the goal of the policy network is to find a policy that maximizes the weighted sum of the expected cumulative reward and the policy entropy, based on the Q-value calculated by the value network, the available policies, the charging current, and the battery's state of charge.

[0033]

[0034] In the formula, π * Let r(s) represent the optimal strategy, π represent the alternative strategies, α represent the weighting factor, and γ represent the reward discount factor; t ,a t The reward for the current time step is denoted as r. t E represents the expected value, which is the Q value estimated by the value network. Represents entropy; a t This represents the charging current at the current time step, and in this embodiment, its value ranges from 1C to 4C; s t The charging state of the battery at the current time step can be specifically represented as s. t =(SoC t V t ,T t ,η t ), where SoC t For the battery's current state of charge at the current time step, V t T represents the battery voltage at the current time step. t η is the battery temperature at the current time step. t This represents the negative electrode potential at the current time step of the battery.

[0035] To reduce the bias in Q-value estimation, this embodiment uses two independent Q-networks for the value network, namely the first Q-network. With the second Q network The value network approximates the state-action value function. To improve training stability, two independent objective Q-networks are also introduced into the value network: the first objective Q-network. With the second target Q network To calculate the target Q value.

[0036] The calculated result of the objective Q-value is the expected return (i.e., the expected return) in the policy network objective. The formula for calculating the objective Q-value is:

[0037]

[0038] In the formula, y t Indicates the target Q value. a represents the expected computation starting at the next time step. t+1 Indicates the charging current at the next time step, st+1 This indicates the charging status at the next time step. Entropy, or entropy, is a metric used to measure the randomness of a policy, and is defined as follows:

[0039]

[0040] This represents the expected value under the current policy. The introduction of entropy encourages the policy to maintain greater randomness in action selection, thereby enhancing the policy's exploratory ability; the weighting factor α is a regularization parameter used to balance the weights between reward and entropy.

[0041] Step 2: Train the battery proxy model using the sampled charging state transition data; use each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model for model training, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery.

[0042] In this embodiment, the battery proxy model includes a multilayer perceptron, with the battery's charging state as input and the next charging state as output. The training dataset for the battery proxy model originates from the charging state transition data obtained in step 1. In this embodiment, the time step interval for training the model is set to 1000, meaning that after every 1000 samplings of charging state transition data, the battery proxy model is trained once based on all data in the current real sampled data experience pool (i.e., the real trajectory experience pool). Throughout the reinforcement learning cycle, the battery proxy model will continue to train to maintain adaptability to the distribution transitions of the sampled data. The battery proxy model is updated according to the following formula:

[0043]

[0044] Where, ω k+1 ω represents the updated battery proxy model parameters. k This represents the current battery proxy model parameters. This means that the loss function of the battery proxy model is defined as the error between the true next charging state and the predicted next charging state based on the current charging state and the action, f. ω (s t ,a t ) represents the predicted next state of charge; This indicates the calculation of the parameter gradient; D represents the expected value calculated on the experience pool of real trajectories; real η represents the experience pool of real trajectories; η represents the learning rate.

[0045] Step 3: Based on the actual charging state trajectory, perform short-branch extrapolation using a battery surrogate model. Starting from a specific charging state in the actual sampling trajectory, take an action different from the next action in the actual charging state trajectory. Utilize the prediction of the battery surrogate model to obtain the next charging state of the battery, thus simulating the battery sampling process and obtaining the extrapolated branch trajectory data for charging state transitions. Store the extrapolated branch trajectory data in the extrapolated branch trajectory data experience pool.

[0046] In this embodiment, a subset of charging states is randomly selected from the real sampled trajectory data experience pool (i.e., the real trajectory experience pool), and a short branch inference is performed using the trained state transition model (i.e., the battery proxy model). The time step interval of the battery proxy model inference is also set to 1000, that is, the battery proxy model inference is performed after each training of the battery proxy model.

[0047] Specifically, during the deduction process, the battery proxy model is given the current charging state s. t And the next predicted state is generated using the battery agent model based on the action selected by the agent according to the current policy:

[0048]

[0049] in, This indicates a new action generated by the agent based on the current policy, wherein the new action is related to the charging current a. t Different charging currents; This indicates the next charging state predicted based on the current charging state and the action taken under the new action. This indicates the next predicted state generated by the battery agent model under the new action.

[0050] Subsequently, based on the deduced predicted state and the next new action generated based on the current strategy Proceed to the next step of the simulation:

[0051]

[0052] in, This represents the prediction result based on the predicted state for the next step and the new action to be taken next, i.e., the predicted state for the next step after that.

[0053] Each simulation generates a pair of states and rewards:

[0054]

[0055] in, This represents the predicted state at step k. This represents the reward calculated based on the state at step k. This represents the reward for the k-th iteration.

[0056] This extrapolation can be performed in multiple steps to generate short branch trajectories. Ultimately, the extrapolation generates additional simulation results. This constitutes a model inference data experience pool, expanding the available training data for the agent, in which This indicates the new action taken in step k; This represents the predicted state at step k+1.

[0057] The above-described model-based prediction diagram is shown below. Figure 3 As shown, by utilizing data from the real trajectory experience pool, this embodiment derives branch trajectory data. A schematic diagram of the branch deduction is shown below. Figure 4 As shown.

[0058] Step 4: Combine the real charging state trajectory data with the inferred branch trajectory data to train the SAC (Soft Actor-Critic) agent, including the policy network and value network; and repeat steps 1-3 above until the reinforcement learning training converges to obtain a reinforcement learning agent that can execute the optimal charging strategy.

[0059] This embodiment uses the real trajectory experience pool D obtained in steps 1-3 in conjunction with the real trajectory experience pool D. real Experience pool D of the derived branch trajectory model Training the agent, i.e., the objective function J(π) of the policy network and the objective function J of the value network (i.e., the Q network). Q (θ) is optimized based on the data from these two experience pools:

[0060]

[0061] in, Q represents the expected value of the union of the actual trajectory experience pool and the derived branch trajectory experience pool. θ (s t ,a t ) represents the result of the value function's calculation of the Q value.

[0062] By combining additional generated inference samples, the agent can explore more combinations of states and actions based on limited actual interaction data, further explore potential policy improvement space, achieve better policy performance, and significantly improve the efficiency of sample utilization.

[0063] This embodiment also provides a lithium battery charging strategy sample efficiency enhancement system based on model short branch inference. This enhancement system is implemented based on the above-mentioned lithium battery charging strategy sample efficiency enhancement method and specifically includes the following modules:

[0064] The sampling module performs charge and discharge tests on the battery based on the actions taken by the reinforcement learning agent, and samples the charging state transition data. The process of sampling the charging state transition data is as follows: the initial charging state parameters are obtained and transmitted to the agent; the agent takes corresponding actions according to the current battery state, and at the same time obtains the corresponding state of the battery at the next moment; the action is to adjust the charging current of the battery.

[0065] The model training module uses the sampled charging state transition data to train the battery proxy model; it uses each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model for model training, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery.

[0066] The extrapolation module, based on the actual charging state trajectory, uses a battery proxy model to perform short-branch extrapolation. That is, taking a certain charging state in the actual sampling trajectory as the starting point, it takes an action different from the next action of the actual charging state trajectory, and uses the prediction of the battery proxy model to obtain the next charging state of the battery. In other words, it simulates the sampling process of the battery and obtains the extrapolated branch trajectory data of the charging state transition.

[0067] The agent training module combines real charging state trajectory data with inferred branch trajectory data to train the agent; and repeats the sampling of charging state transition data, training and inference of the battery agent model until the reinforcement learning training converges, thereby obtaining a reinforcement learning agent that can execute the optimal charging strategy.

[0068] The effectiveness of the proposed method is evaluated using a model-free reinforcement learning (RL) method as a baseline. The learning processes of the model-based short-branch inference-based RL charging method and the model-free RL charging method are as follows: Figure 5As shown, with the increase of training epochs, the constraint violation score approaches the boundary (i.e., zero), indicating that the RL method not only learns voltage and temperature constraints but also learns the optimal solution of the charging strategy (because the optimal solution is located near the constraint boundary). The method proposed in this embodiment found a charging strategy with a Return value of -6.18 (charging time of 1558 seconds, essentially no constraint violations) in the early training phase (defined as the first 100 training epochs), while the optimal charging strategy found by the model-free RL method at this time had a Return value of -9.49 (charging time of 1895 seconds, essentially no constraint violations). The optimal strategy found by the model-based short-branch inference method in the early training phase reduced the charging time by 17.8% compared to the model-free method. After 1000 training epochs, the charging strategy with essentially no constraint violations finally found by the model-based short-branch inference method had a charging time of 1544 seconds, only 0.9% lower than the optimal charging strategy found in the early training phase. Therefore, the charging strategy found by the model-based short-branch inference method was essentially optimized in the early training phase. The model-free method, after 1000 training epochs, finally found a charging strategy with a fundamentally unconstrained violation, which had a charging time of 1565 seconds, a 17.4% reduction compared to the optimal charging strategy found at the beginning of training. Figure 5 As shown in subgraph (b), the optimization of the charging time by the model-free method gradually decreases over 1000 training epochs, meaning that the model-free method still needs a sufficient number of training epochs to achieve convergence.

[0069] Charging curves, including current, SOC, negative electrode potential, and temperature curves, are obtained from the optimal charging strategy found in the early and late stages of training using model-based short-branch inference and model-free methods, respectively. Figure 6 As shown, there is no significant difference in the final convergence results between the proposed method and the model-free method, indicating that the use of the model in this embodiment does not significantly affect the convergence accuracy. Furthermore, in the early stages of training, the charging strategy found by the proposed method is close to optimal, while the charging strategy found by the model-free method is not sufficiently optimized, demonstrating the superiority of this embodiment in terms of sample efficiency.

[0070] This invention constructs a battery proxy model during reinforcement learning and utilizes the branching deduction of the model, which effectively improves the sample efficiency of charging strategy optimization and effectively alleviates the error accumulation problem caused by the recursive use of the model in traditional model-based optimization methods.

[0071] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the protection scope of the claims. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for enhancing the sample efficiency of lithium battery charging strategies based on short-branch model deduction, characterized in that, Includes the following steps: S1. Based on the actions taken by the reinforcement learning agent, charge and discharge tests are performed on the battery, and charging state transition data is sampled. The process of sampling charging state transition data is as follows: obtain the initial charging state parameters and send the initial charging state parameters to the agent; The intelligent agent takes corresponding actions based on the current battery state, and at the same time obtains the battery state at the next moment; the action is to adjust the battery charging current. S2. Train the battery proxy model using the sampled charging state transition data; use each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model to train the model, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery. S3. Based on the actual charging state trajectory, a short branch inference is performed using a battery proxy model. That is, starting from a certain charging state in the actual sampling trajectory, an action different from the next action of the actual charging state trajectory is taken, and the next charging state of the battery is obtained by using the prediction of the battery proxy model. In other words, the sampling process of the battery is simulated to obtain the branch trajectory data of the inferred charging state transition. S4. Combine the real charging state trajectory data with the inferred branch trajectory data to train the agent; and repeat the above steps S1-S3 until the reinforcement learning training converges to obtain a reinforcement learning agent that can execute the optimal charging strategy.

2. The lithium battery charging strategy sample efficiency enhancement method according to claim 1, characterized in that, The intelligent agent consists of a policy network and a value network. The value network is used to calculate the Q-value. The goal of the policy network is to find a strategy that maximizes the weighted sum of the expected cumulative reward and the policy entropy, based on the Q-value calculated by the value network, the available policies, the charging current, and the state of charge of the battery.

3. The lithium battery charging strategy sample efficiency enhancement method according to claim 2, characterized in that, The policy found by the policy network is expressed as: In the formula, π * Let π represent the optimal strategy, α represent the alternative strategy, γ represent the weighting factor, and r(s) represent the reward discount factor. t ,a t ) represents the reward at the current time step; E represents the expected value, which is the Q value estimated by the value network; Represents entropy; a t Indicates the charging current at the current time step; s t The charging state of the battery at the current time step, s t =(SoC t V t ,T t ,η t ), where SoC t For the battery's current state of charge at the current time step, V t T represents the battery voltage at the current time step. t η is the battery temperature at the current time step. t This represents the negative electrode potential at the current time step of the battery.

4. The lithium battery charging strategy sample efficiency enhancement method according to claim 3, characterized in that, The value network of the intelligent agent includes an independent first Q network. With the second Q network The value network approximates the state-action value function; it also includes two independent first-objective Q-networks. With the second target Q network The objective Q-value is calculated; the result of the objective Q-value calculation is the expected value in the policy network objective. The formula for calculating the target Q value is: In the formula, y t Indicates the target Q value. a represents the expected computation starting at the next time step. t+1 Indicates the charging current at the next time step, s t+1 Indicates the charging status at the next time step; Entropy is used to measure the randomness of a policy.

5. The lithium battery charging strategy sample efficiency enhancement method according to claim 2, characterized in that, The battery proxy model includes a multilayer perceptron, with the battery's charging state as input and the next charging state as output. The training dataset for the battery proxy model is derived from the charging state transition data obtained in step S1.

6. The lithium battery charging strategy sample efficiency enhancement method according to claim 5, characterized in that, In step S2, during training, the battery proxy model is updated based on: Where, ω k+1 ω represents the updated battery proxy model parameters. k Indicates the current battery proxy model parameters, a t This represents the charging current at the current time step, s t The charging state of the battery at the current time step, s t+1 Indicates the charging status at the next time step; This means that the loss function of the battery proxy model is defined as the error between the true next charging state and the predicted next charging state based on the current charging state and the action, f. ω (s t ,a t ) represents the predicted next state of charge; Indicates parameter gradient calculation D represents the agent's computational expectation on the real trajectory experience pool; real η represents the actual trajectory experience pool where the charging state transition data is stored; η represents the learning rate.

7. The lithium battery charging strategy sample efficiency enhancement method according to claim 5, characterized in that, Step S3 randomly selects a subset of charging states from the real trajectory experience pool and uses the trained battery agent model to perform short-branch inference; the inference generates additional simulation results that constitute the model inference data experience pool to expand the available training data of the agent.

8. The lithium battery charging strategy sample efficiency enhancement method according to claim 7, characterized in that, In step S3, during the derivation process of the battery proxy model, the charging state s at the current time step is given. t And the next predicted state is generated using the battery agent model based on the action selected by the agent according to the current policy: in, This indicates a new action generated by the agent based on the current policy, wherein the new action is related to the charging current a. t Different charging currents; This indicates the next charging state predicted based on the current charging state and the action taken under the new action. This indicates the next predicted state generated by the battery agent model under the new action; Based on the deduced predicted state and the next new action generated based on the current strategy Proceed to the next step of the simulation: in, This represents the prediction result based on the predicted state for the next step and the new action to be taken next, i.e., the predicted state for the next step after that. Each simulation generates a pair of states and rewards: in, This represents the predicted state at step k. This represents the reward calculated based on the state at step k. This represents the reward for the k-th iteration.

9. The lithium battery charging strategy sample efficiency enhancement method according to claim 2, characterized in that, The actual charging state trajectory data is stored in the actual trajectory experience pool, and the inferred branch trajectory data is stored in the inferred branch trajectory experience pool. Step S4: During agent training, the objective function J(π) of the policy network and the objective function J of the value network are... Q (θ) are all based on the real trajectory experience pool D. real Experience pool D of the derived branch trajectory model Optimize the data: in, Q represents the expected value of the union of the actual trajectory experience pool and the derived branch trajectory experience pool. θ (s t ,a t ) represents the result of the value function's calculation of the Q value; y t s represents the target Q value. t For the battery's state of charge, a t Indicates charging current; r t For r(s) t ,a t () is an abbreviation for the reward at the current time step.

10. A lithium battery charging strategy sample efficiency enhancement system based on model short-branch inference, characterized in that, Based on the lithium battery charging strategy sample efficiency enhancement method according to any one of claims 1-9, the system includes the following modules: The sampling module performs charge and discharge tests on the battery based on the actions taken by the reinforcement learning agent, and samples the charging state transition data. The process of sampling the charging state transition data is as follows: the initial charging state parameters are obtained and transmitted to the agent; the agent takes corresponding actions according to the current battery state, and at the same time obtains the corresponding state of the battery at the next moment; the action is to adjust the charging current of the battery. The model training module uses the sampled charging state transition data to train the battery proxy model; it uses each state and the next state of the real sampled trajectory data as the input and output of the battery proxy model for model training, so as to minimize the prediction error of the battery proxy model for the charging state transition of the battery. The extrapolation module, based on the actual charging state trajectory, uses a battery proxy model to perform short-branch extrapolation. That is, taking a certain charging state in the actual sampling trajectory as the starting point, it takes an action different from the next action of the actual charging state trajectory, and uses the prediction of the battery proxy model to obtain the next charging state of the battery. In other words, it simulates the sampling process of the battery and obtains the extrapolated branch trajectory data of the charging state transition. The agent training module combines real charging state trajectory data with inferred branch trajectory data to train the agent. The process involves repeatedly sampling charging state transition data, training and deducing the battery agent model, until the reinforcement learning training converges, thereby obtaining a reinforcement learning agent capable of executing the optimal charging strategy.

Images