A high-safety and fast charging method for lithium batteries based on deep reinforcement learning

Abstract

A high-safety and fast charging method for lithium batteries based on deep reinforcement learning, which uses an SP+ model to model lithium batteries; according to the pulse working conditions set for lithium batteries in different degradation stages, the SP+ model parameters under different SoHs, different SoCs, different temperatures and different rates are identified; an improved deep reinforcement learning SAC algorithm is constructed, and a reward function is defined; the SP+ model is used as an interactive environment for an intelligent agent of the improved SAC algorithm, and the intelligent agent is trained offline based on the reward function, so that the intelligent agent of the SAC algorithm learns to provide the optimal safe and fast charging strategy under different SoHs, different SoCs, different temperatures and different rates; and the method is used for lithium batteries, and online tests are performed under different SoHs, different SoCs, different temperatures and different rates to output the optimal safe and fast charging strategy; the present invention can better understand and handle complex environments, and has better generalization ability and adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of safe and fast charging of batteries, and in particular to a high-safety and fast charging method for lithium batteries based on deep reinforcement learning. Background Art

[0002] Lithium batteries occupy a core position in the field of electric vehicle energy due to their advantages such as high energy density, low self-discharge rate and long service life, and have become a key driving force for the transformation of modern sustainable transportation. However, lithium batteries face some severe challenges in practical applications, such as thermal runaway, accelerated aging and structural integrity damage, especially in the scenario of fast charging. Blindly pursuing the maximum charging speed is likely to violate critical physical limits and be accompanied by unexpected heat / stress accumulation and side reactions, which will lead to reduced charging efficiency and even safety hazards in the worst cases; therefore, the development of safe and efficient fast charging methods is crucial to meet the growing market demand and alleviate users' range anxiety.

[0003] Authorization announcement number CN114552038B is a lithium battery fast charging method and system based on dynamic programming without lithium deposition. After the SP+ model is built, the lithium battery is fast charged based on the observation results, and the fast charging stage of the lithium battery is optimized through the dynamic programming cost function, thereby realizing the fast charging of the lithium battery. Due to the limited observation objects in the above patent, it is impossible to better understand and handle complex environments. Summary of the invention

[0004] In response to the problems in the prior art, the present invention provides a high-safety and fast charging method for lithium batteries based on deep reinforcement learning, with the aim of automatically learning higher-level representations from environmental state data and better understanding and processing complex environments, so that it has better generalization ability and adaptability.

[0005] A high-safety and fast charging method for lithium batteries based on deep reinforcement learning comprises the following steps:

[0006] Step 1: Use the SP+ model to model the lithium battery and determine the identification parameters required for modeling;

[0007] Step 2: Set pulse conditions at various charge and discharge rates, and conduct degradation experiments on lithium batteries at different temperatures; according to the pulse conditions set for lithium batteries at different degradation stages, identify the SP+ model parameters at different SoH, different SoC, different temperatures, and different rates;

[0008] Step 3: Build an improved deep reinforcement learning SAC algorithm and define the reward function;

[0009] Step 4: Use the SP+ model as the interactive environment of the agent for improving the SAC algorithm, and train the agent offline based on the reward function to enable the agent to learn to provide the optimal safe and fast charging strategy under different SoHs, different SoCs, different temperatures, and different rates;

[0010] Step 5: Based on the offline trained intelligent agent and reward function, and using the physical lithium battery as the interactive environment, online testing is performed at different SoHs, different SoCs, different temperatures, and different rates to output the optimal safe and fast charging strategy.

[0011] Further: record the action value output by the SAC algorithm for: , the input status for: ;in, is the time step, at the time step , Indicates the charge and discharge current rate, represents the SoC value, dSoC represents the SoC increment, Indicates the terminal voltage, represents the negative electrode potential, Indicates temperature;

[0012] The following improvements are made during the training and testing of the SAC algorithm:

[0013] The training process is:

[0014] Step 3.1: Use unilateral sampling instead of bilateral sampling to sample the Gaussian distribution of actions output by the agent of the SAC algorithm:

[0015]

[0016] in, represents the mean of the action Gaussian distribution, Indicates the sample value;

[0017] Step 3.2: Use truncation to convert the action value Standardize within a reasonable range:

[0018]

[0019] in, Indicates the maximum charge and discharge rate. In addition, the state of the intelligent agent is input Normalized to :

[0020]

[0021]

[0022]

[0023] in, and Respectively represent the mean and standard deviation of the state values ​​of all samples in the experience replay pool, Indicates the number of samples in the experience replay pool;

[0024] Step 3.3: Update after each training round and , and The initial values ​​of are 0 and 1 respectively; and the training is stopped when the training round reaches the set threshold;

[0025] The testing process is:

[0026] The action value output by the trained agent is recorded as the strategy action value , strategy action value for:

[0027]

[0028] in, is the terminal voltage safety margin value, is the negative electrode potential safety margin value, Terminal voltage during fast charging The upper cut-off voltage.

[0029] Further: put the tested agent in the environment state Output strategy action value The reward feedback obtained in the environment is used as the reward function. Defined as:

[0030]

[0031] in, Represents a fixed penalty:

[0032]

[0033] Indicates over-release penalty:

[0034]

[0035] Indicates a charging reward or a discharging penalty:

[0036]

[0037] Indicates overvoltage penalty:

[0038]

[0039] Indicates the negative electrode potential penalty:

[0040]

[0041] Indicates over-temperature penalty:

[0042]

[0043] Indicates charging penalty:

[0044]

[0045] is the parameter of the reward or penalty term, Indicates the target SoC value of the charging process, Indicates the maximum allowed charging time. is the maximum allowable temperature.

[0046] Further: Offline training includes the following steps:

[0047] Step 4.1: Randomly initialize the policy network in the agent and predictive value network Network parameters and ;

[0048] Step 4.2: Utilize the predicted value network parameters in the agent To initialize the target value network in the agent Network parameters ;

[0049] Step 4.3: Initialize the experience replay pool capacity , initialize the initial state of the SP+ model ;

[0050] Step 4.4: Set the initial state Normalized to Then input into the policy network In the equation, we get the mean of the action Gaussian distribution and standard deviation , perform unilateral sampling on the action Gaussian distribution to obtain the action sampling value , then Perform truncation operation to output the strategy action at that moment ;

[0051] Step 4.5: Generate the resulting policy action Input into the SP+ model to get the state at the next moment , calculate the reward ,Will Save it as a sample data into the experience replay pool;

[0052] Step 4.6: When the number of samples in the experience replay pool is less than When the state of the next moment is used, steps 4.4 and 4.5 are repeated to obtain a series of sample data and store them in the experience replay pool; when the number of samples in the experience replay pool is greater than When , execute step 4.7;

[0053] Step 4.7: Randomly sample from the experience replay pool Sample data As training data, the target value of each sample is: , the prediction value network of each sample is ;

[0054] in, is the state at the previous moment, It is a strategic network according to Output action, Represents a discount factor, which is used to weaken the impact of future rewards on the sum of current state rewards. Represents action entropy The weighting factor of

[0055] Step 4.8: Update the prediction value network parameters by minimizing the loss function : ,in represents the mean square error, which is used to measure the error between the target value and the predicted value;

[0056] Step 4.9: Update the policy network parameters by minimizing the expected KL divergence : ;

[0057] in, To express expectation; to make The gradient can be calculated normally, so ,in, and Represent the policy network The mean and standard deviation of the output, represents noise sampled from a standard normal distribution;

[0058] Step 4.10: Update target value network parameters using soft update strategy : ,in, Indicates soft update parameters;

[0059] Step 4.11: By minimizing To update the action entropy weighting factor ,in, Represents the target entropy, which is the opposite of the action dimension ;

[0060] Step 4.12: Use the updated network parameters to continue looping steps 4.4 and 4.11, and update the samples in the experience replay pool; when the total number of offline training time steps is greater than the maximum allowed charging time or the current SOC is greater than the target When , execute step 4.13;

[0061] Step 4.13: Update using all samples in the experience replay pool and , execute step 4.14;

[0062] Step 4.14: Use the updated experience replay pool to take step 4.12; when the total training rounds reach the set threshold When , execute step 4.15;

[0063] Step 4.15: Get the intelligent agent after offline training.

[0064] Further, the steps of online testing include:

[0065] Step 5.1: Set terminal voltage safety margin value and negative potential safety margin value ;

[0066] Step 5.2: Load the offline trained agent , and , and the standardized parameters and , the actual state of the lithium battery at the initial moment Normalized to Then input into the policy network In the equation, we get the mean of the action Gaussian distribution and standard deviation , output the best strategy action :

[0067]

[0068] Step 5.3: The optimal policy action will be generated Convert to actual current to charge the lithium battery and get the state of the lithium battery at the next moment ;

[0069] Step 5.4: Repeat steps 5.2 and 5.3 until the lithium battery is fully charged.

[0070] Further: The SP+ model includes the basic process of electrochemical reaction, solid phase diffusion process, liquid phase concentration polarization process, reaction polarization process and ohmic polarization process, and is used to characterize the SoC, terminal voltage, negative electrode potential and temperature changes of lithium batteries during the charging and discharging process.

[0071] Further: the identification process in step 2 is carried out based on the pulse excitation response method and must follow a certain order, that is, at different SOH, different SOC, different temperature, and different rate, the basic characteristic parameter identification of the battery, ohmic polarization parameter identification, electrochemical polarization parameter identification, and diffusion process parameter identification are carried out in sequence, and the parameter spectrum is obtained by interpolation.

[0072] Further, the impulse excitation response method includes the following steps:

[0073] Step 2.1: Prepare a three-electrode battery, fit the inherent characteristic curves of the positive and negative electrodes, and identify the basic characteristic parameters of the battery by combining the basic equations of the battery electrochemical reaction, namely, the initial lithium insertion concentration of the positive and negative electrode active particles, and the maximum variation range of the positive and negative electrode lithium insertion concentration fractions;

[0074] Step 2.2: Use an electrochemical workstation to measure the battery AC impedance spectrum and calculate the equivalent ohmic internal resistance;

[0075] Step 2.3: Calculate the reaction polarization overpotential at the rising edge of the pulse and identify the positive and negative reaction polarization coefficients;

[0076] Step 2.4: Calculate the concentration polarization overpotential at each steady-state operating point, and identify the positive and negative electrode solid phase diffusion time constants, liquid phase diffusion time constants and liquid phase diffusion proportional coefficients.

[0077] The beneficial effects of the present invention are as follows: by improving the SAC algorithm and defining the corresponding reward function, the SP+ model is used as the interactive environment of the intelligent agent of the improved SAC algorithm, and then the intelligent agent is trained offline based on the reward function, so that the intelligent agent can automatically learn higher-level representations from environmental state data and better understand and handle complex environments; then the trained intelligent agent can provide the optimal safe and fast charging strategy for lithium batteries under different SoH, different SoC, different temperatures, and different rates, taking into account the safety and speed of the charging process, minimizing the damage to the lithium battery during the fast charging process, and having better generalization ability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of the present invention;

[0079] Figure 2 A comparison chart of the training and test results of the original SAC algorithm and the improved SAC algorithm provided by the present invention;

[0080] Figure 3 A diagram showing the safe and fast charging results at different temperatures provided by the present invention;

[0081] Figure 4 This is a comparison diagram of the safe and fast charging method provided by the present invention when the ambient temperature is 25°C and the traditional constant current and constant voltage fast charging method. DETAILED DESCRIPTION

[0082] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0083] A high-safety and fast charging method for lithium batteries based on deep reinforcement learning comprises the following steps:

[0084] Step 1: Use the SP+ model to model the lithium battery and determine the identification parameters required for modeling;

[0085] The SP+ model includes the basic process of electrochemical reaction, solid phase diffusion process, liquid phase concentration polarization process, reaction polarization process and ohmic polarization process, and is used to characterize the SoC, terminal voltage, negative electrode potential and temperature changes of lithium batteries during the charging and discharging process;

[0086] The basic electrochemical reaction process of the SP+ model is described by the following formula:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] ,

[0093] in, Positive open circuit potential about The function of Negative open circuit potential about The function of , are the lithium concentration fractions on the surface of the positive and negative active particles of the battery, , are the average lithium insertion concentration fractions inside the positive and negative electrode active particles, respectively. , are the maximum variation range of the positive and negative electrode lithium insertion concentration fractions, is the load current, the discharge is specified to be positive, is the total battery capacity (C);

[0094] The solid phase diffusion process of the SP+ model is described by the following formula:

[0095]

[0096]

[0097]

[0098]

[0099] in, , are the solid diffusion time constants of the positive and negative electrodes, respectively;

[0100] The liquid phase concentration polarization process of the SP+ model is described by the following formula:

[0101]

[0102]

[0103] in, is the ideal gas constant, is the internal temperature of the battery (K), is the Faraday constant, is the initial value of the lithium ion concentration of the electrolyte, generally set to 1000, is the cation migration number, generally 0.3~0.4, is the liquid phase diffusion time constant, is the liquid phase diffusion coefficient;

[0104] The reaction polarization process of the SP+ model is described by the following formula:

[0105]

[0106]

[0107]

[0108] The polarization overpotential of the negative electrode reaction is:

[0109]

[0110] in, , are the positive and negative reaction polarizability coefficients, respectively;

[0111] The ohmic polarization process of the SP+ model is described by the following formula:

[0112]

[0113] in, is the equivalent ohmic internal resistance.

[0114] The terminal voltage of the SP+ model It is described by the following formula:

[0115]

[0116] The parameters that need to be identified include the initial lithium insertion concentration of the positive and negative active particles of the battery , , the maximum variation range of the positive and negative electrode lithium insertion concentration fraction , , positive and negative electrode capacity , , positive and negative electrode solid phase diffusion time constant , , liquid phase diffusion time constant , liquid phase diffusion coefficient , positive and negative reaction polarization coefficients , , equivalent ohmic internal resistance , a total of 13;

[0117] Step 2: Set pulse conditions at various charge and discharge rates, and conduct degradation experiments on lithium batteries at different temperatures; according to the pulse conditions set for lithium batteries at different degradation stages, identify the SP+ model parameters at different SoH, different SoC, different temperatures, and different rates;

[0118] The maximum acceptable charging rate of the battery used is 4C, and the maximum acceptable discharge rate is 4C. Four charging rates are set to 1C, 2C, 3C, and 4C, and four discharge rates are set to 1C, 2C, 3C, and 4C. The battery is subjected to pulse discharge and charging conditions from a fully charged state. Among them, the battery is cross-discharged and charged from small to large rates, first discharged and then charged. Each discharge and charge lasts for 1 minute, and the battery needs to be left alone for 30 minutes after each charge or discharge. After a round of four charging rates and discharge rates from small to large, it is considered that one cycle is completed, and the cycle continues until the discharge cut-off voltage is reached;

[0119] The battery degradation experiment is carried out at different temperatures, and the above pulse conditions are set for the battery at different degradation stages. The degradation experiment refers to the degradation experiment in a constant temperature box at different temperatures (5℃, 10℃, 25℃, 30℃, 40℃), with 1C constant current and constant voltage charging to the charging cut-off voltage, and 1C constant current discharge to the discharge cut-off voltage. The pulse condition is carried out every 50 cycles, and the current corresponding battery SOH is recorded;

[0120] The identification process is based on the pulse excitation response method and must be carried out in a certain order, that is, at different SOH, different SOC, different temperature, and different rate, the basic characteristic parameters of the battery, ohmic polarization parameters, electrochemical polarization parameters, and diffusion process parameters are identified in sequence, and the parameter spectrum is obtained by interpolation; the pulse excitation response method includes the following steps:

[0121] Step 2.1: Prepare a three-electrode battery, fit the inherent characteristic curves of the positive and negative electrodes, and identify the basic characteristic parameters of the battery by combining the basic equations of the battery electrochemical reaction, namely, the initial lithium insertion concentration of the positive and negative electrode active particles, and the maximum variation range of the positive and negative electrode lithium insertion concentration fractions;

[0122] Step 2.2: Use an electrochemical workstation to measure the battery AC impedance spectrum and calculate the equivalent ohmic internal resistance;

[0123] Step 2.3: Calculate the reaction polarization overpotential at the rising edge of the pulse and identify the positive and negative reaction polarization coefficients;

[0124] Step 2.4: Calculate the concentration polarization overpotential at each steady-state operating point, and identify the positive and negative electrode solid phase diffusion time constants, liquid phase diffusion time constants and liquid phase diffusion proportional coefficients;

[0125] Step 3: Build an improved deep reinforcement learning SAC algorithm and define the reward function;

[0126] Reinforcement learning is an efficient machine learning method used to solve various optimization problems. It can automatically search for the optimal solution based on the feedback of the cost function. Compared with reinforcement learning, deep reinforcement learning uses deep neural networks as cost function approximators, which can automatically learn higher-level representations from environmental state data, better understand and handle complex environments, and has better generalization and adaptability.

[0127] Note the action value output by the SAC algorithm for: , the input status for: ;in, is the time step (the time interval of each step is 1 second), and in the time step , Indicates the charge and discharge current rate (positive value is the charge current rate, negative value is the discharge current rate). represents the SoC value, and dSoC represents the SoC increment ( = ), Indicates the terminal voltage, represents the negative electrode potential, Indicates temperature;

[0128] The following improvements are made during the training and testing of the SAC algorithm:

[0129] The training process is:

[0130] Step 3.1: In order to enhance the agent's ability to obtain the boundary value of the action space while maintaining the original high exploration ability, use unilateral sampling instead of bilateral sampling to sample the Gaussian distribution of the action output by the agent of the SAC algorithm:

[0131]

[0132] in, represents the mean of the action Gaussian distribution, Indicates the sample value;

[0133] Step 3.2: Use truncation to convert the action value Standardize within a reasonable range:

[0134]

[0135] in, Indicates the maximum charge and discharge rate. In addition, the state of the intelligent agent is input Normalized to , to enhance the stability of the agent training process:

[0136]

[0137]

[0138]

[0139] in, and Respectively represent the mean and standard deviation of the state values ​​of all samples in the experience replay pool, Indicates the number of samples in the experience replay pool;

[0140] Step 3.3: When the number of training rounds reaches a set threshold, stop training. In specific implementation, the set threshold is 200 times.

[0141] The testing process is:

[0142] Due to the fast charging process, Terminal voltage at time Exceeding the upper cut-off voltage or negative electrode potential When the voltage is less than 0V, it will lead to overvoltage or lithium deposition, causing irreversible damage to the battery. In order to avoid the above situation and further improve the safety of the fast charging process, the intelligent agent needs to autonomously introduce negative charging pulses. Therefore, the action value output by the trained intelligent agent is recorded as the strategy action value. , strategy action value for:

[0143]

[0144] in, is the terminal voltage safety margin value, is the negative electrode potential safety margin value, Terminal voltage during fast charging The upper cut-off voltage;

[0145] The reward function is defined as:

[0146] The tested agent is placed in the environment state Output strategy action value The reward feedback obtained from the environment is used as the reward function. The agent can update parameters based on the reward feedback. Defined as:

[0147]

[0148] in, Represents a fixed penalty:

[0149]

[0150] Indicates over-release penalty:

[0151]

[0152] Indicates a charging reward or a discharging penalty:

[0153]

[0154] Indicates overvoltage penalty:

[0155]

[0156] Indicates the negative electrode potential penalty:

[0157]

[0158] Indicates over-temperature penalty:

[0159]

[0160] Indicates charging penalty:

[0161]

[0162] is the parameter of the reward or penalty term, Indicates the target SoC value of the charging process, Indicates the maximum allowed charging time. is the maximum allowable temperature; the above mentioned parameters are summarized in Table 1, and the parameter values ​​should be adjusted accordingly according to the specific model of the lithium battery;

[0163] Table 1 Parameter setting table

[0164]

[0165] Step 4: Use the SP+ model as the interactive environment of the agent for improving the SAC algorithm, and train the agent offline based on the reward function to enable the agent to learn to provide the optimal safe and fast charging strategy under different SoHs, different SoCs, different temperatures, and different rates;

[0166] Offline training includes the following steps:

[0167] Step 4.1: Randomly initialize the policy network in the agent and predictive value network Network parameters and ;

[0168] Step 4.2: Utilize the predicted value network parameters in the agent To initialize the target value network in the agent Network parameters ;

[0169] Step 4.3: Initialize the experience replay pool capacity , initialize the initial state of the SP+ model ;

[0170] Step 4.4: Set the initial state Normalized to Then input into the policy network In the equation, we get the mean of the action Gaussian distribution and standard deviation , perform unilateral sampling on the action Gaussian distribution to obtain the action sampling value , then Perform truncation operation to output the strategy action at that moment ;

[0171] Step 4.5: Generate the resulting policy action Input into the SP+ model to get the state at the next moment , calculate the reward ,Will Save it as a sample data into the experience replay pool;

[0172] Step 4.6: When the number of samples in the experience replay pool is less than When the state of the next moment is used, steps 4.4 and 4.5 are repeated to obtain a series of sample data and store them in the experience replay pool; when the number of samples in the experience replay pool is greater than When , execute step 4.7;

[0173] Step 4.7: Randomly sample from the experience replay pool Sample data As training data, the target value of each sample is: , the prediction value network of each sample is ;

[0174] in, is the state at the previous moment, It is a strategic network according to Output action, Represents a discount factor, which is used to weaken the impact of future rewards on the sum of current state rewards. Represents action entropy The weighting factor of

[0175] Step 4.8: Update the prediction value network parameters by minimizing the loss function : ,in represents the mean square error, which is used to measure the error between the target value and the predicted value;

[0176] Step 4.9: Update the policy network parameters by minimizing the expected KL divergence : ;

[0177] in, express expectation; in order to make To calculate the gradient normally, we need to use the re-parameterization technique, ,in, and Represent the policy network The mean and standard deviation of the output, represents noise sampled from a standard normal distribution;

[0178] Step 4.10: Update target value network parameters using soft update strategy : ,in, Indicates soft update parameters;

[0179] Step 4.11: By minimizing To update the action entropy weighting factor ,in, Represents the target entropy, which is the opposite of the action dimension ;

[0180] Step 4.12: Use the updated network parameters to continue looping steps 4.4 and 4.11, and update the samples in the experience replay pool; when the total number of offline training time steps is greater than the maximum allowed charging time or the current SOC is greater than the target When , a training round is completed, and step 4.13 is executed;

[0181] Step 4.13: Update using all samples in the experience replay pool and , execute step 4.14;

[0182] Step 4.14: Use the updated experience replay pool to take step 4.12; when the total number of training rounds reaches the set threshold When , execute step 4.15;

[0183] Step 4.15: Get the intelligent agent after offline training;

[0184] The hyperparameter settings of the improved SAC algorithm are shown in Table 2:

[0185] Table 2 Reference table for hyperparameter settings of improved SAC algorithm

[0186]

[0187] Combination Figure 2 As shown, compared with the original SAC algorithm, the improved SAC algorithm improves the agent's ability to explore the action space boundary during the training process. Therefore, during the test process, the agent of the improved SAC algorithm can more effectively obtain the action space boundary value (i.e., the maximum charging current), so that the present invention can obtain a faster and safer charging strategy;

[0188] Step 5: Based on the offline trained agent and reward function, and using the physical lithium battery as the interactive environment, online testing is performed at different SoHs, different SoCs, different temperatures, and different rates to output the optimal safe and fast charging strategy;

[0189] The steps of online testing include:

[0190] Step 5.1: Set terminal voltage safety margin value and negative potential safety margin value ;

[0191] Step 5.2: Load the offline trained agent , and , and the standardized parameters and , the actual state of the lithium battery at the initial moment Normalized to Then input into the policy network In the equation, we get the mean of the action Gaussian distribution and standard deviation , output the best strategy action :

[0192]

[0193] Step 5.3: The optimal policy action will be generated Convert to actual current to charge the lithium battery and get the state of the lithium battery at the next moment ;

[0194] Step 5.4: Repeat steps 5.2 and 5.3 until the lithium battery is fully charged.

[0195] The charging current, terminal voltage, negative electrode potential and temperature curves of the present invention at different ambient temperatures are shown in the attached figure. Figure 2It can be seen that the terminal voltage, negative electrode potential and temperature are always kept within the safe range during the fast charging process, avoiding overvoltage, lithium deposition and overtemperature. This shows that the invention can effectively reduce the irreversible damage to the lithium battery caused by the fast charging process, and can also adapt to different environments and has strong generalization ability. Figure 3 The comparison results between the fast charging method of the present invention and the traditional constant current constant voltage fast charging method are shown; it can be seen that in the traditional constant current constant voltage fast charging method, the negative electrode potential is less than 0 in the middle and late stages of the charging process, and lithium plating occurs, while the fast charging method of the present invention does not; in terms of charging speed, the fast charging method of the present invention is better than the 2C constant current constant voltage fast charging method, but worse than the 3C and 4C constant current constant voltage fast charging methods. Therefore, compared with the traditional constant current constant voltage fast charging method, the fast charging method of the present invention can improve the safety of lithium batteries during the fast charging process and reduce irreversible damage, while maintaining a faster charging speed.

[0196] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A high-safety and fast charging method for lithium batteries based on deep reinforcement learning, characterized in that: The steps include: Step 1: Use the SP+ model to model the lithium battery and determine the identification parameters required for modeling; Step 2: Set pulse conditions at various charge and discharge rates, and conduct degradation experiments on lithium batteries at different temperatures; according to the pulse conditions set for lithium batteries at different degradation stages, identify the SP+ model parameters at different SoH, different SoC, different temperatures, and different rates; Step 3: Build an improved deep reinforcement learning SAC algorithm and define the reward function; Step 4: Use the SP+ model as the interactive environment of the intelligent agent for improving the SAC algorithm, and train the intelligent agent offline based on the reward function to enable the intelligent agent to learn to provide the optimal safe and fast charging strategy under different SoH, different SoC, different temperature, and different rates; offline training includes the following steps: Step 4.1: Randomly initialize the policy network π in the agent θ (s) and predicted value network The network parameters θ and Step 4.2: Utilize the predicted value network parameters in the agent To initialize the target value network in the agent Network parameters Step 4.3: Initialize the experience replay pool capacity D and the initial state s0 of the SP+ model; Step 4.4: Normalize the initial state s0 to z0 and input it into the policy network (μ a ,σ a )=π θ (z0), we get the mean μ of the action Gaussian distribution a and standard deviation σ a , perform unilateral sampling on the action Gaussian distribution to obtain the action sampling value u a , and then to u a Perform truncation operation to output the strategy action a0 at that moment; Step 4.5: Input the generated policy action a0 into the SP+ model, obtain the state s1 at the next moment, calculate the reward r0=r(s0,a0), and save (s0,a0,r0,s1) as a sample data into the experience replay pool; Step 4.6: When the number of samples in the experience replay pool is less than N B When the state of the next moment is used, steps 4.4 and 4.5 are repeated to obtain a series of sample data and store them in the experience replay pool; when the number of samples in the experience replay pool is greater than N B When , execute step 4.7; Step 4.7: Randomly sample N from the experience replay pool B Sample data (s t ,a t1 ,r t ,s t+1 ) as training data, the target value of each sample is: The prediction value network for each sample is Among them, s t+1 is the state at the previous moment, a t1+1 is the policy network π θ (s) According to s t+1 The output action, γ represents the discount factor, which is used to weaken the impact of future rewards on the sum of current state rewards, and α represents the action entropy log(π θ (a t1+1 |s t+1 ))'s weighting factor; Step 4.8: Update the prediction value network parameters by minimizing the loss function Where MSE[·] represents the mean square error, which is used to measure the error between the target value and the predicted value; Step 4.9: Update the policy network parameters θ by minimizing the expected KL divergence: Where E[·] represents the expectation; in order to make J πθ (θ) can calculate the gradient normally, let a t1 =μ a +∈ t *σ a , where μ a and σ a Represent the policy network π θ The mean and standard deviation of the output, ∈ t represents noise sampled from a standard normal distribution; Step 4.10: Update target value network parameters using soft update strategy Among them, τ represents the soft update parameter; Step 4.11: By minimizing To update the action entropy weighting factor α, where H0 represents the target entropy, which is the opposite of the action dimension -dim(a t1 )=-1; Step 4.12: Use the updated network parameters to continue looping steps 4.4 and 4.11, and update the samples in the experience replay pool; when the total number of offline training time steps is greater than the maximum allowed charging time or the current SOC is greater than the target SoC ref When , execute step 4.13; Step 4.13: Update μ using all samples in the experience replay pool s and σ s , execute step 4.14; Step 4.14: Use the updated experience replay pool to take step 4.12; when the total number of training rounds reaches the set threshold M, execute step 4.15; Step 4.15: Get the intelligent agent after offline training; Step 5: Based on the offline trained intelligent agent and reward function, and using the physical lithium battery as the interactive environment, online testing is performed at different SoHs, different SoCs, different temperatures, and different rates to output the optimal safe and fast charging strategy.

2. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 1 is characterized in that: Note the action value a output by the SAC algorithm t For: a t =C t , the input state s t For: t =(SoC t ,dSoC t ,V t ,Nep t ,T t ), where t is the time step. At time step t, C t Indicates the charge and discharge current rate, SoC t represents the SoC value, dSoC represents the SoC increment, V t Indicates terminal voltage, Nep t represents the negative electrode potential, T t Indicates temperature; The following improvements are made during the training and testing of the SAC algorithm: The training process is: Step 3.1: Use unilateral sampling instead of bilateral sampling to sample the Gaussian distribution of actions output by the agent of the SAC algorithm: Among them, μ a represents the mean of the action Gaussian distribution, u a Indicates the sample value; Step 3.2: Use truncation to convert the action value a t Standardize within a reasonable range: Among them, C max represents the maximum charge and discharge rate. In addition, the state s of the agent is input t Normalized to z t : Among them, μ s and σ s They represent the mean and standard deviation of the state values ​​of all samples in the experience replay pool, respectively, and N represents the number of samples in the experience replay pool; Step 3.3: Update μ after each training round s and σ s , μ s and σ s The initial values ​​of are 0 and 1 respectively; and the training is stopped when the training round reaches the set threshold; The testing process is: The action value output by the trained agent is recorded as the strategy action value a t1 , strategy action value a t1 for: Among them, ΔV is the terminal voltage safety margin value, ΔNep is the negative electrode potential safety margin value, V max The terminal voltage V during fast charging t The upper cut-off voltage.

3. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 2 is characterized in that: The tested agent is placed in the environment state s t Output strategy action value a t1 The reward feedback obtained from the environment is used as the reward function, and the reward function r(s t ,a t1 ) is defined as: r(s t ,a t1 )=r step (t)+r SoC (t)+r dSoC (t)+r V (t)+r Nep (t)+r T (t)+r time (t) Among them, r step (t) represents a fixed penalty: r step (t)=-ω1 r SoC (t) indicates over-release penalty: r SoC (t) = ω2 * dSoC t , SoC t <0 and dSoC t <0 r dSoC (t) indicates charging reward or discharging penalty: r V (t) indicates overvoltage penalty: r Nep (t) represents the negative electrode potential penalty: r T (t) indicates over-temperature penalty: r time (t) indicates charging penalty: ω i (i=1~8) is the parameter of reward or penalty item, SoC ref Indicates the target SoC value of the charging process, t max Indicates the maximum allowable charging time, T max is the maximum allowable temperature.

4. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 1 is characterized in that: The steps of online testing include: Step 5.1: Set the terminal voltage safety margin value ΔV and the negative electrode potential safety margin value ΔNep; Step 5.2: Load the agent's θ after offline training. and And the normalization parameter μ s and σ s , the actual lithium battery state s0 at the initial moment is standardized to z0 and then input into the strategy network (μ a ,σ a )=π θ (z0), we get the mean μ of the action Gaussian distribution a and standard deviation σ a , output the best strategy action a t2 : Step 5.3: The optimal policy action a will be generated t2 Convert it into actual current to charge the lithium battery and obtain the state s1 of the lithium battery at the next moment; Step 5.4: Repeat steps 5.2 and 5.3 until the lithium battery is fully charged.

5. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 1 is characterized in that: The SP+ model includes the basic process of electrochemical reaction, solid phase diffusion process, liquid phase concentration polarization process, reaction polarization process and ohmic polarization process, and is used to characterize the SoC, terminal voltage, negative electrode potential and temperature changes of lithium batteries during the charging and discharging process.

6. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 5 is characterized in that: The identification process in step 2 is based on the pulse excitation response method and must be carried out in a certain order, that is, at different SOH, different SOC, different temperature, and different rate, the basic characteristic parameter identification of the battery, ohmic polarization parameter identification, electrochemical polarization parameter identification, and diffusion process parameter identification are carried out in sequence, and the parameter spectrum is obtained by interpolation.

7. The high-safety and fast charging method for lithium batteries based on deep reinforcement learning according to claim 6 is characterized in that: The impulse response method includes the following steps: Step 2.1: Prepare a three-electrode battery, fit the inherent characteristic curves of the positive and negative electrodes, and identify the basic characteristic parameters of the battery by combining the basic equations of the battery electrochemical reaction, namely, the initial lithium insertion concentration of the positive and negative electrode active particles, and the maximum variation range of the positive and negative electrode lithium insertion concentration fractions; Step 2.2: Use an electrochemical workstation to measure the battery AC impedance spectrum and calculate the equivalent ohmic internal resistance; Step 2.3: Calculate the reaction polarization overpotential at the rising edge of the pulse and identify the positive and negative reaction polarization coefficients; Step 2.4: Calculate the concentration polarization overpotential at each steady-state operating point, and identify the positive and negative electrode solid phase diffusion time constants, liquid phase diffusion time constants and liquid phase diffusion proportional coefficients.

Images