Battery SOC online estimation method based on deep reinforcement learning and related equipment

Abstract

The invention provides a deep reinforcement learning-based battery SOC online estimation method and related equipment. The method comprises the steps of obtaining historical state parameters of a target battery; defining an action space corresponding to the historical state parameters, wherein the action in the action space represents the SOC adjustment amplitude of the target battery; constructing a dual reinforcement learning model combining a CNN-BiLSTM network and a dual deep Q network, and generating experience data interacting with the environment according to the dual reinforcement learning model to construct an experience pool; preferentially sampling through a PER experience playback mechanism, and circularly training the double reinforcement learning model until the training of the double reinforcement learning model reaches a preset condition; and obtaining current state parameters of the target battery, inputting the current state parameters into the trained double reinforcement learning model, and outputting the current SOC of the target battery. By applying the method provided by the embodiment of the invention, the state of charge of the battery can be accurately predicted according to deep reinforcement learning.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of batteries, in particular to a battery SOC online estimation method based on deep reinforcement learning and related equipment. BACKGROUND

[0002] The state of charge (SOC) of a battery refers to the proportion of the remaining capacity at a specific moment. Accurate estimation of the SOC of a battery is crucial for many applications, especially in the fields of electric vehicles, energy storage systems, and portable electronic devices. Precise SOC estimation not only helps to optimize the efficiency of battery use and prolong its service life, but also ensures the safety and reliability of the system. For example, in electric vehicles, accurate SOC estimation can prevent overcharging or discharging of the battery, thereby avoiding potential safety risks and improving the vehicle's range.

[0003] Currently, one of the widely used SOC estimation methods is the Coulomb counting method. This method calculates the charging or discharging capacity of the battery by measuring the current flowing through the battery and integrating it over time, thereby estimating the SOC of the battery. However, the Coulomb counting method is heavily dependent on accurate initial SOC values. If the initial SOC value is not accurate, even if the subsequent current measurement is very accurate, the cumulative error will gradually increase, leading to deviation in the entire SOC estimation process. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a battery SOC online estimation method based on deep reinforcement learning and related equipment, which can accurately predict the state of charge of the battery. The specific scheme is as follows:

[0005] A battery SOC online estimation method based on deep reinforcement learning, comprising:

[0006] obtaining historical state parameters of a target battery, the historical state parameters including at least voltage, current, temperature, and an SOC label corresponding to the historical state parameters;

[0007] defining an action space corresponding to the historical state parameters, the action space including actions of multiple action types, each action being used to represent the adjustment amplitude of the SOC of the target battery; the action types including reducing the SOC, keeping the SOC unchanged, or increasing the SOC;

[0008] constructing a double reinforcement learning model combining a CNN-BiLSTM network and a double deep Q network, wherein the first CNN-BiLSTM network in the double reinforcement learning model is used to select actions in the action space and update parameters, and the second CNN-BiLSTM network is used to calculate target Q values.

[0009] generating experience data of interaction with the environment according to the dual reinforcement learning model to build an experience pool;

[0010] preferentially sampling through a PER experience replay mechanism, and training the dual reinforcement learning model cyclically until the dual reinforcement learning model training reaches a preset condition;

[0011] obtaining a current state parameter of the target battery, inputting the trained dual reinforcement learning model, and outputting a current SOC of the target battery, wherein the current state parameter does not include the current SOC.

[0012] Optionally, the CNN-BiLSTM network includes a CNN network, a BiLSTM network, and a full connection layer; the CNN network is configured to extract local spatial features of parameters in the historical state parameter except the SOC label; the BiLSTM network is configured to extract time features of the parameters in the historical state parameter except the SOC label; and the full connection layer is configured to map the local spatial features and the time features of the historical state parameter to the action space.

[0013] Optionally, the CNN network and the BiLSTM network are connected in series, and an output of the CNN network is used as an input of the BiLSTM network.

[0014] Optionally, the generating experience data of interaction with the environment according to the dual reinforcement learning model to build an experience pool includes:

[0015] obtaining a current state parameter of the target battery, and using the first CNN-BiLSTM network to select a current action corresponding to the current state parameter;

[0016] obtaining a next state parameter of the target battery after performing the current action, and generating experience data of interaction with the environment; wherein the experience data includes the current state parameter of the target battery, the current action of the target battery, a reward value, the next state parameter of the target battery, and a completion flag;

[0017] storing the experience data to the experience pool.

[0018] Optionally, in the training process of the dual reinforcement learning model, an ε-greedy strategy is used to randomly select an action; and after the dual reinforcement learning model is trained, the ε-greedy strategy is used to select an optimal action.

[0019] Optionally, after the obtaining the next state parameter of the target battery after performing the current action, the method further includes:

[0020] determining a current Q value according to the current state parameter, the current action and the first CNN-BiLSTM network, and selecting a next action corresponding to the next state parameter according to the first CNN-BiLSTM network;

[0021] determining a target Q value according to the next state parameter, the next action and the second CNN-BiLSTM network;

[0022] updating the model parameters of the first CNN-BiLSTM network according to the current Q value and the target Q value.

[0023] The method described above, optionally, the priority sampling through the PER experience replay mechanism, the cyclic training model, comprising:

[0024] determining the TD error of each experience data stored in the experience pool, and determining the priority of each experience data according to the TD error;

[0025] determining the sampling probability of each experience data according to the priority of each experience data;

[0026] sampling the experience pool according to the sampling probability, and calculating the weight of each experience data obtained by sampling;

[0027] determining the loss function value according to the experience data obtained by sampling, and weighting the loss function value by the weight;

[0028] updating the model parameters of the first CNN-BiLSTM network by using the weighted loss function value, and returning to the step of determining the TD error of each experience data stored in the experience pool;

[0029] In the case of reaching a preset time, copying the model parameters in the first CNN-BiLSTM network to the second CNN-BiLSTM network.

[0030] A battery SOC online estimation device based on deep reinforcement learning, comprising:

[0031] An acquisition unit is configured to acquire historical state parameters of a target battery, wherein the historical state parameters at least include voltage, current, temperature, and an SOC label corresponding to the historical state parameters;

[0032] An execution unit is configured to define an action space corresponding to the historical state parameters, wherein the action space includes actions of multiple action types, each of the actions is used to represent an SOC adjustment amplitude of the target battery, and the action types include reducing the SOC, keeping the SOC unchanged, or increasing the SOC;

[0033] a first constructing unit configured to construct a double reinforcement learning model combining a CNN-BiLSTM network and a double deep Q network, wherein a first CNN-BiLSTM network in the double reinforcement learning model is configured to select an action in the action space and update parameters, and a second CNN-BiLSTM network is configured to calculate a target Q value;

[0034] a second constructing unit configured to generate experience data for interaction with an environment according to the double reinforcement learning model, so as to construct an experience pool;

[0035] a training unit configured to train the double reinforcement learning model through a PER experience replay mechanism for preferential sampling, and cycle the training until the double reinforcement learning model training reaches a preset condition;

[0036] an online estimation unit configured to obtain a current state parameter of a target battery, input the current state parameter into the trained double reinforcement learning model, and output a current SOC of the target battery, wherein the current state parameter does not include the current SOC.

[0037] A storage medium, comprising stored instructions, wherein the instructions, when executed, control a device in which the storage medium is located to perform the battery SOC online estimation method based on deep reinforcement learning as described above.

[0038] An electronic device, comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the battery SOC online estimation method based on deep reinforcement learning as described above.

[0039] Based on the above, the application provides a battery SOC online estimation method based on deep reinforcement learning and related equipment, and the method comprises the following steps: obtaining historical state parameters of a target battery, wherein the historical state parameters at least comprise voltage, current, temperature, and an SOC label corresponding to the historical state parameters; defining an action space corresponding to the historical state parameters, wherein the action space comprises actions of multiple action types, and each action is used for representing an SOC adjustment amplitude of the target battery; the action types comprise reducing the SOC, keeping the SOC unchanged, or increasing the SOC; constructing a double reinforcement learning model combined with a CNN-BiLSTM network and a double deep Q network, wherein a first CNN-BiLSTM network in the double reinforcement learning model is used for selecting an action in the action space and updating parameters, and a second CNN-BiLSTM network is used for calculating a target Q value; generating experience data interacting with an environment according to the double reinforcement learning model to construct an experience pool; preferentially sampling through a PER experience replay mechanism, and cyclically training the double reinforcement learning model until the double reinforcement learning model training reaches a preset condition; obtaining a current state parameter of the target battery, inputting the current state parameter into the trained double reinforcement learning model, and outputting a current SOC of the target battery, wherein the current state parameter does not comprise a current SOC. By applying the method provided in the application, the state of charge of the battery can be accurately predicted according to the double reinforcement learning model. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0041] Figure 1 A method flow chart of a battery SOC online estimation method based on deep reinforcement learning provided by the present application;

[0042] Figure 2 A structure schematic diagram of a first CNN-BiLSTM network provided by the present application;

[0043] Figure 3 A schematic diagram of a data processing process of an agent provided by the present application;

[0044] Figure 4 A flow chart of a training process of an online network provided by the present application;

[0045] Figure 5 A learning result schematic diagram of a neural network provided by the present application;

[0046] Figure 6 A performance test result schematic diagram of a neural network provided for the present application is shown in the following figure;

[0047] Figure 7 A structure schematic diagram of a battery SOC online estimation device based on deep reinforcement learning provided for the present application is shown in the following figure;

[0048] Figure 8 A structure schematic diagram of an electronic device provided for the present application is shown in the following figure. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] In the present application, the term “comprises”, “comprising” or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement “comprises a” does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0051] The embodiments of the present application provide a battery SOC online estimation method based on deep reinforcement learning, which can be applied to an electronic device, and the electronic device can be a computer, a tablet, a smart phone, a smart wearable device, etc. The method flowchart of the method is shown in the following figure, and specifically includes: Figure 1

[0052] S101: Obtain historical state parameters of a target battery, and the historical state parameters at least include voltage, current, temperature, and a state of charge (SOC) label corresponding to the historical state parameters.

[0053] In the present embodiment, the historical state parameters can further include one or more parameters such as power, internal resistance, and equivalent cycle number. The advantages of integrating multiple influencing factors can improve the SOC estimation accuracy, robustness and adaptability of the battery under dynamic, complex and aging working conditions.

[0054] S102: Define an action space corresponding to the historical state parameters, and the action space includes actions of multiple action types, and each action is used to represent an SOC adjustment amplitude of the target battery; and the action types include reducing the SOC, keeping the SOC unchanged or increasing the SOC.​

[0055] In this embodiment, each action of the action space can include different adjustment values to effectively cover the needs of battery dynamic adjustment. For example, reduce x1SOC, reduce x2SOC, …, SOC remains unchanged, increase y1SOC, increase y2SOC, … The adjustment value of each action is discrete, and through the discrete action space, the fixed action mode simplifies the decision space, reduces the model complexity, and can improve the inference efficiency and reduce the extreme prediction of the model on unseen data.

[0056] S103: Construct a double reinforcement learning model combining a CNN-BiLSTM network and a double deep Q network, wherein the first CNN-BiLSTM network in the double reinforcement learning model is used to select an action in the action space and update parameters, and the second CNN-BiLSTM network is used to calculate a target Q value.

[0057] In this embodiment, the first CNN-BiLSTM network in the double reinforcement learning model can be an online network in the DDQN algorithm, which can be represented as wherein θ is the network parameter of the first CNN-BiLSTM network, a represents an action, represents the adjustment value to the SOC, and takes values from the action space; s is the current state, including voltage, current, temperature, power, internal resistance, equivalent cycle number and other battery state parameters.

[0058] Optionally, the second CNN-BiLSTM network can be a target network in the DDQN algorithm, represented as , θ- is the network parameter of the second CNN-BiLSTM network.

[0059] S104: Generate experience data interacting with the environment according to the double reinforcement learning model to build an experience pool.

[0060] In this embodiment, the current battery state parameters in the environment are input into the double reinforcement learning model, the action is selected by the double reinforcement learning model, the reward value and the next battery state parameters of the environment are obtained after the action is performed in the environment, the interaction with the environment is completed, and the experience data can include the current battery state parameters, the action selected by the double reinforcement learning model and the next battery state parameters.

[0061] Optionally, the experience pool stores a plurality of experience data.

[0062] S105: Prioritize sampling through the PER experience replay mechanism, and train the double reinforcement learning model in a loop until the double reinforcement learning model training reaches a preset condition.

[0063] In this embodiment, the double reinforcement learning model is trained according to the experience data sampled from the experience pool through the PER experience replay mechanism.

[0064] In this embodiment, the preset condition can be that the loss function converges or the number of training times of the double reinforcement learning model is greater than a threshold of the number of training times.

[0065] S106: Obtain a current state parameter of the target battery, input the trained double reinforcement learning model, and output a current SOC of the target battery, wherein the current state parameter does not include the current SOC.

[0066] In this embodiment, when the double reinforcement learning model training reaches the preset condition, the double reinforcement learning model can be used for online estimation of the battery SOC. Specifically, the current state parameter of the target battery can be input into the double reinforcement learning model to obtain the current SOC of the target battery. It should be noted that the current state parameter does not include the current SOC itself, but estimates the SOC through other measurement data related to the battery state, which can reduce the dependence on the initial SOC, that is, does not directly use the SOC as the input, reduces the data dependence, enhances the adaptability to different battery types and use cases, and in actual application, it is often difficult to accurately obtain the initial SOC, avoiding the problem that the model may fall into a circular dependence if the initial SOC is directly used as the input, enhancing the adaptability and robustness of the model, and avoiding error accumulation.

[0067] In some embodiments, after obtaining the current SOC of the target battery, at least one operation of charge-discharge strategy optimization, fault diagnosis, and energy management of the target battery can be performed using the current SOC of the target battery.

[0068] The method provided in the embodiments of the present application can accurately identify the state of charge of the battery according to the double reinforcement learning model, and enhance the adaptability and robustness of the SOC estimation under different initial SOC conditions.

[0069] In an embodiment provided in the present application, based on the above scheme, the CNN-BiLSTM network includes a CNN network, a BiLSTM network, and a fully connected layer; the CNN network is used to extract local spatial features of parameters in the historical state parameters except the SOC label; the BiLSTM network is used to extract time features of the parameters in the historical state parameters except the SOC label, and the fully connected layer is used to map the local spatial features and the time features of the historical state parameters to an action space.

[0070] In the embodiment, the CNN network includes multiple convolutional layers, for example, can include a first convolutional layer and a second convolutional layer. The parameters in the historical state parameters except the SOC label are first subjected to feature extraction via the first convolutional layer, and then coupled with a batch normalization layer (BatchNorm1d, bn1) for normalizing the data of the convolutional output to maintain the stability of the distribution of the output data in the training process; and then a nonlinear activation function ReLU is applied. The second convolutional layer and the batch normalization layer and the activation function connected therewith are sequentially arranged in the same manner. Further, a dropout layer is introduced after each convolutional layer and the corresponding activation function, which suppresses the model overfitting by randomly shielding part of the neuron connections in the forward propagation process with a preset probability, wherein the preset probability is a configurable hyperparameter. Finally, the local spatial features of the parameters in the historical state parameters except the SOC label are obtained.

[0071] Optionally, the BiLSTM network includes a bidirectional long short-term memory network layer, the number of hidden units of which can be customized as Hlstm, and the number of network layers can be customized as nlayers. The output of the bidirectional long short-term memory network is the final hidden state, which is formed by splicing the forward propagation hidden state hforward and the backward propagation hidden state hbackward in the feature dimension. Through the above structure, the module can effectively capture the long-term time dependence relationship in the historical state parameters, and simultaneously utilize the historical and future context information to jointly realize accurate prediction of the SOC. In this way, by using multiple time steps in the history as input, combined with the selective memory function of the LSTM structure, the model can balance the use of short-term and long-term information for comprehensive estimation of the SOC. At the same time, the model does not directly use the SOC as input, forcing the model to learn and infer the SOC from the measured data, avoiding circular dependence and error accumulation.

[0072] In the embodiment, the fully connected layer is used to map the high-dimensional feature vector of the fused spatial and temporal features to a predefined action space, and output the state-action value function Q value corresponding to each action. In the reinforcement learning framework, the training goal is to maximize the long-term cumulative reward value; for any state s, the agent is configured to select an action a based on the current Q value, which is used to evaluate the expected cumulative reward that can be obtained by performing the action a in the state s. Through iterative training, the agent gradually learns to select the action that can maximize the Q value, thereby deciding the optimal action in a given state and ultimately obtaining a battery management strategy that can maximize the cumulative reward.

[0073] In an embodiment provided in the present application, based on the above scheme, the CNN network and the BiLSTM network are connected in series, and the output of the CNN network is used as the input of the BiLSTM network.

[0074] In the embodiment, asFigure 2 As shown, the battery state parameters such as voltage, current, temperature and power information of the historical state parameters can be input to the CNN-BiLSTM network for feature extraction of the historical state parameters by the CNN network part in the CNN-BiLSTM network to obtain local spatial features. Specifically, the input data dimension is set as D in = B x L x F, B represents the batch size, L represents the history length, and F represents the number of features (such as voltage, current, temperature, etc.). The CNN network part first processes the data through two convolutional layers Conv1 and Conv2, and the output data dimension is B x C out2 x L. Here C out2 is the number of output channels of the second convolutional layer.

[0075] Optionally, after the CNN network part in the CNN-BiLSTM network extracts the spatial features, the spatial features are input to the BiLSTM network in the CNN-BiLSTM network to obtain time features. The input data dimension of the BiLSTM network part is B x L x C out2 . The output dimension is B x (H lstm x 2).

[0076] Further, the output of the BiLSTM network is taken as the input of the full connection layer, and the output dimension of the full connection layer is , where represents the cardinality of the action space, and the action space includes but is not limited to: reducing the SOC, keeping the SOC unchanged, increasing the SOC, and other discrete actions. By fusing the local spatial features extracted by the convolutional neural network and the bidirectional time-dependent features captured by the bidirectional long short-term memory network, the effective information in the historical state sequence can be fully mined, and the estimation accuracy of the SOC and the reliability of the strategy generation can be significantly improved.

[0077] In an embodiment provided in the present application, based on the above scheme, optionally, experience data for interacting with the environment is generated according to the double reinforcement learning model to construct an experience pool, including:

[0078] Obtaining the current state parameters of the target battery, using the first CNN-BiLSTM network to select the current action corresponding to the current state parameters;

[0079] Obtaining the next state parameters of the target battery after executing the current action, and generating experience data for interacting with the environment; wherein the experience data includes the current state parameters of the target battery, the current action of the target battery, the reward value, the next state parameters of the target battery, and the completion flag;

[0080] Storing the experience data into the experience pool.

[0081] In the embodiment, by performing the current action, one of the following processes can be performed on the current state of charge of the battery: increasing, maintaining, and decreasing, and the processed state of charge is obtained as the new estimated SOC of the target battery. In some embodiments, an adjustment value in the current action can be obtained, and the adjusted SOC of the target battery can be obtained according to the adjustment value and the current state of charge.

[0082] In some embodiments, in the case where the current action is a SOC decreasing action type or a SOC increasing action type, the current state of charge can be changed by an adjustment value in the current action. The adjustment value (an increasing amount or a decreasing amount) in the current action can be determined according to a preset action execution strategy. The adjustment value can be a fixed value, or a dynamic value determined according to at least one of a use scenario and an operating state of the target battery. For example, the current state of charge of the target battery is 50%, and the current action is a SOC decreasing action type, and the adjustment value is 1%. The current state of charge of the target battery at the current time can be decreased, and the decreased state of charge is 49%.

[0083] In the embodiment, the current state parameter s of the target battery, the current action a of the target battery, the reward value r, the next state parameter s' of the target battery, and the completion flag can be used as an experience data, and the experience data is stored in the experience pool, and the experience data is represented as wherein d is the completion flag.

[0084] Optionally, in the case where the experience data is stored in the experience pool, the priority of the experience data can be set, and the priority is specifically as follows:

[0085]

[0086] wherein p i is the priority of the experience data i, δi is the time difference TD error of the experience data i, and ∈ is a small positive number, which is used to prevent the priority from being zero.

[0087] In an embodiment provided in the application, based on the above scheme, optionally, in the training process of the double reinforcement learning model, an ε-greedy strategy is used to randomly select an action; and after the double reinforcement learning model is trained, the ε-greedy strategy is used to select an optimal action.

[0088] In this embodiment, in the training phase, the dual reinforcement learning model adopts an epsilon-greedy strategy to perform random action selection to balance the exploration of unknown states and the utilization of existing knowledge. This strategy can select a non-optimal action with a certain probability, thereby promoting sufficient exploration of the state space. As the training process progresses, the dual reinforcement learning model gradually converges to a more optimal action strategy, so that after the training is completed, the optimal action can be directly selected, thereby improving the overall estimation accuracy.

[0089] Through the reinforcement learning mechanism, the CNN-BiLSTM network ultimately obtains the optimal SOC adjustment strategy and optimizes the strategy based on the reward signal (e.g., SOC estimation error). This enables the model to dynamically and adaptively adjust the SOC estimation strategy, thereby effectively responding to different battery state changes and environmental conditions.

[0090] In an embodiment provided in the present application, based on the above scheme, after obtaining the next state parameter of the target battery after performing the current action, the method further includes:

[0091] determining the current Q value according to the current state parameter, the current action, and the first CNN-BiLSTM network, and selecting the next action corresponding to the next state parameter according to the first CNN-BiLSTM network;

[0092] determining the target Q value according to the next state parameter, the next action, and the second CNN-BiLSTM network;

[0093] updating the model parameters of the first CNN-BiLSTM network according to the current Q value and the target Q value.

[0094] In this embodiment, the target Q value is a reference value used by the model to update its parameters during the training process. It provides a clear target for the model, enabling it to optimize its parameters by minimizing the difference between the target Q value and the current Q value.

[0095] In this embodiment, the current state parameter and the current action can be input into the first CNN-BiLSTM network to obtain the current Q value. Based on the first CNN-BiLSTM network and the current Q value, the next action corresponding to the next state parameter can be selected.

[0096] The second CNN-BiLSTM network is used to evaluate the target Q value of the next action, specifically as follows:

[0097]

[0098] where r is the reward value, γ is the discount factor, d is the completion flag (done), y represents the target Q value, and a' represents the next action.

[0099] Optionally, the reward value r is calculated as follows:

[0100] r = q*r error + e*r smooth

[0101] wherein, r is a reward value, r error is an accuracy reward value, r error = -|SOC 实际 -SOC 预测 |, r smooth is a stability reward, r smooth = -|SOC 预测1 -SOC 预测0 |, q is a weight of r error , e is a weight of r smooth , SOC 实际 is a SOC in a current state parameter, SOC 预测 is a current SOC of a target battery predicted by the first CNN-BiLSTM network, SOC 预测1 is a SOC obtained by performing a next action. SOC 预测0 is a SOC obtained by performing a current action.

[0102] In an embodiment provided in the present application, based on the above scheme, the PER experience replay mechanism is optionally used for priority sampling, and the model is trained in a loop, including:

[0103] determining a TD error of each experience data stored in the experience pool, and determining a priority of each experience data according to the TD error;

[0104] determining a sampling probability of each experience data according to the priority of each experience data;

[0105] sampling the experience pool according to the sampling probability, and calculating a weight of each experience data obtained by sampling;

[0106] determining a loss function value according to the experience data obtained by sampling, and weighting the loss function value by the weight;

[0107] updating the model parameters of the first CNN-BiLSTM network by using the weighted loss function value, and returning to the step of determining the TD error of each experience data stored in the experience pool;

[0108] in the case of reaching a preset time, copying the model parameters in the first CNN-BiLSTM network to the second CNN-BiLSTM network.

[0109] In this embodiment, the data in the experience pool can be sampled to update the network parameters of the first CNN-BiLSTM network by the sampled data, and the sampling probability of the experience data sampled can be determined according to the priority pi The determination is specifically as follows:

[0110]

[0111] wherein, is a sampling probability, a controls the degree of influence of priority in the sampling probability, and represents the sensitivity of the sampling probability to the TD error. When a = 0, the sampling probability of all experiences is equal, which is equivalent to random sampling, and at this time, the effect of prioritizing experience replay is equivalent to the ordinary experience replay mechanism. When a = 1, sampling is completely performed according to the size of the TD error, and the greater the error, the higher the probability of prioritizing the sampling of the experience. K represents the total amount of data in the experience pool.

[0112] In some embodiments, in order to correct the bias, an importance sampling weight wi can be introduced:

[0113]

[0114] wherein N is the size of the experience pool, and β gradually increases from an initial value to 1, and controls the degree of weight; the purpose of defining it is to correct the bias caused by priority sampling.

[0115] In this way, the prioritized experience replay makes the experience with a larger TD error more likely to be sampled, thereby accelerating learning, improving sampling efficiency, making the model pay more attention to the experience with a high value, and improving learning efficiency. At the same time, the importance sampling weight can correct the bias, ensure the unbiasedness of the estimation, and ensure the unbiasedness of the estimation.

[0116] In the prioritized experience replay, some experiences are given a higher sampling probability according to their TD error, and the sampling probability of other experiences may be reduced, so a β is introduced to adjust the sampling weight, so that the model can more fairly use the sampled experience, especially those samples with a lower sampling probability. The initial value of β is usually small (close to 0), and gradually increases during the training process, and finally approaches 1. The purpose of this is that in the early stage of training, the model can focus more on learning samples with a larger TD error, but as the training proceeds, the model is prevented from being biased because of excessive attention to high-error samples.

[0117] In the present embodiment, the loss function is calculated using the sampled experience data, and the loss function value is weighted by the weight, specifically as follows:

[0118]

[0119] wherein, represents the weighted loss function value, wherein, represents the expected value under the experience sampling distribution of state-action-reward-next state-completion flag.

[0120] Optionally, after the weighted loss function value is calculated, the network parameters of the first CNN-BiLSTM network can be updated using an optimization algorithm such as gradient descent.

[0121] In some embodiments, the priority of each experience data in the experience pool is updated according to the new TD error.

[0122] Referring to Figure 3 A schematic diagram of a data processing process of an agent provided in the present application, the agent includes an online network (a first CNN-BiLSTM network) and a target network (a second CNN-BiLSTM network), both networks use a CNN-BiLSTM architecture and share the same model structure to simplify the structure, but the network parameters are updated independently, the online network is used to select actions and update parameters, and the target network is used to calculate target Q values, and the network parameters are fixed for a period of time, thereby decoupling action selection and evaluation through two independent value functions, reducing overestimation problems, the introduction of the target network provides a stable learning target, and improves the stability and convergence speed of training. The online network can select an action a under a state s, and calculate a reward value r, and store the state s, the action a, the reward value r, the actual state s' at the next moment, and the completion flag into a memory pool (experience pool). When the amount of data stored in the memory pool meets the quantity condition, the data in the memory pool can be sampled to select experience data for training. Specifically, the online network can select the next action, and then the target network can evaluate the Q value of the action, and calculate the loss function value based on the Q value, so as to update the online network according to the loss function value. Finally, the network parameters of the online network are copied to the target network at regular intervals.

[0123] Among them, the priority experience replay (PER) aims to improve the efficiency of experience sampling, so that experiences with larger TD errors are more likely to be sampled, thereby accelerating learning. After the online network updates the model parameters, the priority of the experience data in the memory pool can be updated.

[0124] Referring to Figure 4 A flowchart of a training process of an online network provided in the present application, the state s in the environment can be processed by the online network to obtain the rating values of each action in the action space, and then the action a is selected according to the ε-greedy strategy. Then the action a is executed to obtain the next state s', the reward r and the completion flag d. The The experience pool is stored, and when the capacity of the experience pool reaches the batch size, training begins. A batch of experiences can be sampled from the experience pool, and the sampling weight is calculated. The target Q value and the current Q value are calculated, the loss is calculated, and the online network parameters are updated. The priority and sampling weight are updated. Then the online network parameters are periodically copied to the target network. The next state s' is updated to the current state s. After the training is completed, the model performance is evaluated using test data, and evaluation indicators (such as: MSE, MAE, RMSE, R² score) are calculated.

[0125] Taking the battery as an example, the model training and testing process is introduced as follows:

[0126] First, collect 4 times of vehicle driving cycle discharge cycle data, a total of 4x10000 records (sampling at 1 second intervals). In order to reduce the training cost and speed up the testing and training speed, extract the data at 20 points, get 4x500 records (time step is enlarged by 20 times), and then train the four groups of data in turn. Since the algorithm is based on step-by-step adjustment, directly merging the data set may cause excessive adjustment at the initial stage, affecting the training efficiency. In order to avoid forgetting and prevent overfitting, the four groups of data can be randomly selected alternately for mixed training in the total number of epochs (epochs).

[0127] In the model testing phase, considering the large dynamic changes in actual driving cycles, 20 points of data can be randomly extracted from each of the four cycles to form a test set of 500 time points per group. In addition, in order to evaluate the model's autoregressive ability, the initial SOC can be set to 60% in the test set, and the influence of the model's estimation and the model's performance under the condition of unknown or inaccurate initial SOC can be observed. The experimental results are shown in Figure 5 The blue curve represents the true SOC, and the orange curve represents the predicted SOC. After only 100 epochs of training, the R² of the true SOC and the model estimated SOC of the four cycles in the training set is ≥0.99, indicating that the model can effectively learn the SOC changes under different conditions. The performance of the test set is also excellent, proving the model's good generalization ability. Even if the initial SOC is set to 60%, the model can quickly regress to the true SOC trend within 100 time steps.

[0128] Figure 6The training set and test set current and voltage parameter distributions of the first two cycles are shown, where the blue line represents the true SOC value over time steps, and the orange line is the algorithm-predicted SOC value over time steps. The initial SOC is set to 60%, aiming to evaluate whether the model can quickly correct itself to prove its adaptability when the initial SOC is unknown. The figure also shows the changes in current Current and voltage Voltage data, which change greatly during the driving cycle and are complex enough to effectively verify the prediction accuracy of the model under complex conditions. The Voltage Density Distribution part shows the voltage data distribution, and the Current Density Distribution part shows the current data distribution. The peak value represents the most common voltage or current value, and the curve width reflects the dispersion of the data. In addition, Figure 6 The voltage data histogram (voltage Histogram) part and the current data histogram (Current Histogram) part with superimposed density curves further show the dispersion of the data and indicate that the data distribution range of the training set and the test set is consistent, but the probability of the numerical value appearing is different, where cyan is the training data and pink is the test data.

[0129] The training set is used to learn how to make SOC predictions, while the test set is used to evaluate the actual prediction effect of the model under no prompt conditions, which is equivalent to simulating the performance in actual use. In this way, the performance of the model and its generalization ability can be fully evaluated. It is shown that there is a certain difference between the two and the distribution range is extensive, further verifying the robustness of the data.

[0130] In this embodiment, the indicators for estimating the performance of the model include MSE (Mean Squared Error), MAE (Mean Absolute Error), RMSE (Root Mean Squared Error), and R² (Coefficient of Determination), which help to quantify the deviation between the predicted values and the true values of the model, so as to comprehensively evaluate the accuracy of the model. MSE calculates the average of the squares of the differences between the predicted values and the true values, and is particularly sensitive to large errors, so it can effectively reflect the deviation of the model on some samples; MAE is the average of the absolute values of all prediction errors, and each error contributes equally, which is suitable for application scenarios where all errors are equally treated, and directly reflects the average error of the model; RMSE is the square root of MSE, which restores the error to the same magnitude as the original data, retains the sensitivity to large errors, and the result is more intuitive and consistent in units, which is easy to understand; R² measures the explanatory power of the model to the data, and its value is between 0 and 1, 1 indicates perfect fitting, 0 indicates that the prediction ability of the model is the same as using the mean value, and close to 1 indicates that the model has strong prediction ability. By comprehensively using these indicators, the performance of the model can be comprehensively and accurately evaluated.

[0131] In this embodiment, the specific evaluation indicators of the model are as shown in Table 1:

[0132] Table 1

[0133]

[0134] With Figure 1 Corresponding to the method, the embodiment of the application also provides a battery SOC online estimation device based on deep reinforcement learning, which is used for the specific implementation of the method in the Figure 1 The structural diagram of the device is as shown in Figure 7 It includes:

[0135] The acquisition unit 701 is configured to acquire historical state parameters of a target battery, wherein the historical state parameters at least include voltage, current, temperature, and an SOC label corresponding to the historical state parameters;

[0136] The execution unit 702 is configured to define an action space corresponding to the historical state parameters, wherein the action space includes actions of multiple action types, and each action is used to represent an SOC adjustment amplitude of the target battery; the action types include reducing the SOC, keeping the SOC unchanged, or increasing the SOC;

[0137] The first construction unit 703 is configured to construct a double reinforcement learning model combined with a CNN-BiLSTM network and a double deep Q network, wherein the first CNN-BiLSTM network in the double reinforcement learning model is used to select an action in the action space and update parameters, and the second CNN-BiLSTM network is used to calculate a target Q value;

[0138] a second construction unit 704, configured to generate experience data of interaction with the environment according to the double reinforcement learning model, to construct an experience pool;

[0139] a training unit 705, configured to cyclically train the double reinforcement learning model by using a PER experience replay mechanism to preferentially sample, until the double reinforcement learning model training reaches a preset condition;

[0140] an online estimation unit 706, configured to obtain a current state parameter of a target battery, input the trained double reinforcement learning model, and output a current SOC of the target battery.

[0141] In an embodiment provided in the present application, based on the above scheme, optionally, the CNN-BiLSTM network comprises a CNN network, a BiLSTM network and a full connection layer; the CNN network is configured to extract local spatial features of parameters in the historical state parameters except the SOC label; the BiLSTM network is configured to extract time features of the historical state parameters; and the full connection layer is configured to map the local spatial features and the time features of the historical state parameters to the action space.

[0142] In an embodiment provided in the present application, based on the above scheme, optionally, the CNN network and the BiLSTM network are connected in series, and an output of the CNN network is used as an input of the BiLSTM network.

[0143] In an embodiment provided in the present application, based on the above scheme, optionally, the second construction unit comprises:

[0144] a first acquisition sub-unit, configured to acquire a current state parameter of the target battery, and use the first CNN-BiLSTM network to select a current action corresponding to the current state parameter;

[0145] a second acquisition sub-unit, configured to acquire a next state parameter of the target battery after performing the current action, and generate experience data of interaction with the environment; wherein the experience data comprises the current state parameter of the target battery, the current action of the target battery, a reward value, the next state parameter of the target battery and a completion flag;

[0146] a storage sub-unit, configured to store the experience data to the experience pool.

[0147] In an embodiment provided in the present application, based on the above scheme, optionally, in the training process of the double reinforcement learning model, an ε-greedy strategy is used to randomly select an action; and after the double reinforcement learning model is trained, the ε-greedy strategy is used to select an optimal action.

[0148] In an embodiment provided by the present application, based on the above scheme, optionally, the second construction unit further comprises:

[0149] a selection unit configured to determine a current Q value according to the current state parameter, the current action and the first CNN-BiLSTM network, and select a next action corresponding to the next state parameter according to the first CNN-BiLSTM network;

[0150] a first determination unit configured to determine a target Q value according to the next state parameter, the next action and the second CNN-BiLSTM network;

[0151] an update unit configured to update model parameters of the first CNN-BiLSTM network according to the current Q value and the target Q value.

[0152] In an embodiment provided by the present application, based on the above scheme, optionally, the training unit 705 comprises:

[0153] a second determination unit configured to determine a TD error of each of the experience data stored in the experience pool, and determine a priority of each of the experience data according to the TD error;

[0154] a third determination unit configured to determine a sampling probability of each of the experience data according to the priority of each of the experience data;

[0155] a sampling unit configured to sample the experience pool according to the sampling probability, and calculate a weight of each of the experience data sampled;

[0156] a fourth determination unit configured to determine a loss function value according to the experience data sampled, and weight the loss function value by the weight;

[0157] a first execution unit configured to update the model parameters of the first CNN-BiLSTM network by using the loss function value weighted, and return to the step of determining the TD error of each of the experience data stored in the experience pool;

[0158] a second execution unit configured to copy the model parameters in the first CNN-BiLSTM network to the second CNN-BiLSTM network in the case of reaching a preset time.

[0159] The specific principles and execution processes of each unit and module in the battery SOC online estimation device based on deep reinforcement learning disclosed in the embodiments of the present application are the same as those of the battery SOC online estimation method based on deep reinforcement learning disclosed in the embodiments of the present application. For details, refer to the corresponding part of the battery SOC online estimation method based on deep reinforcement learning provided in the embodiments of the present application, which will not be described here.

[0160] The embodiments of the present application also provide a storage medium including stored instructions, wherein the instructions, when executed, control a device where the storage medium is located to perform the battery SOC online estimation method based on deep reinforcement learning.

[0161] The embodiments of the present application also provide an electronic device, a structural schematic diagram of which is shown in FIG. 8. Figure 8 The embodiments of the present application also provide an electronic device, a structural schematic diagram of which is shown in FIG. 8.

[0162] It should be noted that each of the embodiments in the present specification adopts a progressive manner for description, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0163] Finally, it should also be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0164] For the convenience of description, the above device is described as various units in terms of functions and is described respectively. Of course, in the implementation of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0165] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software and the necessary universal hardware platform. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product in essence or in the part of the contribution to the prior art, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the application.

[0166] The above describes in detail the battery SOC online estimation method based on deep reinforcement learning provided by the application, the principle and implementation of the application are described by applying specific examples, and the above embodiment is only used to help understand the method and core idea of the application; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation and application range will be changed, and the above description should not be understood as limiting the application.

Claims

1. A battery SOC online estimation method based on deep reinforcement learning, characterized in that, include: Obtain historical state parameters of the target battery, including at least voltage, current, temperature, and the SOC tag corresponding to the historical state parameters; Define the action space corresponding to the historical state parameters. The action space includes multiple action types, each of which represents the adjustment range of the SOC of the target battery. The action types include decreasing SOC, keeping SOC unchanged, or increasing SOC. A dual reinforcement learning model combining a CNN-BiLSTM network and a dual deep Q-network is constructed, wherein the first CNN-BiLSTM network in the dual reinforcement learning model is used to select actions in the action space and update parameters, and the second CNN-BiLSTM network is used to calculate the target Q-value. Based on the dual reinforcement learning model, experience data on interaction with the environment is generated to construct an experience pool; The dual reinforcement learning model is trained cyclically by prioritizing sampling through the PER experience replay mechanism until the training of the dual reinforcement learning model reaches the preset conditions. The current state parameters of the target battery are obtained and input into the trained dual reinforcement learning model to output the current SOC of the target battery. The current state parameters do not include the current SOC.

2. The method according to claim 1, characterized in that, The CNN-BiLSTM network includes a CNN network, a BiLSTM network, and a fully connected layer; the CNN network is used to extract local spatial features of the parameters other than the SOC label in the historical state parameters; The BiLSTM network is used to extract the temporal features of the parameters other than the SOC label in the historical state parameters, and the fully connected layer is used to map the local spatial features and temporal features of the historical state parameters to the action space.

3. The method according to claim 2, characterized in that, The CNN network and the BiLSTM network are connected in series, with the output of the CNN network serving as the input of the BiLSTM network.

4. The method according to claim 2, characterized in that, The step of generating experiential data on interaction with the environment based on the dual reinforcement learning model to construct an experience pool includes: Obtain the current state parameters of the target battery, and use the first CNN-BiLSTM network to select the current action corresponding to the current state parameters; The next state parameters of the target battery after performing the current action are obtained, and experience data on interaction with the environment is generated; wherein, the experience data includes the current state parameters of the target battery, the current action of the target battery, the reward value, the next state parameters of the target battery, and the completion flag; The experience data is stored in the experience pool.

5. The method according to claim 4, characterized in that, During the training of the dual reinforcement learning model, an ε-greedy strategy is used to randomly select actions; after the dual reinforcement learning model is trained, an ε-greedy strategy is used to select the optimal action.

6. The method according to claim 4, characterized in that, After obtaining the next state parameter of the target battery after performing the current action, the method further includes: The current Q value is determined based on the current state parameter, the current action, and the first CNN-BiLSTM network, and the next action corresponding to the next state parameter is selected based on the first CNN-BiLSTM network. The target Q-value is determined based on the next state parameter, the next action, and the second CNN-BiLSTM network; The model parameters of the first CNN-BiLSTM network are updated based on the current Q value and the target Q value.

7. The method according to claim 4, characterized in that, The method of prioritizing sampling and iteratively training the model through the PER experience replay mechanism includes: Determine the TD error of each piece of experience data stored in the experience pool, and determine the priority of each piece of experience data based on the TD error; The sampling probability of each empirical data point is determined based on its priority. The experience pool is sampled according to the sampling probability, and the weight of each sampled experience data is calculated. The loss function value is determined based on the empirical data obtained from sampling, and the loss function value is weighted by the weights. The model parameters of the first CNN-BiLSTM network are updated using the weighted loss function value, and the process returns to the step of determining the TD error for each of the empirical data stored in the empirical pool. Upon reaching a preset time, the model parameters in the first CNN-BiLSTM network are copied to the second CNN-BiLSTM network.

8. A battery SOC online estimation device based on deep reinforcement learning, characterized in that, include: The acquisition unit is used to acquire historical state parameters of the target battery, wherein the historical state parameters include at least voltage, current, temperature, and the SOC tag corresponding to the historical state parameters; An execution unit is used to define the action space corresponding to the historical state parameters. The action space includes multiple action types, each of which represents the adjustment range of the SOC of the target battery. The action types include decreasing SOC, keeping SOC unchanged, or increasing SOC. The first building unit is used to build a dual reinforcement learning model that combines a CNN-BiLSTM network and a dual deep Q network. In the dual reinforcement learning model, the first CNN-BiLSTM network is used to select actions in the action space and update parameters, and the second CNN-BiLSTM network is used to calculate the target Q value. The second construction unit is used to generate experiential data on interaction with the environment based on the dual reinforcement learning model, so as to construct an experience pool; The training unit is used to preferentially sample and cyclically train the dual reinforcement learning model through the PER experience replay mechanism until the dual reinforcement learning model training reaches the preset conditions. An online estimation unit is used to obtain the current state parameters of the target battery, input into the trained dual reinforcement learning model, and output the current SOC of the target battery. The current state parameters do not include the current SOC.

9. A storage medium, characterized in that, The storage medium includes stored instructions, wherein, when the instructions are executed, the device in which the storage medium resides controls the execution of the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes memory, and one or more instructions, wherein one or more instructions are stored in memory and configured to be executed by one or more processors as described in any one of claims 1-7.

Images